ExStats.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
Display Hive table access stats for get statistics

Fix for bug 1413406 get statistics for query would hang when hive table

involved.

Add code to retrieve and display hive table statistics for get

statistics command, which is similar to display the hbase table stats

info.

Change-Id: I38d7633a8333cbb08263570ebd6b68b192f2a882

  1. … 5 more files in changeset.
Add max I/O time to Hbase runtime stats

Collect and display the maximum Hbase operator I/O time. In a parallel

plan, this is chosen from the biggest Hbase I/O elapsed time among all

the same Hbase tcb instances at the runtime stats merge stage. For

serial plan the value equals to the Hbase I/O elapsed time.

In addition, the displayed Hbase I/O is changed from byte to MB.

Change-Id: Icb4cbcbee5eeaf7143762444d48288f7bf6f5784

  1. … 4 more files in changeset.
Various changes, details listed below.

-- fixed error msg 1429 text

-- added code to set objectUID & owner for metadata, histogram and

sequence tables during creation of metadata structs for these objects.

-- removed previously added code in binder that computed objectUID

for sequence.

-- Updated method lookupObjectUid to call an existing method

to get objectuid.

-- removed obsolete code for reorg, replicate and load

Change-Id: I60d161cfa72bcc674dc6c64e3a07237c7522ee6c

  1. … 28 more files in changeset.
removal of obsolete code that is no longer valid and unused in Trafodion

Change-Id: Ie9212627d102661fe1a7b02fd6e9ae9a23198f2a

  1. … 35 more files in changeset.
Merge "Enabling runtime stats for hbase tables and operators"

  1. … 1 more file in changeset.
Query Invalidation triggered by DDL, phase 3

This check-in allows invalidation of queries which have been prepared

and are held by the EXE for execution or re-execution. It does not

invalidate running queries.

When an attempt is made to (re)execute an invalidated query, a

special SQLCODE, 8738, is raised and the query is sent ack to the

compiler via the AQR mechanism. The check-in include a new test cases

in the executor/TEST122 regression test which demonstrate the

functionality.

Change-Id: I98ea484f9d7fc9e43372e3057d8e8288162865c1

Implements: blueprint ddl-query-invalidation

  1. … 17 more files in changeset.
Enabling runtime stats for hbase tables and operators

This is the third set of changes to collect the runtime stats info. Part

is to address the comments and suggestions from last review.

1) Instead of passing the hbase access stats entry to every htable

calls, set the pointer in the EXP hbase interface layer with first init

call in the tcb work methods (not the task work methods), then

eventually to the htable JNI layer from getHTableClient()

(sql/exp/ExpHbaseInterface.cpp).

2) Rewrite the way to construct the hbase operator names from one

methord and use it for display both tdb contents and tcb stats.

3) Populate the hbase I/O bytes counter for both read and insert

(sql/executor/HBaseClient_JNI.cpp).

4) Fix the problem that parsing stats variable text string could go

beyond the end of the string (getSubstrInfo() in

sql/executor/ExExeUtilGetStats.cpp).

Change-Id: I62618b57894039bc1ca5bc0f3c9b89efec5cc42e

  1. … 15 more files in changeset.
Query Invalidation triggered by DDL, phase 1

This first check-in implements most of the framework which will

be used to complete the QI DDL feature. It redefines the old

security invalidation key (SQL_SIKEY) to handle DDL operations in

addition to REVOKE. In a limited number of DDL operations, the object

UIDs of affected Seabase objects are propagated to all nodes for

use by the compiler to invalidate NATable cache entries, as

well as a limited number of types of cached queries. Later this

month, the framework will be complete by allowing prepared queries

that have already been returned from the compiler to be invalidated.

Then the next step for the framework will be support for invalidating

the HTable cache. Finally an effort will be made to cover all of

the necessary DDL operations and all types of cached queries.

The check-in include a new regression test (executor/TEST122) that

demonstrates the cases that are covered. Specifically, a table will

be dropped and recreated with the same name but different definition

in one sqlci session. In another session, which has already populated

NATable cache and query cache for INSERT, UPDATE, DELETE, SELECT,

SELECT COUNT(*), INVOKE and SHOWDDL statements, those some types

of statements will be resubmitted and correctly compiled.

Change-Id: Ie61ce751089b57ce1894f1764c338e9400bb7b8a

Closes-Bug: #1329358

Implements: blueprint ddl-query-invalidation

  1. … 41 more files in changeset.
Enabling runtime stats for hbase tables and operators

This is the second set of changes to collect the runtime stats info for

hbase tables and operators. It contains:

1) Stats for hbase IUD operations

2) Moved incActualRowsReturned() call to

ExHbaseAccessTcb::moveRowToUpQueue()

3) Added Hbase call counter

4) Display full hbase operator names instead of generic

"EX_HBASE_ACCESS" for hbase operator runtime stats

Change-Id: I94d727c897876a429b588f9acb3fec465dd56fe5

  1. … 11 more files in changeset.
Enable RMS monitoring

The query execution statistics can now be obtained while the query is running

via

a) GET STATISTICS FOR QID <qid>

b) Using STATISTICS table-valued function

c) Using CLI calls – SQL_EXEC_GetStatistics2 and SQL_EXEC_GetStatisticsItems

The statistics can be viewed in accumulated, pertable, progress and operator

stats format.

You can also obtain statistics for each fragment instance and process wise

statistics for each operator of an active query.

Light-weight SQL Offender (LSO) feature is also enabled. With LSO, you can get

1. Queries offending CPU resource

2. Queries blocked in Trafodion Engine

3. Queries that are not active

4. Queries that haven’t finished but blocked in client

5. Queries offending memory resource

Explain in RMS feature can be enabled by setting the

cqd explain_in_rms ‘on’ before compiling a query.

EXPLAIN.. FOR QID <qid> command is now supported.

GET PROCESS STATISTICS FOR <pid> is also supported

Change-Id: I59eb54dd4fd347a37e0b7e7842ee242e553ef60d

  1. … 48 more files in changeset.
Enabling runtime stats for hbase operators

This is the first set of changes to collect the runtime stats info for

hbase tables and operators. It contains:

1) Populate the estimated row count in hbase access TDB.

2) Collect the hbase access time and accessed row count at the JNI layer

(only for select operations now).

Partially reviewed by Mike H. and Selva G.

Removed the part that devides the estimated rows by number of ESPs based on the comments

Change-Id: I5a98a8ae9c4462aa53ad889edfe4cd8563502477

  1. … 11 more files in changeset.
Security changes to support authorization

Added support for authorization commands:

- initialize authorization [, drop]

- create/drop roles

- register/unregister components

- create/drop component operations

- grant/revoke object privileges

- grant/revoke role privileges

- grant/revoke component privileges

- updates to GET and SHOWDDL statements

- checking of privileges for DML requests

- checking of privileges for DDL requests

- regression tests added to catman1 library

Fixed a testware problem in catman1 TEST135 and TEST139

Fixed a parser problem introduced by compGeneral/TESTTOK2 which was recently

introduced.

More details:

This delivery was part of code worked on by many people for several

months on a remote branch. This team held bi-weekly meetings

for several months to design and implement these features. These

meetings also included extensive code reviews.

The security features which include authentication (which was delivered

in June) and authorization is turned off by default. The

traf_authentication_setup script located in $MY_SQROOT/sql/scripts needs

to be run to enable both authentication and authorization. This

procedure is described on the Trafodion Twiki page and will be updated once this

delivery completed to include authorzation.

Delivery updates:

Updated traf_authentication_setup to return consistent error messages

and added a comment to ComSmallDefs.h to address a buf size issue for

metadata tables.

Change-Id: I896f1ee006590284653b2c9882901c05b5f2ba22

  1. … 100 more files in changeset.
Enable basic Runtime Management System (RMS)

The goal of these initial changes is to get the shared memory

segment created, build and start the persistent server processes

mxssmp and mxsscp, make EXE processes register in the shared segment

and let mxssmp cleanup the shared segment as the processes stop.

Later I will address specific features of RMS that support

Query Invalidation.

Change-Id: Ib3dd10525b6073d32d1e362f301afe9c88012647

Implements: blueprint enable-rms-infrastructure

  1. … 27 more files in changeset.
Initial code drop of Trafodion

  1. … 4886 more files in changeset.