Clone
 

justin du <justin.du@hp.com> in Trafodion

Stop sending user CQDs at embedded compile switch

Do not send user entered CQDs when switching to embedded META CmpContext

to compile metadata queries.

Also do the holdAndSet needed CQDs only when the embedded (META)

CmpContext is first switched to. These CQDs won't be restored when

switching back so that we won't send the same set of CQDs again next

time the CmpContext is used.

LP bug 1438372, Compiler returns internal error from

../optimizer/opt.cpp line 6907

LP bug 1444739, compGeneral/TEST006 failed with tmp fix for bug 1438372

Change-Id: I92398aeeaf53820d3f8af1fb73e9c1d6f449ce5e

    • -82
    • +66
    /sql/regress/compGeneral/EXPECTED042
    • -53
    • +93
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
Updated known diff file for compGeneral/TEST006

Temporarily allow failures at propagating MODE_SPECIAL_1 cqd. The real

problem will be traced on another LP bug.

Change-Id: I45be0496eaa01f13cac7b953fd3313ad91da2447

Temporary fix for LP bug 1438372

Hold and set CQD optimization_level to 3 during DDL query compilation

when hash joins and merge joins are turned off. This is to prevent the

optimization level set by user to other than 3 (maximum) to affect

meta-data queries. In the future, any secondary compiler won't get any

user CQDs, thus this is temporary fix as work-around.

Change-Id: Idc36d3e98f7729937e6c8a16ab95a41f68c8823d

Hash buffer size adjustment

Increased the hash buffer size for hash join operator to accommodate

the needs of extra space at runtime. This is to make sure that each hash

buffer can store at least one (big) row.

Fix bug 1438943, select query with outer join of 3 repos tables

from trafci failing with error 201 and esp core.

Change-Id: I58aed6546637e1097dd832b39e53dd53dcb56ab8

Preventing user CQDs from affecting metadata queries

We now won't pass user CQDs to new CmpContext instance (CI) during

embedded CI switch. This is to prevent metadata query plans

being influenced by user CQDs unintentionally. Also, temporarily added a

flag to retrieve the latest hbase snapshot info for user tables

disrespect to whether we use the snapshots for scan or not. The CQD

TRAF_TABLE_SNAPSHOT_SCAN will only affect the user query plan

generations.

Reworked LP 1420826, Turning off nested_joins fails meta-data query

compilation, by reverting the changes because when compiling the

metadata queries, the default optimization level would be '3' and won't

be affected by user setting anymore.

LP 1426479: Row mismatch between index/table cause init auth to fail.

Fixed LP 1411355: User-level CQDs also seem to apply to metadata

queries.

Change-Id: Idd0388b626ecd3dbf3c3d1e75c1d6dc30b0ce021

    • -67
    • +77
    /sql/regress/compGeneral/EXPECTED042
    • -3
    • +29
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
Adding more run-time memory allocations from NAHeap

This set of changes moves some of the string vector variables in HBase

access operators from standard string template to our NAList and

NAString (or HbaseStr for row IDs). In the process, allocationis of the

objects will be from our HAHeap instead of the system heap. This would

help us tracking memory usage and detecting leaks easier.

In addition, a change in ExHbaseAccessTcb::setupListOfColNames()

prevents unnecessary allocations to populate the columns list unless it

is empty. The Google profiling tools helped us on identifying this

problem.

also, removed ExHbaseAccessDeleteTcb operator which was not used.

Change-Id: I87ab674ab8e3d291f2fc9563718d88de537ae96b

    • -29
    • +61
    /sql/executor/HBaseClient_JNI.cpp
    • -17
    • +78
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
    • -2
    • +25
    /sql/sqlcomp/CmpSeabaseDDLtable.cpp
Enable SQL memory overflow check for debug tests

This testware changes will turn on the SQL memory overflow checking for

NAMemory allocation while running debug regression tests. For some

regression test suites, this could increase the run time to nearly 20%

while the increased time should be less for rest of test suites.

Plus a note added in the comment section for using the memory allocation

tracing feature.

Change-Id: I2e7e5f6b29fb2f0cee8f3a2928ac67b52c914ad5

    • -6
    • +1
    /sql/regress/tools/runregr_executor.ksh
Fix memory leak when ComDiagsArea is not deleted

In ExControlTcb, the allocated ComDiagsArea for the embedded compiler

to use did not get freed, resulting 336-byte block leak in the

ContextCli heap each time a set query default statement was executed.

This fix gets the memory for the Statement heap and frees it after use.

Change-Id: I0ef68e6d3c9372a8963ff8da375479af8d64c198

Fix memory leak when ComDiagsArea is not deleted

In ExControlTcb, the allocated ComDiagsArea for the embedded compiler

to use did not get freed, resulting 336-byte block leak in the

ContextCli heap each time a set query default statement was executed.

This fix gets the memory for the Statement heap and frees it after use.

Change-Id: I0ef68e6d3c9372a8963ff8da375479af8d64c198

Display Hive table access stats for get statistics

Fix for bug 1413406 get statistics for query would hang when hive table

involved.

Add code to retrieve and display hive table statistics for get

statistics command, which is similar to display the hbase table stats

info.

Change-Id: I38d7633a8333cbb08263570ebd6b68b192f2a882

    • -2
    • +136
    /sql/executor/ExExeUtilGetStats.cpp
Fix for LP bug 1404090

A fix for LP bug 1404090 by disabling HBase rowset vsbb access

if executor predicates exist. This will make the getColumn tests pass in

jdbc test suite found during daily build regression.

Change-Id: I1283c0d2dd8f5d50be77b3b95af018598af4b82c

Add max I/O time to Hbase runtime stats

Collect and display the maximum Hbase operator I/O time. In a parallel

plan, this is chosen from the biggest Hbase I/O elapsed time among all

the same Hbase tcb instances at the runtime stats merge stage. For

serial plan the value equals to the Hbase I/O elapsed time.

In addition, the displayed Hbase I/O is changed from byte to MB.

Change-Id: Icb4cbcbee5eeaf7143762444d48288f7bf6f5784

    • -8
    • +15
    /sql/executor/ExExeUtilGetStats.cpp
    • -13
    • +13
    /sql/executor/HBaseClient_JNI.cpp
Miscellaneous minor changes

1) Revert a part of previous change to ComTdbHbaseAccess::displayRowId() for

SHOWPLAN. Each valid row id should have 2 bytes of length followed by the

value of id, if it is not missing or null. We need to treat the id as

binary value so we'll print each id byte as 2-digit hex value up to 20

bytes.

2) Remove the part of comments in sql/sqlcomp/DefineConstants.h that

requires not to delete and reuse existing constants. This is no longer

mandatory in Trafodion.

3) Enable NA memory overflow checking when running executor and qat

suites in debug flavor. This would increase the run time for these two

suites by about 10%.

Rework according to review comments. Explanations above changed too.

Change-Id: I854ec776fa5fb815c39677d469f8f46f18d51402

    • -20
    • +11
    /sql/comexe/ComTdbHbaseAccess.cpp
    • -0
    • +5
    /sql/regress/tools/runregr_executor.ksh
SHOWPLAN related changes

Reworked the fix for bug 1392522 - mxosrvr core dumped doing showplan

(with N.E. enabled), plus other related code changes.

1) The native code (also known as native expression), if generated, is

stored in the expression's constant area. SHOWPLAN will dump the native

code in the assembly format. The display can be disabled by CQD

PCODE_NE_IN_SHOWPLAN to "OFF". It is "ON" by default. This part had been

reviewed by Jim Capps and Mike Hanlon.

2) Add several SHOWPLAN statements in core/TEST019 without logging the

output. This is to ensure no core generated when getting executor

operator (TDBs) info via SHOWPLAN.

3) A temporary fix to ComTdbHbaseAccess::displayRowId(). The current way

of retrieving begin or end row IDs from the HbaseScanRows for SHOWPLAN

does not match with the way those row IDs are generated (see

HbaseAccess::genListsOfRows), causing core dumps in some cases.

Change-Id: Id33e8c854452c03e15c4bd21f668f123180d9471

    • -26
    • +31
    /sql/exp/ExpPCodeOptsNativeExpr.cpp
Enabling runtime stats for hbase tables and operators

This is the third set of changes to collect the runtime stats info. Part

is to address the comments and suggestions from last review.

1) Instead of passing the hbase access stats entry to every htable

calls, set the pointer in the EXP hbase interface layer with first init

call in the tcb work methods (not the task work methods), then

eventually to the htable JNI layer from getHTableClient()

(sql/exp/ExpHbaseInterface.cpp).

2) Rewrite the way to construct the hbase operator names from one

methord and use it for display both tdb contents and tcb stats.

3) Populate the hbase I/O bytes counter for both read and insert

(sql/executor/HBaseClient_JNI.cpp).

4) Fix the problem that parsing stats variable text string could go

beyond the end of the string (getSubstrInfo() in

sql/executor/ExExeUtilGetStats.cpp).

Change-Id: I62618b57894039bc1ca5bc0f3c9b89efec5cc42e

    • -69
    • +178
    /sql/comexe/ComTdbHbaseAccess.cpp
    • -47
    • +54
    /sql/executor/ExExeUtilGetStats.cpp
    • -79
    • +107
    /sql/executor/HBaseClient_JNI.cpp
Enabling runtime stats for hbase tables and operators

This is the second set of changes to collect the runtime stats info for

hbase tables and operators. It contains:

1) Stats for hbase IUD operations

2) Moved incActualRowsReturned() call to

ExHbaseAccessTcb::moveRowToUpQueue()

3) Added Hbase call counter

4) Display full hbase operator names instead of generic

"EX_HBASE_ACCESS" for hbase operator runtime stats

Change-Id: I94d727c897876a429b588f9acb3fec465dd56fe5

    • -128
    • +120
    /sql/executor/ExHbaseIUD.cpp
    • -15
    • +92
    /sql/executor/HBaseClient_JNI.cpp
Enabling runtime stats for hbase operators

This is the first set of changes to collect the runtime stats info for

hbase tables and operators. It contains:

1) Populate the estimated row count in hbase access TDB.

2) Collect the hbase access time and accessed row count at the JNI layer

(only for select operations now).

Partially reviewed by Mike H. and Selva G.

Removed the part that devides the estimated rows by number of ESPs based on the comments

Change-Id: I5a98a8ae9c4462aa53ad889edfe4cd8563502477

SQL Memory allocation tracing and overflow detection

1) Added code to record stack info when given size of memory blocks are

allocated and dump these info out to a file or terminal if the

deallocation is not made at the time the heap is distructed. This works

for heaps constructed from NAMemory.

2) Fixed the memory debug code to detected memory overflow at the

de-allocation time.

These two features are enabled through environment variables in objects

compiled in debug mode. See sql/common/NAMemory.cpp for details.

3) Fixes for memory usage issues found when testing above code.

Change-Id: Id0e180aee3d069de11836904e80a4290b180dc67

Fix for bug 1360493 - Use correct row ID length

Closes-Bug: 1360493

Change to use the key length value, if explicitly set, as hbase table row ID length.

Also, add the tests that uncover this bug to the regression test set.

Change-Id: I29c47348ef3242d557c42f5ce4e92f7b7fe6b2b8

Squashed commit of the following:

commit 1b8106079000418f4afa6ce0e247e81f3f5b2e2c

Merge: 7b7c311 e79bbdf

Author: Justin Du <justin.du@hp.com>

Date: Thu Jun 26 13:41:50 2014 -0800

Merge remote branch 'gerrit/master' into bp/reentrant_cmp

Conflicts:

sql/common/arkcmp_proc.cpp

Change-Id: I56a9cd33544e863c233a1b4e6a811bfb15efe27e

commit 7b7c3110c55f837672240f25dd5496540212ba17

Author: Justin Du <justin.du@hp.com>

Date: Mon Jun 23 13:33:27 2014 -0800

Expect file changes for executor/TEST013

To report the expected error (8193) when the schema name has reserved

word as perfix for either types of statements.

Change-Id: I8ce1854afa1d916eecd01c70d27a89857fb6c2a9

commit 376666657d99b200fbcf8cd5e177096619d44c0e

Author: Justin Du <justin.du@hp.com>

Date: Thu Jun 19 13:03:06 2014 -0800

Pass diags info between CmpContext instances

1) Preserve the diags info when restoring cqds and controls after meta

data access.

2) Pass diags info from current CmpContext to the previous CmpContext at

the CmpContext switch back call.

Change-Id: Ibac9ff19c82f8dc17f278ec4327f39837085503e

commit b22407a2b78ff8c2b7f1ea14c908833fe8e98540

Author: Justin Du <justin.du@hp.com>

Date: Fri Jun 13 17:08:24 2014 -0800

Rework after review

Added warning (SQLCODE 2032) and assert when improperly using CmpContext

switch logic

Change-Id: I32e6e4f2168e3cf52dc58ec85da1555f4fd29051

commit 2870519283471f6b3c5ea844875546b0dda32a04

Author: Justin Du <justin.du@hp.com>

Date: Thu Jun 12 10:14:12 2014 -0800

Bug fixes related to CmpContext switch

1. CmpContext switch takes place only if the embedded compiler is

involved.

2. Fix for TMF error 75 (process doesn't have the active transaction)

see in DDL operations.

Change-Id: Icdca929c4c782464e1f8267d497bf91518a8b3a1

commit 1cebd4b2c6134097a2b4bd419b00fdee71d93f1f

Merge: 8145ba5 0862f91

Author: Justin Du <justin.du@hp.com>

Date: Tue Jun 10 11:21:11 2014 -0800

Merge remote branch 'gerrit/master' into bp/reentrant_cmp

Conflicts:

sql/arkcmp/CmpStatement.cpp

Change-Id: Iba12fa3fb64809f9c9393d06db0215d7bbf6ee7d

commit 8145ba58927d0d954b68183031747a77367e000f

Author: Justin Du <justin.du@hp.com>

Date: Tue Jun 10 10:12:59 2014 -0800

Associate global empty input LogProp with CmpContext

1. Changed GLOBAL_EMPTY_INPUT_LOGPROP as alias (via #define) instead of

thread pointer to access the default input LogProp from curent

CmpContext.

2. Restored recursion counter for embedded compiler so that histogram

access is done by external compiler

Change-Id: I32560e3b0b1dfe5fc2b1ee9a839d75bfdb57fa9a

commit 80b36c32a0db1cf0dc5ae5cead2a04522da1231e

Author: Justin Du <justin.du@hp.com>

Date: Mon Jun 2 20:58:47 2014 -0800

First set of recursive compilation with CmpContext switch

1. Disable recursion counter when entering compileDirect

2. Starting or reuse CmpContext for metadata access during compilation

3. Fixed few problems in CmpContext switching code.

Change-Id: Iff66309319e989a247b80d92ea8e3e32a35e1755

Change-Id: Ib096a46288a8616fc26c9f52474873778e76ed8b

  1. … 4 more files in changeset.
Using embedded compiler will not get error 2031

There is no need to raise error 2031 (Could not use embedded compiler to

compile the statement: ...) when failed to compile a query. Removed the

AQR entry for SQL error 2031.

Change-Id: I3979574ea9405d0b35c0c7924e30ad64475c6c10

Make file name change

Renamed the make file in sqf/sql to make it compliant with other make

file names.

Change-Id: I280e74f704a4051df8a56411aabc77ffd058499f

fix for LaunchPad bug #1322451: SPJ calls do not work in Trafci

The issue is that the statement type was not set properly at compile time for CALL statements

Added a member variable in BindWA class to indicate call statement. The variable is set at bind stage and the statement type is set based on its value by the generator.

Change-Id: I7174a28ff90acc9b10875f8fe7fd49a933e3cb19