Clone
 

eric owhadi <eric.owhadi@esgyn.com> in Trafodion

[TRAFODION-2422] populateSortCols was flagged as major perf offender during profiling the reason being an unbound iterative string search on huge string. Added a minor fix: When not setting category in log conf file, extra code is executed, potentially impacting performance

[TRAFODION-2009] fix parallel scanner issue not scanning full regions. Was linked with cache size handling. Cache was kept same size as original cache size on parallele scanner, now it is divided by the number of threads. Added a test case in regression test to validate the fix.

    • -81
    • +236
    /core/sql/regress/executor/EXPECTED140
oops, forgot the .gitignore, so jenkns build was failing, while dev build was ok

Merge branch 'master' of git://git.apache.org/incubator-trafodion into parallelScan

Conflicts:

core/sql/sqlcomp/DefaultConstants.h

[TRAFODION-1421] Implement parallel scanner primitive new CQD: hbase_dop_parallel_scanner - default 0.0 = disabled - from 0.x to 1.0 -> the DOP will be dynamically assessed as a percentage of region to scan. If 10 region, and cqd is .3 (30%), then 3 threads will be used... - from 2.0 and above, will take ceiling, and use it as fixed number of threads. Advantage of parallel scanner parallelism over ESP is on resource consumption. Parallel scanner is just creating threads, while ESP are full proceses wth high memory consumption. For now, this feature is just a primitive that can be manually exercised. The next obvious step is to make the compiler aware of it and pick it over ESP DOP when appropriate...

    • -1
    • +4
    /core/sql/executor/HBaseClient_JNI.cpp
  1. … 7 more files in changeset.
Add comments to explain the use of stack variable to minimize impact on runtime code based optimization.

[TRAFODION-1900]

When doing MDAM scans, we are performing interlaced scan for PROBE and for real scan. The probes always return only 1 row, then we close the scanner immediately, therefore should use always small scanner. I will make it conditional on the existing CQD HBASE_SMALL_SCANNER (ether SYSTEM or ON). In addition, caching of blocks retrieved by probe should always we at least receiving one succesfull cache hit on the next MDAM scan, therefore forcing caching ON for MDAM prob is a good idea. Again, will make this forcing conditional on HBASE_SMALL_SCANNER SYSTEM or ON.

Then for the real scan part of MDAM, I will use the following heuristic: If previous scan fitted in one hbase block, then it is likelly than next will also fit in one hbase block, therefore enable small scanner for next scan. Again all this only if CQD above is ON or SYSTEM.

Also includes a fix where SMALL_SCANNER would be turned on for MDAM scan because the compiler for MDAM is not polulating the expected number of rows returned.

Results of using small scanner on MDAM when it make sense showed a 1.39X speed improvement...

    • -4
    • +23
    /core/sql/comexe/ComTdbHbaseAccess.h
Using getDefaultClass is better practice that initial implementation looking at NULL string. Also added improvement suggested about checking for DEFAULT_CURRENT. This is indeed good addition...

fix EXPECTED056 to remove the workaround introduced with PR340

Merge branch 'master' of github.com:apache/incubator-trafodion into jira1863

Fix jira1863. Pushdown V2 is disabled on columns with non null default as we are resolving default at select time and hbase don t know about the default values.

Implement TRAFODION-1420 Use ClientSmallScanner for small scans to improve perdormance Hbase implements an optimization for small scan (defined as scanning less than a data block ie 64Kb) resulting in 3X performance improvement. The underlying trick is about cutting down RPC calls from 3 (OpenScan/Next/Close) to 1, and use pread stateless instead of seek/read state-full and locking method to read data. This JIRA is about improving the compiler who can be aware if a scan will be acting on single data block (small) or not, and pass this data to executor so that it can use the right parameter for scan. (scan.setSmall(boolean)). reference: https://issues.apache.org/jira/browse/HBASE-9488 https://issues.apache.org/jira/browse/HBASE-7266

    • -14
    • +2
    /core/sql/comexe/ComTdbHbaseAccess.cpp
    • -3
    • +4
    /core/sql/executor/HBaseClient_JNI.cpp
    • -1
    • +1
    /core/sql/regress/executor/DIFF013.KNOWN
    • -14
    • +15
    /core/sql/regress/executor/EXPECTED013.SB
    • -0
    • +453
    /core/sql/regress/executor/EXPECTED131
  1. … 13 more files in changeset.
Merge branch 'master' of github.com:apache/incubator-trafodion into predicatePushdownV2

disable core/TESTRTS as it has a random bug causing it to fail with core dump. See trafodion JIRA 1771 for this issue.

    • -1
    • +1
    /core/sql/regress/tools/runregr_core.ksh
Merge branch 'master' of github.com:apache/incubator-trafodion into predicatePushdownV2

fix logical issue introduced after the rework on isAddedColumnWithNonNullDefault (check for ITM_BASECOLUMN or ITM_INDEXCOLUMN before checking if it is not an added column with non null default).

Merge branch 'master' of github.com:apache/incubator-trafodion into predicatePushdownV2

    • -53
    • +88
    /core/sql/sqlcomp/nadefaults.cpp
following code review outcome on pull request 255: - for all files, fix tab with white space - ExHbaseAccess.cpp, add comment - Initialize nac in file GenPreCode.cpp, line 11938 and 11417 - Remove checks for ITM_REFERENCE (used only in parser/binder). File GenPreCode.cpp - Remove isFirstAndLayer code in recursive function generating the predicate to push down in RPN - add comments in ValueDesc.cpp - fix indentation in HTableClient.java - Fix 4 buffer overrun vulnerability in GenExplain.cpp - create a reusable isAddedColumnWithNonNullDefault in ValueId class. (ValueDesc.cpp and .h)

    • -33
    • +34
    /core/sql/executor/ExHbaseAccess.cpp
    • -262
    • +245
    /core/sql/generator/GenExplain.cpp
    • -412
    • +305
    /core/sql/generator/GenPreCode.cpp
    • -18
    • +18
    /core/sql/regress/executor/EXPECTED140
    • -15
    • +15
    /core/sql/sqlcomp/nadefaults.cpp
Fix issue where optimization on key column addition should be turned off for MDAM scans, and update EXPECTEDTESTRTS to showcase the new value expected as byte read showing a 53% improvement over previous code

    • -695
    • +695
    /core/sql/regress/core/EXPECTEDRTS
Merge branch 'master' of github.com:apache/incubator-trafodion into predicatePushdownV2

Merge branch 'master' of github.com:apache/incubator-trafodion into predicatePushdownV2

    • -70
    • +124
    /core/sql/optimizer/ValueDesc.cpp
    • -0
    • +84
    /core/sql/regress/seabase/EXPECTED010
First commit for advanced predicate pushdown feature (also known as pushdown V2) associated JIRA TRAFODION-1662 Predicate push down revisited (V2). The JIRA contains a blueprint document, useful to understand what the code is supposed to do. This code is enabled using CQD hbase_filter_preds '2', and bypassed otherwise. Except for the change implemented in ValueDesc.cpp that is a global bug fix whereValueIdSet are supposed to contain set of valueID ANDed together, and should not contain any ValueID with operator ITM_AND.

    • -47
    • +45
    /core/sql/executor/ExHbaseAccess.cpp
    • -3
    • +281
    /core/sql/generator/GenExplain.cpp
    • -21
    • +566
    /core/sql/generator/GenPreCode.cpp
    • -0
    • +911
    /core/sql/regress/executor/EXPECTED140
    • -0
    • +42
    /core/sql/regress/executor/FILTER140
    • -0
    • +96
    /core/sql/regress/executor/TEST140
    • -188
    • +266
    /core/sql/regress/seabase/EXPECTED010
    • -10
    • +12
    /core/sql/regress/seabase/EXPECTED011
    • -77
    • +84
    /core/sql/regress/seabase/EXPECTED016