Clone
 

qchen <qifan.chen@hp.com> in Trafodion

ESP-colocation fixes (Ravisha and Qifan)

    • -0
    • +3
    /core/sql/executor/HBaseClient_JNI.cpp
    • -7
    • +11
    /core/sql/optimizer/OptPhysRelExpr.cpp
    • -603
    • +603
    /core/sql/regress/core/EXPECTEDRTS
    • -63
    • +63
    /core/sql/regress/seabase/EXPECTED016
Allow scan esps to take # of partitions.

    • -24
    • +33
    /core/sql/optimizer/OptPhysRelExpr.cpp
Squashed commit of the following:

commit 94e16c214c20c3ab74bf8bf7cef5dedc91285488

Author: qchen <qifan.chen@hp.com>

Date: Fri Jun 5 16:55:51 2015 +0000

port AS, turn of AS for now

commit 1311090436706506c9ce4e8d455c425d896812c8

Author: qchen <qifan.chen@hp.com>

Date: Fri Jun 5 16:54:55 2015 +0000

port AS, rework

commit 7377923f5ef036adcb07e50ec92aca2625ed3fe0

Author: qchen <qifan.chen@hp.com>

Date: Thu Jun 4 01:49:17 2015 +0000

AS porting

Change-Id: I6ba4349409cbee8d344cba97802febf9fec5c1ea

fix LP1420826 NJ off fails meta query compilation

Change-Id: Ic443f6c9a4d25a84241fc66acd7ff54e00ab1b46

port skew buster to Trafodion

1. add simplified TEST062

2. reuse cached partitioning expression when only doVarCharCast

is the same as the when the expression is created in

TableHashPartitioningFunction::createPartitioningExpressionImp().

3. rework

4. fix a time monitor bug reporting incorrect processor time (CPU time

computed from clock() calls)

5. comment out the assert on the size of NATable cache not decreasing.

This is to fix regression failure with seabase/TEST020. Selva will check

in a formal fix that allocates space for NATable objects from a single

heap, and then reenables this assert check.

Change-Id: I9eeee4f36ba8678e90e0ac68a85bfc733599d932

    • -1
    • +131
    /sql/executor/ex_split_bottom.cpp
    • -1
    • +110
    /sql/generator/GenRelEnforcer.cpp
    • -2
    • +270
    /sql/optimizer/OptPhysRelExpr.cpp
  1. … 12 more files in changeset.
fix LP 1347930

The main idea of the fix is as follows.

1. MC stats for compilation usage becomes multi-interval version,

for key columns in a HBase table

2. MC stats stored in histogram cache now contains the

multi-interval histogram.

Change-Id: Ief2f0746d154b714a9b780cce55ff536406b8fa1

disable compGeneral/TEST042

Change-Id: I12708e3fe43efc163913e09ba29f0095ba747a3a

fix lp 1392099 (cqd hive_max_varchar_length)

There are two contribuing factors.

1. A fully specified partition requirement (partition of 1) in

FileScan::synthHiveScanPhysicalProperty() is not recognized

and a modification of the ESPs renders the partition requirement

not satisfiable.

2. The reduction in CQD value for hive_max_varchar_length reduces

the dop from 5 to 1. In Exchange::createContextForChild(), a dop of

1 renders the parallel version of plan infeasible (see line near 2004

on test "rppForMe->getCountOfPipelines() > 1").

The fix is to address issue #1.

Added a safety check for ApproximatelyNPartitions and improvement

for hive sampling logic.

remove update to cg/TEST011

Change-Id: Idbf24be0b00b336dbac93afdddb6896ed0937e23

    • -17
    • +33
    /sql/optimizer/OptPhysRelExpr.cpp
fix 1361754: missing sort for fastTrafInsert

Change-Id: Ib7d768cba5b89ad44f5423cd9dd499fd537eaf66

fix LP1383581:serial plan generated for salt table

Change-Id: I36cbf70119218dcce1826a2cf87306e66ded1d64

remove WITH DIVISION clause

Change-Id: I986fe56a8559f88f2781adf700933d5b03c9e697

fix cardinality errors for better DoP for OE

Problems fixed are as follows.

1. TableAnalysis::getLocalPredsOnPrefixOfList() prematurely breaks

out of the loop to collect predicates on key columns.

2. TableDesc::getSaltColumnAsSet() does not return the result in

the form of base columns valueIds.

3. When the dop is estimated lower than the low bound of ESPs, the

low bound value is used.

4. SimpleFileScanOptimizer::computeSingleSubsetSize() incorrectly

applies the computed predicates on salted column which reduces the

subset size by CQD HIST_NO_STATS_UEC (default to 2). A subset size

specifies the # of rows processed.

Change-Id: I85bebf0d31d8f3e2db5bfe7c00cb4467b53308ac

    • -2
    • +8
    /sql/optimizer/SimpleScanOptimizer.cpp
    • -189
    • +188
    /sql/regress/seabase/EXPECTED010
Index-join scan trimming heuristics rework II

Change-Id: I9cf63be7967a012559e27dc4ca950bb28b8ccd3b

fix #1388277 (optimizer core)

Change-Id: I24a662c86e93451afc514c24210d415811369773

fix #lp1384430: full sort order for plan1 of NJ

Change-Id: I7f3162af34d16bc5e801305c24438d098a9b09ef

    • -70
    • +72
    /sql/optimizer/OptPhysRelExpr.cpp
rework fix to move global variabls to optDefaults

Change-Id: I70303eb6c2587fe7c151e0737977d2e1802054cf

delete three new data members in ~OptDefaults

Change-Id: Ibd757043e780bb1dddcaebdf70f7fdcd16399f7a

Move three global variables into opDefauls.

Change-Id: I1588c746e1a61418c19d4ded13f5c201c581d0bd

DoP adjustment for small queries. Rework

Change-Id: Idd70130ca07dc4c82eb2a83422093789fd510c81

dop check-in rework.

Change-Id: Idb21dfdbdff76228f9634abc0d1feac6e9e1810b

    • -11
    • +22
    /sql/optimizer/OptPhysRelExpr.cpp
Squashed commit of the following:

commit 221d4199001b3f06d5629b82ed1281bdfb95f043

Author: qchen <qifan.chen@hp.com>

Date: Mon Sep 8 19:32:05 2014 +0000

DoP: use a better version of resource estimator

Change-Id: Idf6ea5caa7e4915c65b4d54f58d7e483c26871f4

commit 75c92f255ad3841102757f6188d749db42eaedf3

Author: qchen <qifan.chen@hp.com>

Date: Fri Sep 5 15:55:11 2014 +0000

do not add ESP partition requirement to partial GB leaf1 and leaf2.

commit f7c652adf2953d18f10da4a952f78d3815b5b3d4

Author: qchen <qifan.chen@hp.com>

Date: Fri Sep 5 15:20:40 2014 +0000

finalize

commit a419954b1528ca0d1f84c5298b806b46f561a713

Author: qchen <qifan.chen@hp.com>

Date: Wed Sep 3 23:45:13 2014 +0000

add a check on stats smart ptr before its use.

commit 8a2dcc24f0081d554ec3d69560c050f720da04dd

Author: qchen <qifan.chen@hp.com>

Date: Wed Sep 3 18:31:50 2014 +0000

partial gb root running in Master; use rowcount from stats for Hbase

in AppliedStatMan::getStatsForCANodeId().

commit 55f46ededa9fcb30d38d17be6692e1535b4a274c

Author: qchen <qifan.chen@hp.com>

Date: Fri Aug 29 17:46:00 2014 +0000

improve the logic to disable parallel GB partial root when rows < 5000

commit 743783720d030f38e50a065737c8c218fa47c6b5

Author: Ravisha Neelakanthappa <Ravisha.Neelakanthappa@hp.com>

Date: Wed Aug 20 17:20:09 2014 +0000

Fix for bug 1348317. Enable Adaptive Segmentation logic to

compute DoP based on resource estimation.

Change-Id: I7c2b0756854d5a86d8bde162dad064b4e924233c

Change-Id: Id68e1d8265e478f128ed61d1cabc4849873dae23

    • -37
    • +72
    /sql/optimizer/OptPhysRelExpr.cpp
  1. … 5 more files in changeset.
Fix Bug #1356523 and stats-split does not take place

The 1st bug is due to the lacking of a test on # salted columns and #

splits. The 2nd bug is due to compression on the leading primary key col.

Change-Id: I6a2594a41261f4325e81e334bf13fa228e38a113

fix bug 1343661(cleanup HBASE partitioning), bug 1347819 and bug 1343566

1343661: please refer to sql/sqlcomp/DefaultConstConstants.h for the

definition of CQD HBASE_PARTITIONING.

1347819: the change is in ExExeUtilLoad.cpp to disable HASH2 for fast load.

1343566: method NADefaults::getTotalNumOfESPsInCluster() now returns the

correct value if CQD PARALLEL_NUM_ESPS is set to an integer value.

Rework 1 to address Dave's review comments.

Rework 2 to address Khaled's review comments as follows. A Boolean flag

(isTrafLoadPrep_) is added to class BinWA to better control the type of

partitioning functions needed for the traf preparation step. When we are

binding all nodes, the flag is set to TRUE which instructs createNAFileSet()

not to create hash2.

Rework 3 to address Hans's review comments. If force to have hash2 and

the partitioning function in the cached table is range, do not return the

cached object.

Rework 4 to address seabase/TEST015 core, which is reported in bug

1349990. Bug 1349990 is fixed in this rework.

Change-Id: I9f886ce5361ff4ede16c592e8a3c4e5ca89d76b7

    • -53
    • +60
    /sql/optimizer/OptPhysRelExpr.cpp
    • -20
    • +20
    /sql/regress/seabase/EXPECTED015
    • -127
    • +127
    /sql/regress/seabase/EXPECTED016
    • -0
    • +146
    /sql/regress/seabase/EXPECTED018
  1. … 5 more files in changeset.
Enables HASH2 partitioning representation of salted tables + 2 reworks.

Change-Id: Id2c32fd99a18dea0c4384e49356e67502eaf4127

Closes-Bug: #1336450

    • -11
    • +18
    /sql/optimizer/OptPhysRelExpr.cpp
    • -45
    • +43
    /sql/regress/seabase/EXPECTED013
    • -153
    • +119
    /sql/regress/seabase/EXPECTED016
Modify the semantics of CQD MAX_ESPS_PER_CPU_PER_OP to become a factor in computing the max #ESPs per core. The #esps per node = CQD * cores per, node and the total number of ESPs per cluster is #esps per node * #nodes.

The CQD now takes a real number.

The work consolidates all references to the CQD to three new methods

NADefaults::getNumOfESPsPerNode(), NADefaults::getNumOfESPsPerNodeInFloat()

and NADefaults::getTotalNumOfESPsInCluster().

Change-Id: I03b29be7d6bcdb70385320a84445ed36ce968a80

Closes-Bugs: #1336452

    • -77
    • +19
    /sql/optimizer/OptPhysRelExpr.cpp
Fix the number of ESPs per node for seabase/TEST013 and TEST016

Change-Id: Ia52d9d9e59fffdd6db60bf4d11317451a792dd64

Code cleanup (06042014)

Change-Id: Ib7f2f39e42085e026ac1000e43048c9a185b6976

    • -77
    • +77
    /sql/regress/seabase/EXPECTED016