ex_sort.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-2888] Streamline setjmp/longjmp concepts in Trafodion

First set of changes to streamline setjmp/longjmp.

a) Removed the setjmp in heap management within Trafodion.

b) Removed obsolete code related to No-wait operation concepts in SQL

  1. … 30 more files in changeset.
Merge [TRAFODION-2758] PR-1250 Sort operator that uses TopN sort at runtime is not accuratelt determined at compile time

  1. … 9 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 4. Remove legacy/obsolete pragmas

  1. … 392 more files in changeset.
[TRAFODION-2758] Sort operator that uses TopN sort at runtime is not accurately determined at compile time

This commit includes fixes for these three issues. Please see JIRA for a description of each fix.

[TRAFODION-2758] Sort operator that uses TopN sort at runtime is not accurately determined at compile time

[TRAFODION-2759] Use of rowset with large memory leads to various errors at runtime

[TRAFODION-2760] hbase cache blocks is OFF for broad table with narrow index

[TRAFODION-2758]

comexe/ComTdbSort.cpp

comexe/ComTdbSort.h

executor/ex_sort.cpp

generator/GenExplain.cpp

generator/GenPreCode.cpp

generator/GenRelMisc.cpp

optimizer/RelExpr.cpp

[TRAFODION-2759]

bin/SqlciErrors.txt

generator/GenRelPackedRows.cpp

sqlcomp/DefaultConstants.h

sqlcomp/nadefaults.cpp

[TRAFODION-2760]

generator/GenRelScan.cpp

  1. … 11 more files in changeset.
lcov: commit #1

  1. … 333 more files in changeset.
[TRAFODION-2733] Provide an improved memory quota assignment for big memory operators (BMO)

Removed the 2nd sortHeap_ creation.

Ensured that CQD NUM_ESP_FRAGMENTS can take value upto 8. Removed the environment variable concept to set the number of esp fragments. Removed the CQDs ESP_NUM_FRAGMENTS_WITH_QUOTAS and ESP_MULTI_FRAGMENT_QUOTAS. Use the corresponding CQD ESP_NUM_FRAGMENTS and ESP_MUTLI_FRAGMENT instead (cherry picked from commit 72534e89633f3d4b8a61ab26d471aaa8b7f3e12a)

Conflicts:

core/sql/executor/ex_frag_rt.cpp

core/sql/executor/ex_frag_rt.h

core/sql/generator/GenRelMisc.cpp

  1. … 45 more files in changeset.
Merge branch 'master' of git://git.apache.org/incubator-trafodion into bmo_memory_quota

Conflicts:

core/sql/cli/Globals.cpp

core/sql/executor/cluster.cpp

core/sql/regress/hive/EXPECTED009

core/sql/regress/seabase/EXPECTED010

core/sql/regress/seabase/EXPECTED011

core/sql/regress/seabase/EXPECTED016

  1. … 20 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
Following changes are done in BMO memory quota

1) Enabled memory quota per node. The CQD BMO_MEMORY_LIMIT_PER_NODE

(renamed from EXE_MEMORY_LIMIT_PER_CPU) is set to 10240 MB by default.

Old attribute Old value Renamed Attribute New value

EXE_MEMORY_LIMIT_PER_CPU 0 BMO_MEMORY_LIMIT_PER_CPU 10240

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 25

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 25

EXE_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 10 BMO_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 25

EXE_MEMORY_LIMIT_LOWER_BOUND_SORT 10 BMO_MEMORY_LIMIT_LOWER_BOUND_SORT 200

2) Changes in EXPLAIN

Estimated memory per node for all BMOs at ROOT operator

Estimated memory per instance for every BMO operator

Memory quota per instance for every BMO operator

3) BMO TDB contains the memory quota per esp instance now.

4) Root TDB now contains the limit per node and estimated memory per node.

This can be used by WMS to change the memory allocation during

runtime without compilation. - Not yet implemented.

4) Added a CQD BMO_MEMORY_LIMIT_UPPER_BOUND to gap the memory

consumed by BMO by the same queries with less number of

bmos.

5) The unused memory quota is yielded to other fragments in the process

also.

6) Removed the code to limit the ESPs from being assigned to a fragement

based on the BMO memory quota.

7) Added a new CQD BMO_MEMORY_ESTIMATE_RATIO_CAP to gap the memory

estimate skew by any one BMO operator to 0.7.

8) To disable the memory quota per node, set BMO_MEMORY_LIMIT_PER_NODE to 0.

9) This memory quota is distributed proportionally based on the estimated memory

taking into consideration the number of bmo instances per operator and

the number of nodes available in the cluster to host these instances.

Hence, this memory quota should be valid in multi-fragments independent of the

number of fragments in an ESP.

10) Removed the CQD EXE_MEMORY_LIMIT_NONBMOS_PERCENT and EXE_MEMORY_RESERVED_FOR_MXOSRVR_IN_MB

11) Fixed BMO stats WM to be at least the allocated memory.

12) Changed the sort operator to account the bmo memory correctly.

(cherry picked from commit ba19c04a58890fdd845b03f8d915abdd487b6407)

Conflicts:

core/sqf/src/seatrans/hbase-trx/src/main/java/org/apache/hadoop/hbase/coprocessor/transactional/SplitBalanceHelper.java

core/sql/cli/Context.cpp

core/sql/executor/ex_frag_rt.cpp

core/sql/executor/ex_sort.cpp

core/sql/regress/executor/EXPECTED131

core/sql/regress/executor/EXPECTED140

core/sql/regress/hive/EXPECTED009

core/sql/regress/hive/EXPECTED030

core/sql/regress/hive/FILTER009

core/sql/regress/seabase/EXPECTED010

core/sql/regress/seabase/EXPECTED011

core/sql/regress/seabase/EXPECTED016

core/sql/sqlcomp/DefaultConstants.h

  1. … 44 more files in changeset.
Merge branch 'master' of git://git.apache.org/incubator-trafodion into trafodion-2420_4

Conflicts:

core/sql/executor/ex_sort.cpp

core/sql/executor/ex_sort.h

  1. … 2 more files in changeset.
fix to address review comments. NADELETE used where needed.

[TRAFODION-2604] sort merge phase memory pool improvements

  1. … 4 more files in changeset.
[TRAFODION-2420] RMS enhancements

New counters introduced as part of BMO

InterimRowCount - Counter to keep track intermediate rows

produced by hash join and for intermediate

runs in sort

phase - BMO Phase for hash-join and sort

get statistics for statement s1, options 'cs' now works.

Removed attaching BMO stats to Non-root Paritial and leaf partition hash group by operators.

These operators are non-blocking BMOs.

Introduced 3 phases for hash grby operator to be visible in RMS Stats

options 'sl' now works with get statistics for qid.

The relevant new counters are now part of accumulated statistics type.

Extended the support upto 512 opeartors from 256 operators in RMS

Fixed both Type 2 and mxosrvr code to obtain all the externalized counters in accumualted stats correctly.

  1. … 27 more files in changeset.
[TRAFODION-2420] RMS enhancements

Improved BMO operation accounting. Two new counters are added

a) scratchIOSize - Size in KB of the scratch IO

b) scratchIOMaxTime - Time in microseconds taken by any long pole

ESP instance to do scratch IO operations

The ScrBufSize, ScrBufRead and ScrBufWritten are no longer displayed

in the formatted get statistics command outputs.

Reduced the default value of CQD HBASE_NUM_CACHE_ROWS_MAX to 1024 from

10000 to reduce stress on the client JVM memory.

Disabled the memory pressure triggering in the BMO operators by increasing

the threshold GEN_MEM_PRESSURE_THRESHOLD to 10000 from 100. The memory

pressure detection code was inadvertently enabled when the cap to limit

memory pressure constant to 100 was removed.

Memory pressure triggers in BMO operators will be enabled later when

we understand the memory pressure detection better.

  1. … 14 more files in changeset.
[TRAFODION-2259] Add TopN stats counter and realign BMO stats.

  1. … 7 more files in changeset.
fixes reuse of allocated pools.

  1. … 1 more file in changeset.
Additional minor change to enable/disable topn.

  1. … 2 more files in changeset.
additional fix to address regression test failure.

[TRAFODION-2326] additional change.

[TRAFODION-2326] feedback comments fixed.

[TRAFODION-2326] TopN sort reuse of released tuples

  1. … 2 more files in changeset.
[TRAFODION-2259] changes from feedback, drive TopN sort from GETN

  1. … 14 more files in changeset.
[TRAFODION-2259] changes from feedback, drive TopN sort from GETN

  1. … 14 more files in changeset.
[TRAFODION-2259] TopN sort changes. This includes first of changes related to sort implementation in executor. cqd gen_sort_topn_size 'N' forces sort to use topn. Subsequent changes in compiler will be able to push down topn to sort.

  1. … 13 more files in changeset.
[TRAFODION-2259] TopN sort changes. This includes first of changes related to sort implementation in executor. cqd gen_sort_topn_size 'N' forces sort to use topn. Subsequent changes in compiler will be able to push down topn to sort.

  1. … 13 more files in changeset.
Scratchfile creation cleanup and new code to use CQD SCRATCH_DISKS to specify scratch disk locations for overflow. Removed references to NSK volume patterns and validation routines that expected volume/subvolume format. Added new validation code for linux path names. Cleaned up and removed STFS_LOCATION envvar usage from ms.env and STFS layer. Added a new method that can be called to tell STFS to honor the CQD specification over the envvar specification.

  1. … 27 more files in changeset.
Part 2 of changes to update license text in Trafodion source

This check-in updates license text in all source files in the

core/sql directory. Remaining directories will be done in

later check-ins.

  1. … 3147 more files in changeset.
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.