Globals.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Changes as per review comments of PR 1820

[TRAFODION-3280] Reduce path length in Trafodion for improved performance and scalability

This PR fixes the following issues to improve performance.

1. When the session is dropped for the default context, we were unnecessarily

dropping HBase connections.

2. There were 2 HBase connections per mxosrvr - one for the default context and another

for repository context. Now it has been changed to one HBase connection per process.

  1. … 4 more files in changeset.
[TRAODION-3280] Reduce path length in Trafodion for improved performance and scalability

cli_globals::currContext() is now optimized for default context.

  1. … 1 more file in changeset.
[TRAFODION-2821] Trafodion core code base needs to be thread safe

Java exceptions thrown while calling the java methods from JNI in

Trafodion are stored in the current context. However in a multi-threaded

ESP environment, these exceptions should be stored in thread specific

variable to enable error handling to be thread safe. Otherwise, the JNI

errors could be overwritten by the JNI errors from another thread.

Also streamlined JNI error handling by extending the getExceptionDetails()

methods to log the errors also.

Incorporated error handling in SequenceFileReader JNI methods.

  1. … 22 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 15 more files in changeset.
[TRAFOFION-2731] CodeCleanup: Removed the obsolete code in NAMemory and CliGlobals classes

  1. … 8 more files in changeset.
[TRAFODION-2888] Streamline setjmp/longjmp concepts in Trafodion

First set of changes to streamline setjmp/longjmp.

a) Removed the setjmp in heap management within Trafodion.

b) Removed obsolete code related to No-wait operation concepts in SQL

  1. … 30 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 4. Remove legacy/obsolete pragmas

  1. … 392 more files in changeset.
lcov: commit #1

  1. … 333 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 2: Remove obsolete code

This phase handles the following:

-- removed files:

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

exp/exp_space.h

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

sqlshare/catapirequest.*

-- removed defines and code referencing them:

-- NA_STD_NAMESPACE

-- NA_NO_CMPCONTEXT

-- NA_CATMAN_SIM, NA_CATMAN_SIM_FS

-- common/purify.h

-- DONT_USE_MATH_H

-- NT_PORT

-- NA_MSVC

-- NA_NO_FRIENDS_WITH_TEMPLATE_REFS

-- NA_FLEXBUILD

-- removed multiple obsolete sqlci features and syntax:

(report writer, MACL, Help, Simulators, Utils, MXCS mode, Help,

and few others).

-- removed following files in sqlci dir:

CSInterface.h

CharSetConstants.cpp

CharSetConstants.h

MsgCat.cpp

MsgCat.h

MxciEHCallBack.cpp

MxciEHCallBack.h

RWInterface.cpp

RWInterface.h

SqlciCSCmd.cpp

SqlciCSCmd.h

SqlciCSSimulator.cpp

SqlciHelp.cpp

SqlciRWCmd.cpp

SqlciRWCmd.h

SqlciRWSimulator.cpp

SqlciUsage.cpp

SqlciUtil.cpp

SqlciUtil.h

UtilInt.cpp

UtilInt.h

UtilMsg.cpp

UtilMsg.h

immudefs.cpp

immudefs.h

  1. … 85 more files in changeset.
Merge remote branch 'origin/master' into ansharma_t23cleanup_br

  1. … 7 more files in changeset.
code cleanup, commit #1

  1. … 128 more files in changeset.
Ensured that CQD NUM_ESP_FRAGMENTS can take value upto 8. Removed the environment variable concept to set the number of esp fragments. Removed the CQDs ESP_NUM_FRAGMENTS_WITH_QUOTAS and ESP_MULTI_FRAGMENT_QUOTAS. Use the corresponding CQD ESP_NUM_FRAGMENTS and ESP_MUTLI_FRAGMENT instead (cherry picked from commit 72534e89633f3d4b8a61ab26d471aaa8b7f3e12a)

Conflicts:

core/sql/executor/ex_frag_rt.cpp

core/sql/executor/ex_frag_rt.h

core/sql/generator/GenRelMisc.cpp

  1. … 45 more files in changeset.
Merge branch 'master' of git://git.apache.org/incubator-trafodion into bmo_memory_quota

Conflicts:

core/sql/cli/Globals.cpp

core/sql/executor/cluster.cpp

core/sql/regress/hive/EXPECTED009

core/sql/regress/seabase/EXPECTED010

core/sql/regress/seabase/EXPECTED011

core/sql/regress/seabase/EXPECTED016

  1. … 20 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
Following changes are done in BMO memory quota

1) Enabled memory quota per node. The CQD BMO_MEMORY_LIMIT_PER_NODE

(renamed from EXE_MEMORY_LIMIT_PER_CPU) is set to 10240 MB by default.

Old attribute Old value Renamed Attribute New value

EXE_MEMORY_LIMIT_PER_CPU 0 BMO_MEMORY_LIMIT_PER_CPU 10240

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 25

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 25

EXE_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 10 BMO_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 25

EXE_MEMORY_LIMIT_LOWER_BOUND_SORT 10 BMO_MEMORY_LIMIT_LOWER_BOUND_SORT 200

2) Changes in EXPLAIN

Estimated memory per node for all BMOs at ROOT operator

Estimated memory per instance for every BMO operator

Memory quota per instance for every BMO operator

3) BMO TDB contains the memory quota per esp instance now.

4) Root TDB now contains the limit per node and estimated memory per node.

This can be used by WMS to change the memory allocation during

runtime without compilation. - Not yet implemented.

4) Added a CQD BMO_MEMORY_LIMIT_UPPER_BOUND to gap the memory

consumed by BMO by the same queries with less number of

bmos.

5) The unused memory quota is yielded to other fragments in the process

also.

6) Removed the code to limit the ESPs from being assigned to a fragement

based on the BMO memory quota.

7) Added a new CQD BMO_MEMORY_ESTIMATE_RATIO_CAP to gap the memory

estimate skew by any one BMO operator to 0.7.

8) To disable the memory quota per node, set BMO_MEMORY_LIMIT_PER_NODE to 0.

9) This memory quota is distributed proportionally based on the estimated memory

taking into consideration the number of bmo instances per operator and

the number of nodes available in the cluster to host these instances.

Hence, this memory quota should be valid in multi-fragments independent of the

number of fragments in an ESP.

10) Removed the CQD EXE_MEMORY_LIMIT_NONBMOS_PERCENT and EXE_MEMORY_RESERVED_FOR_MXOSRVR_IN_MB

11) Fixed BMO stats WM to be at least the allocated memory.

12) Changed the sort operator to account the bmo memory correctly.

(cherry picked from commit ba19c04a58890fdd845b03f8d915abdd487b6407)

Conflicts:

core/sqf/src/seatrans/hbase-trx/src/main/java/org/apache/hadoop/hbase/coprocessor/transactional/SplitBalanceHelper.java

core/sql/cli/Context.cpp

core/sql/executor/ex_frag_rt.cpp

core/sql/executor/ex_sort.cpp

core/sql/regress/executor/EXPECTED131

core/sql/regress/executor/EXPECTED140

core/sql/regress/hive/EXPECTED009

core/sql/regress/hive/EXPECTED030

core/sql/regress/hive/FILTER009

core/sql/regress/seabase/EXPECTED010

core/sql/regress/seabase/EXPECTED011

core/sql/regress/seabase/EXPECTED016

core/sql/sqlcomp/DefaultConstants.h

  1. … 44 more files in changeset.
Merge PR 1188 [TRAFODION-2695] SSMP process () sees too many opens from the master process

  1. … 2 more files in changeset.
[TRAFODION-2695] SSMP process ($ZSMxxx) sees too many opens from the master process

Mxosrvr/Any master process opens a connection to mxssmp for the following:

1) For get statistics command

Managed via ssmpManager_ in the context. This can have connections to all the

ssmps in the cluster

2) To cancel a query

Was managed via cbServer_ in ExCancelTcb. This connection was expected to go away

when the cancel is passed on the mxssmp.

3) To Send query started /Query finished message

Was managed via cbServer_ in ContextCli. If this cbServer_ was taken up already by

the query with query started message pending, every statement being executed would have

created a connection to ssmp and managed via cbServer_ in ex_root_tcb of the query

In an ssmp core there were many opens from a mxosrvr. To avoid many open connections,

all the ssmp connections are now managed via ssmpManager_ in the default context.

This connection is shared by multiple message streams by increasing the no-wait depth.

  1. … 9 more files in changeset.
[TRAFODION-2628] Runtime work for elasticity

Adding a CPU array, similar to what's used in the compiler, to the CLI

globals. Make sure that when the compiler didn't specify a CPU (node)

id for an ESP, don't pick a node id that no longer exists.

Made a similar change for the SSMP.

Also got rid of some dead or now obsolete code.

  1. … 6 more files in changeset.
[TRAFODION-2400] RMS enhancements

Removed FsDp2MsgStats and ExeDp2Stats

Introduced a new class ExeSEStats to cater the different

storaage engine supported by Trafodion.

The counters pertaining to storage engine were not getting

in GET STATISTICS FOR QID ... ACCUMULATED mode. Fixed this issuue

Removed references to obsolete Measuree concepts

  1. … 15 more files in changeset.
Part 2 of changes to update license text in Trafodion source

This check-in updates license text in all source files in the

core/sql directory. Remaining directories will be done in

later check-ins.

  1. … 3147 more files in changeset.
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.