Context.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-3280] Reduce path length in Trafodion for improved performance and scalability

This PR fixes the following issues to improve performance.

1. When the session is dropped for the default context, we were unnecessarily

dropping HBase connections.

2. There were 2 HBase connections per mxosrvr - one for the default context and another

for repository context. Now it has been changed to one HBase connection per process.

  1. … 4 more files in changeset.
TRAFODION - 3218 User still has privilege after user's role has been revoked ...

Partial support for column level privileges with QI support for:

column select

column insert

column references

column update

Also, as part of this, updated privilege code in a couple of areas:

Changed object caching code in NATable and NARoutine to store all privileges

assigned to the object when the object is cached (privDescs_). During the load

operation, the code creates bitmaps (privInfo_) for the current user. Privilege

checks are performed against the user bitmaps (privInfo_). This is in

anticipation for some performance updates when connecting to Trafodion (mxosrvr)

with different users.

Change getRoleList to include the roleID and the granteeID that granted the

privilege. The grantee can be a user or a role.

When a privilege is revoked from a role, send QI keys for every user that has

been granted to role.

  1. … 40 more files in changeset.
  1. … 11 more files in changeset.
Address review comments

  1. … 5 more files in changeset.
Support to provide a locking mechanism for LOB insert/update operations

  1. … 28 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or errors in

all the utility commands.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

  1. … 14 more files in changeset.
[TRAFODION-2205] invalid char at create schema when authorization name is long

There is code that converts the user ID to its username. The buffer size

requested was not big enough to hold the return value. In addition, the buffer

size check was returning an error but did not add an error to the ComDiags area;

therefore the returned error was ignored.

- Changed max len in calls to getAuthNameFromAuthID to the correct size.

- Set up the Diags area when buffer size is too small so the error is reported

correctly, returns error 20235:

"Error returned while converting auth ID to auth name, status: xx ID: xx."

Also removed redundant methods:

- getDBUserNameFromID - calls getAuthNameFromID instead

- getDBUserIDFromName - calls GetAuthIDFromName instead

  1. … 5 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

Removed the need to allocate ComDiagsArea in advance for load command

by refactoring the ExeCliInterface::executeImmediate and

ExeCliInterface::executeImmediateExec methods.

Changes as per the review comments are also made in this commit

  1. … 16 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or error in

all the utility commands except load. In case of load, the ComDiagsArea

is allocated in advance to report error rows count.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

[TRAFODION-3017] Simplify the hive client access in Trafodion

Hive Client functions are now moved to a new file HiveClient_JNI.h and

HiveClient_JNI.cpp. Most of the HiveClient functions are static functions

allowing to use HiveClient in Trafodion with ease.

  1. … 33 more files in changeset.
[TRAFODION-2853] memory leak of ComDiagsArea in CmpContext heap of mxosrvr

Fixes for the regression failures seen with b97982c4494e078c5de2d883442d86265f24dadc

This includes the change to report the error at the time of compilation

for invoke, showddl commands. Earlier errors were ignored during

prepare time and reported only at the time of execute for these commands

  1. … 15 more files in changeset.
[TRAFODION-2853] memory leak of ComDiagsArea in CmpContext heap of mxosrvr

The ComDiagsArea is allocated in many places and from different heaps in Trafodion, making it difficult to

detect the source of the leak. Hence, a different approach is taken to fix this issue.

Currently, ComDiagsArea is allocated in many places unconditionally even when SQL statement completes

execution without any error or warnings. Then it is deallocated. Changed this strategy and

allocate ComDiagsArea only when there is an error or warning while compiling or executing the SQL statement.

This should help the product in two ways

1) To reduce the pathlength. The smaller query execution would benefit the most by chopping of few more microseconds.

2) Reduce the memory growth due to leaked ComDiagsArea

  1. … 24 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 41 more files in changeset.
Merge [TRAFODION-2886] PR-1380 fix the nullpointer error scanned by TScanCode

  1. … 4 more files in changeset.
fix the nullpointer error scaned by TScanCode

  1. … 20 more files in changeset.
[TRAFODION-2888] Streamline setjmp/longjmp concepts in Trafodion

First set of changes to streamline setjmp/longjmp.

a) Removed the setjmp in heap management within Trafodion.

b) Removed obsolete code related to No-wait operation concepts in SQL

  1. … 30 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 4. Remove legacy/obsolete pragmas

  1. … 392 more files in changeset.
[TRAFODION-2739] RMS semaphore handling need to log the error for easy resolution of the issue

RMS semaphore handling had the following:

1) Passed in parameters that are no longer used. It had these parameters for legacy reasons

2) Was returning errno as short as return code. This might have caused unexplainable behavior

at times.

3) The caller was aborting for non-zero return value, but the return value is not easily visible.

All the above problems have been fixed in this PR.

  1. … 20 more files in changeset.
Merge branch 'master' of git://git.apache.org/incubator-trafodion into bmo_memory_quota

Conflicts:

core/sql/cli/Globals.cpp

core/sql/executor/cluster.cpp

core/sql/regress/hive/EXPECTED009

core/sql/regress/seabase/EXPECTED010

core/sql/regress/seabase/EXPECTED011

core/sql/regress/seabase/EXPECTED016

  1. … 20 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
Following changes are done in BMO memory quota

1) Enabled memory quota per node. The CQD BMO_MEMORY_LIMIT_PER_NODE

(renamed from EXE_MEMORY_LIMIT_PER_CPU) is set to 10240 MB by default.

Old attribute Old value Renamed Attribute New value

EXE_MEMORY_LIMIT_PER_CPU 0 BMO_MEMORY_LIMIT_PER_CPU 10240

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 25

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 25

EXE_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 10 BMO_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 25

EXE_MEMORY_LIMIT_LOWER_BOUND_SORT 10 BMO_MEMORY_LIMIT_LOWER_BOUND_SORT 200

2) Changes in EXPLAIN

Estimated memory per node for all BMOs at ROOT operator

Estimated memory per instance for every BMO operator

Memory quota per instance for every BMO operator

3) BMO TDB contains the memory quota per esp instance now.

4) Root TDB now contains the limit per node and estimated memory per node.

This can be used by WMS to change the memory allocation during

runtime without compilation. - Not yet implemented.

4) Added a CQD BMO_MEMORY_LIMIT_UPPER_BOUND to gap the memory

consumed by BMO by the same queries with less number of

bmos.

5) The unused memory quota is yielded to other fragments in the process

also.

6) Removed the code to limit the ESPs from being assigned to a fragement

based on the BMO memory quota.

7) Added a new CQD BMO_MEMORY_ESTIMATE_RATIO_CAP to gap the memory

estimate skew by any one BMO operator to 0.7.

8) To disable the memory quota per node, set BMO_MEMORY_LIMIT_PER_NODE to 0.

9) This memory quota is distributed proportionally based on the estimated memory

taking into consideration the number of bmo instances per operator and

the number of nodes available in the cluster to host these instances.

Hence, this memory quota should be valid in multi-fragments independent of the

number of fragments in an ESP.

10) Removed the CQD EXE_MEMORY_LIMIT_NONBMOS_PERCENT and EXE_MEMORY_RESERVED_FOR_MXOSRVR_IN_MB

11) Fixed BMO stats WM to be at least the allocated memory.

12) Changed the sort operator to account the bmo memory correctly.

(cherry picked from commit ba19c04a58890fdd845b03f8d915abdd487b6407)

Conflicts:

core/sqf/src/seatrans/hbase-trx/src/main/java/org/apache/hadoop/hbase/coprocessor/transactional/SplitBalanceHelper.java

core/sql/cli/Context.cpp

core/sql/executor/ex_frag_rt.cpp

core/sql/executor/ex_sort.cpp

core/sql/regress/executor/EXPECTED131

core/sql/regress/executor/EXPECTED140

core/sql/regress/hive/EXPECTED009

core/sql/regress/hive/EXPECTED030

core/sql/regress/hive/FILTER009

core/sql/regress/seabase/EXPECTED010

core/sql/regress/seabase/EXPECTED011

core/sql/regress/seabase/EXPECTED016

core/sql/sqlcomp/DefaultConstants.h

  1. … 44 more files in changeset.
[TRAFODION-2698] Ensure RMS can be disabled properly

  1. … 2 more files in changeset.
[TRAFODION-2695] SSMP process ($ZSMxxx) sees too many opens from the master process

Mxosrvr/Any master process opens a connection to mxssmp for the following:

1) For get statistics command

Managed via ssmpManager_ in the context. This can have connections to all the

ssmps in the cluster

2) To cancel a query

Was managed via cbServer_ in ExCancelTcb. This connection was expected to go away

when the cancel is passed on the mxssmp.

3) To Send query started /Query finished message

Was managed via cbServer_ in ContextCli. If this cbServer_ was taken up already by

the query with query started message pending, every statement being executed would have

created a connection to ssmp and managed via cbServer_ in ex_root_tcb of the query

In an ssmp core there were many opens from a mxosrvr. To avoid many open connections,

all the ssmp connections are now managed via ssmpManager_ in the default context.

This connection is shared by multiple message streams by increasing the no-wait depth.

  1. … 9 more files in changeset.
[TRAFODION-2420] RMS Enhancements

Added yet another offender feature to list query ids that has a total

IO time for any storage engine opertor consuming longer than a given

number of seconds.

./offender -s se_offender

Will list the query ids along with the table name. SEE

$TRAF_HOME/export/limited-support-tools/LSO/README

The "Number of SQL Processes" counter is now made multi-fragment aware

and hence contains the actual number of ESPs used + 1 for master

process.

  1. … 7 more files in changeset.
[TRAFODION-2400] RMS enhancements

Removed FsDp2MsgStats and ExeDp2Stats

Introduced a new class ExeSEStats to cater the different

storaage engine supported by Trafodion.

The counters pertaining to storage engine were not getting

in GET STATISTICS FOR QID ... ACCUMULATED mode. Fixed this issuue

Removed references to obsolete Measuree concepts

  1. … 15 more files in changeset.
TRAFODION-2330 Using trafci, a select from a table succeeds even if the user does not have the priv

There is a problem when the session user changes in a mxosrvr process. The

existing compiler caches are not getting cleared so the new user will be

accessing the previous users' caches. This could lead to allowing someone

that does not have privileges to gain access to an object.

The change is to clear all caches during a session user change operation.

  1. … 2 more files in changeset.
TRAFODION-2327 Reduce I/O when loading objects into caches

For each authorization ID (user, role, or PUBLIC), a bitmap containing the

accumulated privileges (across all grantors) is stored with the object desc.

When the object desc is loaded into cache, the privilege bitmaps associated

with the current user, PUBLIC, and the current users' roles are extracted and

unioned together to calculate the final set of privileges. This unioned list

is used during privilege checking.

Today, an I/O is performed to retrieve the list of roles granted to the current

user for each object loaded into NATable and NARoutine cache. Since this list

does not change unless the current user changes (a new session with a different

user) or a grant/revoke role for the current user is performed, these extra

I/O's are not needed.

To remove the extra I/O's for each object, the list of roles will be stored in

the ContextCli. Therefore, this in-memory role list can be used instead of

rereading metadata.

This checkin creates two new CLI requests:

- GetRoleList - returns the list of roles associated with the user

If the list exists in ContextCli, it returns the stored values

If the list does not exist, it retrieves them from Metadata, stores

them and returns the values

- ResetRoleList - removes the list of roles from ContextCli

The first time GetRoleList is called in a session, the users' roles are

stored in ContextCli. They remain in memory until the session ends and

restarts as a different user, or another process grants or revokes a role

from the current user.

If another process revokes a role from the current user, a query invalidation

key is created. When the revoke role query invalidation key for the current

user is detected, ResetRoleList is called. The next time GetRoleList is called

an updated role list is retrieved from metadata and stored in ContextCli.

If another process grants a role to the current user, there could be two

outcomes. If the current user already has the privilege from another source

then nothing happens. If the current user does not have the privilege, then

one recompilation is attempted. Prior to performing the retry, code was

added to ResetRoleList. The recompilation then gets the latest role list and

either succeeds or fails depending on the granted privileges.

  1. … 16 more files in changeset.
Create NACollection objects on NAHeap. Avoid use of system heap as far as possible

  1. … 50 more files in changeset.
[TRAFODION-2129] Trafodion to avoid use of deprecated HBase APIs/Classes

Avoided the caching of Hbase table instances on the client side to avoid

compaction queue size build up while constantly loading the data

  1. … 11 more files in changeset.
Merge remote branch 'origin/master' into lobglobals_fix

Conflicts:

core/sql/cli/Context.cpp

core/sql/exp/ExpLOBaccess.cpp

  1. … 13 more files in changeset.