ExExeUtilLoad.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
iChanges based on review comments

  1. … 14 more files in changeset.
Fixees to support stroring Udr libraries s Blobs.

  1. … 45 more files in changeset.
[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

  1. … 28 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

The CQD USE_LIBHDFS is now made effective for updateLOB and extract LOB utility

commands

  1. … 2 more files in changeset.
Merge branch 'master' of github.com:apache/trafodion into trafodion_2917_1

  1. … 6 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

Ensured that the CQD USE_LIBHDFS 'ON' works as expected via the

older implementation

  1. … 2 more files in changeset.
TRAFODION-3086 additional fixes

  1. … 7 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

This feature is enabled by default. To disable, set a variable USE_LIBHDFS=1 in

$TRAF_HOME/etc/ms.env and restart the trafodion cluster.

This feature includes the following:

1. Uses single FSDataInputStream for each LOB column in a query as

opposed to the opening the hdfs file for every row.

2. Uses FSDataOutputStream to write the lob data but closes it

immediately to allow concurrent writes to the hdfs file. HDFS supports

a single writer at a time. Need to conform if multiple writes can

be done without the need for RMS lock feature.

3. Improved error messaging that displays the java exception stack to the

end user.

4. LOB worker threads are no longer created

  1. … 12 more files in changeset.
Address review comments

  1. … 5 more files in changeset.
Support to provide a locking mechanism for LOB insert/update operations

  1. … 28 more files in changeset.
Merge remote branch 'origin/pr/1538/head' into merge_1538

  1. … 2 more files in changeset.
Removing unneeded CLI call to get lob location and instead using the CQD value - LOB_STORAGE_LOCATION to get/save it in Tdb for use later at runtime.

  1. … 3 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or errors in

all the utility commands.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

  1. … 14 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

Changes to avoid allocation of ComDiagsArea in some more places unless it is

needed to pass errors or warnings.

[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Disabling USE_LIBHDFS_SCAN by default. The new implementation of Hdfs Scan is now used to

scan the text type hive files.

Hdfs Scan is now stopped gracefully when the hive scan is cancelled. This avoids the random

core seen with new implementation.

  1. … 4 more files in changeset.
  1. … 2 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

Removed the need to allocate ComDiagsArea in advance for load command

by refactoring the ExeCliInterface::executeImmediate and

ExeCliInterface::executeImmediateExec methods.

Changes as per the review comments are also made in this commit

  1. … 16 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or error in

all the utility commands except load. In case of load, the ComDiagsArea

is allocated in advance to report error rows count.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

[TRAFODION-3017] Simplify the hive client access in Trafodion

Hive Client functions are now moved to a new file HiveClient_JNI.h and

HiveClient_JNI.cpp. Most of the HiveClient functions are static functions

allowing to use HiveClient in Trafodion with ease.

  1. … 33 more files in changeset.
Merge remote branch 'origin/release2.2' into TRAFODION-3014

Conflicts:

core/sql/cli/Cli.cpp

core/sql/clitest/blobtest.cpp

core/sql/executor/ExExeUtilLoad.cpp

core/sql/exp/ExpLOBaccess.cpp

core/sql/exp/ExpLOBaccess.h

core/sql/exp/ExpLOBenums.h

core/sql/exp/ExpLOBinterface.cpp

core/sql/exp/ExpLOBinterface.h

core/sql/generator/GenRelExeUtil.cpp

core/sql/regress/executor/EXPECTED130

core/sql/regress/hive/EXPECTED018

docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc

install/python-installer/scripts/traf_ldap.py

pom.xml

  1. … 11 more files in changeset.
Merge PR 1489 [TRAFODION-2853] memory leak of ComDiagsArea in CmpContext heap of mxosrvr

  1. … 1 more file in changeset.
[TRAFODION-2821] Trafodion core code base needs to be thread safe

Java exceptions thrown while calling the java methods from JNI in

Trafodion are stored in the current context. However in a multi-threaded

ESP environment, these exceptions should be stored in thread specific

variable to enable error handling to be thread safe. Otherwise, the JNI

errors could be overwritten by the JNI errors from another thread.

Also streamlined JNI error handling by extending the getExceptionDetails()

methods to log the errors also.

Incorporated error handling in SequenceFileReader JNI methods.

  1. … 22 more files in changeset.
[TRAFODION-2853] memory leak of ComDiagsArea in CmpContext heap of mxosrvr

Fixed a memory leak in ComDiagsArea that was missed out earlier. Also

took care of the review comments from the earlier PR

  1. … 17 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 41 more files in changeset.
Merge PR 1417 [TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 9 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 27 more files in changeset.
New syntax to retrieve the LOB HDFS filename for both external and internal LOBs . Also added syntax to return starting offset of a particular LOB handle in the LOB Hdfs data file.

  1. … 15 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-1 changes.

Created a new class org.trafodion.sql.HDFSClient. Any direct HDFS access

will be routed to this class via JNI instead of using libhdfs.

Modified the existing code expect for the following to route the HDFS request via this class

1. LOB access

2. Direct HDFS scan of the table

3. Sample data creation during update stats

Added a new class org.trafodio.sql.HdfsScan for scanning one or many ranges of a Hive

text formatted table. This class will be used for Direct HDFS scan in near future.

  1. … 21 more files in changeset.
Removed usage of system new and moved allocation of LOB globals to be derived from a heap. Moved xLObHdfsRequest also to be derived from a heap

  1. … 13 more files in changeset.
TRAFODION-2873. Fix for avoiding cores during expression eval for inserts into LOB tables. The fix involves cleaning up the LOBGlibals and eliminating unneeded datamemebers and keeping only the ExLobGlobals members. Also cleaned up all LobLoad related code since it is obsolete and worked only with DP2.

Conflicts:

core/sql/executor/ExExeUtilLoad.cpp

  1. … 9 more files in changeset.
Merge remote branch 'origin/master' into traf_misc

  1. … 3 more files in changeset.
Fix for AQR for missing hive tables and misc lob related fixes TRAFODION-2770

  1. … 7 more files in changeset.