ExpLOBaccess.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Chages to disable LOB locking and refactor IUD operations

  1. … 7 more files in changeset.
Remove code that empties the LOB data file when creating an empty_blob.

  1. … 2 more files in changeset.
[TRAFODION-2626] Make logs directory location configurable

Makes path to log directory independent of TRAF_HOME. The default location

is still $TRAF_HOME/logs. The DCS and REST log directories are now

sub-directories of the main TRAF_LOG directory.

  1. … 72 more files in changeset.
[TRAFODION-3110] LOB: Extract lobtofile() to a hdfs file returns 8442 error

Fixes as per the review comments for the commit ba00576e1f47a9f0e0b3f344da8742aeecbc3ce

Also took care of truncate option.

[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

LOB: Extract lobtofile() to a hdfs file returns 8442 error

Implemented the missing code to support this functionality via the new

implementation.

  1. … 8 more files in changeset.
[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

  1. … 28 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

The CQD USE_LIBHDFS is now made effective for updateLOB and extract LOB utility

commands

  1. … 2 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClien

Ensured that hdfs scan with CQD USE_LIBHDFS set to 'ON'

  1. … 3 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

Used a single CQD USE_LIBHDFS to switch to older implementation of

using libhdfs for both hdfs scan and LOB access. The CQD USE_LIBHDS_SCAN is

retired.

In addition fixed the following:

1. Fixed a memory leak of LOB Heap structures

2. Possible fix for memory corruption at times

3. Avoid unnecessary creation of LOB threads when not needed

  1. … 23 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

This feature is enabled by default. To disable, set a variable USE_LIBHDFS=1 in

$TRAF_HOME/etc/ms.env and restart the trafodion cluster.

This feature includes the following:

1. Uses single FSDataInputStream for each LOB column in a query as

opposed to the opening the hdfs file for every row.

2. Uses FSDataOutputStream to write the lob data but closes it

immediately to allow concurrent writes to the hdfs file. HDFS supports

a single writer at a time. Need to conform if multiple writes can

be done without the need for RMS lock feature.

3. Improved error messaging that displays the java exception stack to the

end user.

4. LOB worker threads are no longer created

  1. … 12 more files in changeset.
  1. … 2 more files in changeset.
Merge remote branch 'origin/pr/1503/head' into merge_1503

  1. … 8 more files in changeset.
Performance changes for LOB and increasing limit for larger varchars/bobs to use with batch input.

  1. … 17 more files in changeset.
Merge remote branch 'origin/release2.2' into TRAFODION-3014

Conflicts:

core/sql/cli/Cli.cpp

core/sql/clitest/blobtest.cpp

core/sql/executor/ExExeUtilLoad.cpp

core/sql/exp/ExpLOBaccess.cpp

core/sql/exp/ExpLOBaccess.h

core/sql/exp/ExpLOBenums.h

core/sql/exp/ExpLOBinterface.cpp

core/sql/exp/ExpLOBinterface.h

core/sql/generator/GenRelExeUtil.cpp

core/sql/regress/executor/EXPECTED130

core/sql/regress/hive/EXPECTED018

docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc

install/python-installer/scripts/traf_ldap.py

pom.xml

  1. … 11 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 41 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 15 more files in changeset.
Review fixes

  1. … 1 more file in changeset.
New syntax to retrieve the LOB HDFS filename for both external and internal LOBs . Also added syntax to return starting offset of a particular LOB handle in the LOB Hdfs data file.

  1. … 15 more files in changeset.
Removed usage of system new and moved allocation of LOB globals to be derived from a heap. Moved xLObHdfsRequest also to be derived from a heap

  1. … 13 more files in changeset.
change str_sprintf to snprintf

  1. … 2 more files in changeset.
Fixes to datamodcheck

  1. … 1 more file in changeset.
Merge remote branch 'origin/master' into traf_misc

  1. … 3 more files in changeset.
Fix for AQR for missing hive tables and misc lob related fixes TRAFODION-2770

  1. … 7 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
[TRAFODION-2754] Get statistics cores sqlci or mxosrvr at str_sprintf()

Switched str_sprinf to use the standard sprintf function to ensure

that the format specification and the passed in parameters are consistent

  1. … 65 more files in changeset.
Merge remote branch 'origin/pr/1216/head' into merge_1216

  1. … 11 more files in changeset.
Main change:

Added a method for JDBC driver to be able to close an open cursor abruptly by passing in a 0 length to the extract syntax. This will ensure internal opens are not left around if the application does not retrieve all the data with the stream API.

This abrupt close can be called when the result set is closed or the connection is closed by keeping track of all the open Blob objects. (in the driver)

Added a freeResources method and destructor to LobExtract and LobUpdate Tcbs.

Minor changes and fixes :

Refactored the ExLob::closeCursor

Removed the LOB_CLI_SELECT_CLOSE call from ::fetchCursor so it gets closed only when ExLob::closeCursor is called. Makes the code consistent and readable.

Took care of a few memory allocations and deletions.

Refactored the LOB length calculation into on ecommon method so it works correctly for both regular LOB dat aas well as external LOB files. Earlier the external lobs length was being returned incorrectly. Now it reports the length of the external HDFS file .

Fixed one place in generator where the CQD value was not being multiplied by the MB units correctly.

  1. … 10 more files in changeset.
Changes to take care of closing internal cursor statement when extract operation is done and the exact length of data is passed in to be extracted.

  1. … 2 more files in changeset.
Fixes from testing JDBC usage of blob datatype.

  1. … 2 more files in changeset.
Fixed review comments

  1. … 5 more files in changeset.