ExpLOBaccess.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Chages to disable LOB locking and refactor IUD operations

  1. … 7 more files in changeset.
[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

  1. … 28 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

This feature is enabled by default. To disable, set a variable USE_LIBHDFS=1 in

$TRAF_HOME/etc/ms.env and restart the trafodion cluster.

This feature includes the following:

1. Uses single FSDataInputStream for each LOB column in a query as

opposed to the opening the hdfs file for every row.

2. Uses FSDataOutputStream to write the lob data but closes it

immediately to allow concurrent writes to the hdfs file. HDFS supports

a single writer at a time. Need to conform if multiple writes can

be done without the need for RMS lock feature.

3. Improved error messaging that displays the java exception stack to the

end user.

4. LOB worker threads are no longer created

  1. … 12 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 41 more files in changeset.
New syntax to retrieve the LOB HDFS filename for both external and internal LOBs . Also added syntax to return starting offset of a particular LOB handle in the LOB Hdfs data file.

  1. … 15 more files in changeset.
Remove default param for heap

  1. … 1 more file in changeset.
Removed usage of system new and moved allocation of LOB globals to be derived from a heap. Moved xLObHdfsRequest also to be derived from a heap

  1. … 13 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
Merge remote branch 'origin/pr/1216/head' into merge_1216

  1. … 11 more files in changeset.
Main change:

Added a method for JDBC driver to be able to close an open cursor abruptly by passing in a 0 length to the extract syntax. This will ensure internal opens are not left around if the application does not retrieve all the data with the stream API.

This abrupt close can be called when the result set is closed or the connection is closed by keeping track of all the open Blob objects. (in the driver)

Added a freeResources method and destructor to LobExtract and LobUpdate Tcbs.

Minor changes and fixes :

Refactored the ExLob::closeCursor

Removed the LOB_CLI_SELECT_CLOSE call from ::fetchCursor so it gets closed only when ExLob::closeCursor is called. Makes the code consistent and readable.

Took care of a few memory allocations and deletions.

Refactored the LOB length calculation into on ecommon method so it works correctly for both regular LOB dat aas well as external LOB files. Earlier the external lobs length was being returned incorrectly. Now it reports the length of the external HDFS file .

Fixed one place in generator where the CQD value was not being multiplied by the MB units correctly.

  1. … 10 more files in changeset.
Fixed review comments

  1. … 5 more files in changeset.
Support for insert-select for tables containing LOB columns.

  1. … 16 more files in changeset.
Reworks

  1. … 4 more files in changeset.
[TRAFODION-2440] Replace fixed length buffers with strings in LOB code

  1. … 1 more file in changeset.
[TRAFODION-2342] hdfs access threads init/cleanup changes

The worker threads to read hdfs directly was being started

unnecessarily in many cases. Also, ensured that these worker

threads are cleanly exited when the statement is deallocated.

  1. … 10 more files in changeset.
similarity check changes, commit #1

  1. … 44 more files in changeset.
Additional changes to JIRA TRAFODION-2060 plus couple other fixes

-- interval computations involving tinyint are fixed.

-- hive data mod mismatch error now prints the compile and runtime

timestamps that caused the mismatch.

-- hive test003 updated to truncate data before insert

  1. … 21 more files in changeset.
Adding AQR retry for ENOENT and EAGAIN which may be encountered after an hdfsOpen on a file that has gone through a timestamp check. Squashed commit of the following:

commit 52551b6461e0b761aeb37958ecd8673ad29bec89

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Tue Jul 12 20:27:33 2016 +0000

Check for NULL pointer

commit a7bb285fb7874b314acb6c74cde5e0b6911d6930

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Tue Jul 12 16:12:17 2016 +0000

Rework to pass errno at the lowest layer as output param

commit abb236336ee68765fcf9c534fbeae3fe6af5cdb6

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Mon Jul 11 19:22:04 2016 +0000

Removed all retries and added AQR retry for missing hdfs file after timestamp check.

commit 2530c4ebb8274bea8394067d2d2212b5bc3c97ef

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Mon Jul 11 07:11:35 2016 +0000

timeout to 20

commit 825bd5be2b0db4006bf82a0a9348338f7fc66236

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Mon Jul 11 03:59:11 2016 +0000

increase sleep

commit e471a4655ca8efe469e5dd8ab14748660fa5cdec

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sun Jul 10 19:00:03 2016 +0000

Unconditionally retrying on open errors.

commit b0402243a17094f7fec9899a6afc5c31c48cc38b

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sun Jul 10 03:31:01 2016 +0000

Retrying for ENOENT too

commit cbb2175282095edb73b30e79a9728f689b80c435

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sat Jul 9 23:14:31 2016 +0000

Rework and returning errno

commit dd99c24deec09add76342e583193f79646fc9661

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sat Jul 9 05:05:43 2016 +0000

fixing build issue

commit 66941774b1087ce564932af0c30ce50f15d0da67

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sat Jul 9 04:40:59 2016 +0000

Increasing retry and add hdfsClose to cleanup .

commit 1934f2b3b7b32ee4a0396d9e56457ffc9f2cb9cf

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Fri Jul 8 21:59:00 2016 +0000

Adding retry logic when making hdfsOpenFile calls to read an hdfs file to handle potential EAGAIN errors. [TRAFODION-2097]

  1. … 6 more files in changeset.
Merge remote branch 'origin/master' into lobglobals_fix

Conflicts:

core/sql/cli/Context.cpp

core/sql/exp/ExpLOBaccess.cpp

  1. … 13 more files in changeset.
Changes to move hdfFs handing to the context globals level and remove all disconnects from hdfs from the various components.

  1. … 27 more files in changeset.
JIRA TRAFODION-2022 Some more changes related to hive truncate

-- support for 'TRUNCATE' command for hive tables

-- support for truncate with PARTITION specification

-- separation of traf purgedata and hive truncate code into

different classes and tdbs

-- some cleanup of obsolete older purgedata code

-- new tests added to regress/hive/TEST005

  1. … 23 more files in changeset.
hive data modification detection: commit #4

  1. … 24 more files in changeset.
hive data modification detection: commit #3

  1. … 14 more files in changeset.
hive data modification detection: commit #2

  1. … 13 more files in changeset.
hive data modification detection: commit #1

  1. … 16 more files in changeset.
support for get command for LOB tables. Support for tracing LOB operations.

  1. … 30 more files in changeset.
update

  1. … 298 more files in changeset.
Changes for performing GC. Changes to pass in hdfsServer and port via CQD and remove hardcoding. Changes to pass in lobLocation via CQD and remove hardcoding. Changes to remove all mxlobsrvr process functionality from ExLOBprocess.cpp. Retaining only the infrasturcture so mxlobsrvr process can be used in the future. The executable is still built but the process is not started as part of Trafodion anymore. We may use it in the future.

  1. … 28 more files in changeset.
Anoop's delete expresssion change

  1. … 8 more files in changeset.
Add a new table to keep track of lob offsets for GC.

  1. … 9 more files in changeset.