ExpLOBinterface.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Merge remote branch 'origin/master' into traf_remove_loblock

Conflicts:

core/sql/exp/ExpLOBinterface.cpp

  1. … 1 more file in changeset.
Chages to disable LOB locking and refactor IUD operations

  1. … 7 more files in changeset.
[TRAFODION-3234] Refactor hive meta calls

Hive similarity check was detecting if there is a change in the hive

directory timestamp only. It has been changed to detect the DDL changes too.

Cleaned up code further and removed all the unwanted code.

  1. … 16 more files in changeset.
[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

  1. … 28 more files in changeset.
[TRAFODION-3065] Trafodion to support compressed Hive Text formatted tables

Compressed text files are now supported via the new implementation using

HDFS java APIs. When the hadoop is not configured to support a particular type

of compression, an error is thrown.

[TRAFODION-2982] JNI HDFS interface should support varied sized large buffers for read/write

A new CQD HDFS_IO_INTERIM_BYTEARRAY_SIZE_IN_KB is introduced to chunk

the read and write when byteArray is involved.

  1. … 16 more files in changeset.
Merge PR 1417 [TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 9 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 27 more files in changeset.
New syntax to retrieve the LOB HDFS filename for both external and internal LOBs . Also added syntax to return starting offset of a particular LOB handle in the LOB Hdfs data file.

  1. … 15 more files in changeset.
Removed usage of system new and moved allocation of LOB globals to be derived from a heap. Moved xLObHdfsRequest also to be derived from a heap

  1. … 13 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
Merge remote branch 'origin/pr/1216/head' into merge_1216

  1. … 11 more files in changeset.
Main change:

Added a method for JDBC driver to be able to close an open cursor abruptly by passing in a 0 length to the extract syntax. This will ensure internal opens are not left around if the application does not retrieve all the data with the stream API.

This abrupt close can be called when the result set is closed or the connection is closed by keeping track of all the open Blob objects. (in the driver)

Added a freeResources method and destructor to LobExtract and LobUpdate Tcbs.

Minor changes and fixes :

Refactored the ExLob::closeCursor

Removed the LOB_CLI_SELECT_CLOSE call from ::fetchCursor so it gets closed only when ExLob::closeCursor is called. Makes the code consistent and readable.

Took care of a few memory allocations and deletions.

Refactored the LOB length calculation into on ecommon method so it works correctly for both regular LOB dat aas well as external LOB files. Earlier the external lobs length was being returned incorrectly. Now it reports the length of the external HDFS file .

Fixed one place in generator where the CQD value was not being multiplied by the MB units correctly.

  1. … 10 more files in changeset.
Support for insert-select for tables containing LOB columns.

  1. … 16 more files in changeset.
[TRAFODION-2315] Heuristic decision for common subexpressions

This includes various fixes in addition to the heuristic decision,

mainly an improvement on how to navigate between parent and child

common subexpressions.

  1. … 29 more files in changeset.
[TRAFODION-2342] hdfs access threads init/cleanup changes

The worker threads to read hdfs directly was being started

unnecessarily in many cases. Also, ensured that these worker

threads are cleanly exited when the statement is deallocated.

  1. … 10 more files in changeset.
similarity check changes, commit #1

  1. … 44 more files in changeset.
Additional changes to JIRA TRAFODION-2060 plus couple other fixes

-- interval computations involving tinyint are fixed.

-- hive data mod mismatch error now prints the compile and runtime

timestamps that caused the mismatch.

-- hive test003 updated to truncate data before insert

  1. … 21 more files in changeset.
Adding AQR retry for ENOENT and EAGAIN which may be encountered after an hdfsOpen on a file that has gone through a timestamp check. Squashed commit of the following:

commit 52551b6461e0b761aeb37958ecd8673ad29bec89

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Tue Jul 12 20:27:33 2016 +0000

Check for NULL pointer

commit a7bb285fb7874b314acb6c74cde5e0b6911d6930

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Tue Jul 12 16:12:17 2016 +0000

Rework to pass errno at the lowest layer as output param

commit abb236336ee68765fcf9c534fbeae3fe6af5cdb6

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Mon Jul 11 19:22:04 2016 +0000

Removed all retries and added AQR retry for missing hdfs file after timestamp check.

commit 2530c4ebb8274bea8394067d2d2212b5bc3c97ef

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Mon Jul 11 07:11:35 2016 +0000

timeout to 20

commit 825bd5be2b0db4006bf82a0a9348338f7fc66236

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Mon Jul 11 03:59:11 2016 +0000

increase sleep

commit e471a4655ca8efe469e5dd8ab14748660fa5cdec

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sun Jul 10 19:00:03 2016 +0000

Unconditionally retrying on open errors.

commit b0402243a17094f7fec9899a6afc5c31c48cc38b

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sun Jul 10 03:31:01 2016 +0000

Retrying for ENOENT too

commit cbb2175282095edb73b30e79a9728f689b80c435

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sat Jul 9 23:14:31 2016 +0000

Rework and returning errno

commit dd99c24deec09add76342e583193f79646fc9661

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sat Jul 9 05:05:43 2016 +0000

fixing build issue

commit 66941774b1087ce564932af0c30ce50f15d0da67

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Sat Jul 9 04:40:59 2016 +0000

Increasing retry and add hdfsClose to cleanup .

commit 1934f2b3b7b32ee4a0396d9e56457ffc9f2cb9cf

Author: Sandhya Sundaresan <sandhya@apache.org>

Date: Fri Jul 8 21:59:00 2016 +0000

Adding retry logic when making hdfsOpenFile calls to read an hdfs file to handle potential EAGAIN errors. [TRAFODION-2097]

  1. … 6 more files in changeset.
Changes to move hdfFs handing to the context globals level and remove all disconnects from hdfs from the various components.

  1. … 27 more files in changeset.
hive data modification detection: commit #4

  1. … 24 more files in changeset.
hive data modification detection: commit #2

  1. … 13 more files in changeset.
hive data modification detection: commit #1

  1. … 16 more files in changeset.
update

  1. … 298 more files in changeset.
Changes for performing GC. Changes to pass in hdfsServer and port via CQD and remove hardcoding. Changes to pass in lobLocation via CQD and remove hardcoding. Changes to remove all mxlobsrvr process functionality from ExLOBprocess.cpp. Retaining only the infrasturcture so mxlobsrvr process can be used in the future. The executable is still built but the process is not started as part of Trafodion anymore. We may use it in the future.

  1. … 28 more files in changeset.
Recommitting the last commit. Some files did notget added. Context/changes are the same as earlier commit.

  1. … 24 more files in changeset.
Merge remote branch 'origin/master' into lob_work_files2

  1. … 25 more files in changeset.
Fixes for extract to files and dealing with issues ith 2 or more lob columns.

  1. … 20 more files in changeset.
Part 2 of changes to update license text in Trafodion source

This check-in updates license text in all source files in the

core/sql directory. Remaining directories will be done in

later check-ins.

  1. … 3147 more files in changeset.
Support for chunking during insertion into lobs and extracting into external files. Currently supported for lob insert and lob extract to/from local linux files.

  1. … 17 more files in changeset.
Merge remote branch 'core/master'

  1. … 108 more files in changeset.