ExpLOB.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Chages to disable LOB locking and refactor IUD operations

  1. … 7 more files in changeset.
[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

  1. … 28 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClien

Ensured that hdfs scan with CQD USE_LIBHDFS set to 'ON'

  1. … 3 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

Used a single CQD USE_LIBHDFS to switch to older implementation of

using libhdfs for both hdfs scan and LOB access. The CQD USE_LIBHDS_SCAN is

retired.

In addition fixed the following:

1. Fixed a memory leak of LOB Heap structures

2. Possible fix for memory corruption at times

3. Avoid unnecessary creation of LOB threads when not needed

  1. … 23 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

This feature is enabled by default. To disable, set a variable USE_LIBHDFS=1 in

$TRAF_HOME/etc/ms.env and restart the trafodion cluster.

This feature includes the following:

1. Uses single FSDataInputStream for each LOB column in a query as

opposed to the opening the hdfs file for every row.

2. Uses FSDataOutputStream to write the lob data but closes it

immediately to allow concurrent writes to the hdfs file. HDFS supports

a single writer at a time. Need to conform if multiple writes can

be done without the need for RMS lock feature.

3. Improved error messaging that displays the java exception stack to the

end user.

4. LOB worker threads are no longer created

  1. … 12 more files in changeset.
Address review comments

  1. … 5 more files in changeset.
Support to provide a locking mechanism for LOB insert/update operations

  1. … 28 more files in changeset.
Removed usage of system new and moved allocation of LOB globals to be derived from a heap. Moved xLObHdfsRequest also to be derived from a heap

  1. … 13 more files in changeset.
TRAFODION-2873. Fix for avoiding cores during expression eval for inserts into LOB tables. The fix involves cleaning up the LOBGlibals and eliminating unneeded datamemebers and keeping only the ExLobGlobals members. Also cleaned up all LobLoad related code since it is obsolete and worked only with DP2.

Conflicts:

core/sql/executor/ExExeUtilLoad.cpp

  1. … 9 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
[TRAFODION-2754] Get statistics cores sqlci or mxosrvr at str_sprintf()

Switched str_sprinf to use the standard sprintf function to ensure

that the format specification and the passed in parameters are consistent

  1. … 65 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
[TRAFODION-2308]JDBC support insert LOB

  1. … 41 more files in changeset.
Fixed review comments

  1. … 5 more files in changeset.
Support for insert-select for tables containing LOB columns.

  1. … 16 more files in changeset.
Support for insert-select from varchar/char colummns into lob columns. Supposed for insert with parameter.This work is done as part of [TRAFODION-2516]

  1. … 5 more files in changeset.
Reworks

  1. … 4 more files in changeset.
The cause ffor the core could be due to the use of an address of a stack variable being passed out of scope. Removed the use of the stack intermediate variable and simplified the passing of a user buffer address to the LOB interfaces. In all cases now, the user buffer address contents are used directly eliminating one level of indirection.

  1. … 2 more files in changeset.
Fixed misc issues from review and testware issues

  1. … 7 more files in changeset.
Merge remote branch 'origin/master' into empty_lob_work

  1. … 22 more files in changeset.
Changes to support new syntax and implementation empty_blob/empty_clob for insert/updates Changes to support new syntax for updating a lob directly through a lobhandle without scanning the table.This is done via new ExeUtil operator. Changes to add columnname info to LOBMD_ table. This helps in mapping any given lobhandle back to the table and the column. Added new sections to the regression test executor/TEST130 to test these changes

  1. … 40 more files in changeset.
[TRAFODION-2342] hdfs access threads init/cleanup changes

The worker threads to read hdfs directly was being started

unnecessarily in many cases. Also, ensured that these worker

threads are cleanly exited when the statement is deallocated.

  1. … 10 more files in changeset.
Changes to move hdfFs handing to the context globals level and remove all disconnects from hdfs from the various components.

  1. … 27 more files in changeset.
Merge remote branch 'origin/master'

Conflicts:

core/sql/regress/executor/EXPECTED130

  1. … 6 more files in changeset.
Support for external lobs. Data files are stored externally in HDFS and only filehandles and file locations are stored in internal Trafodion tables.

  1. … 25 more files in changeset.
Improve error message for lob functions called on invalid LOB types.

  1. … 2 more files in changeset.
Better error message and test addition in regressions.

  1. … 2 more files in changeset.
Fixes for values() clause on lob columns and functions

  1. … 2 more files in changeset.
Fix for Trafodion-1941 to allow syntax for secifying length for blob/clob columns.iInvolves syntax changees and ene runtime checks and a new testcase in executor/TEST130.

Error handling fix related to JIRA TRAFODION-1943. The drop schema code will now exit as soon as there is an error and not continue executing.ALso fixed a problem in drop schema code where "if exists" was not specified correctly leading to syntax errors.

  1. … 11 more files in changeset.
update

  1. … 298 more files in changeset.