ExHbaseAccess.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

  1. … 28 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text format hive tables

Part-6 changes

Removed the dependency of libhdfs while assigning ranges for hdfs scan at runtime

and for sample file creation during bulk load

  1. … 12 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-1 changes.

Created a new class org.trafodion.sql.HDFSClient. Any direct HDFS access

will be routed to this class via JNI instead of using libhdfs.

Modified the existing code expect for the following to route the HDFS request via this class

1. LOB access

2. Direct HDFS scan of the table

3. Sample data creation during update stats

Added a new class org.trafodio.sql.HdfsScan for scanning one or many ranges of a Hive

text formatted table. This class will be used for Direct HDFS scan in near future.

  1. … 21 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 2: Remove obsolete code

This phase handles the following:

-- removed files:

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

exp/exp_space.h

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

sqlshare/catapirequest.*

-- removed defines and code referencing them:

-- NA_STD_NAMESPACE

-- NA_NO_CMPCONTEXT

-- NA_CATMAN_SIM, NA_CATMAN_SIM_FS

-- common/purify.h

-- DONT_USE_MATH_H

-- NT_PORT

-- NA_MSVC

-- NA_NO_FRIENDS_WITH_TEMPLATE_REFS

-- NA_FLEXBUILD

-- removed multiple obsolete sqlci features and syntax:

(report writer, MACL, Help, Simulators, Utils, MXCS mode, Help,

and few others).

-- removed following files in sqlci dir:

CSInterface.h

CharSetConstants.cpp

CharSetConstants.h

MsgCat.cpp

MsgCat.h

MxciEHCallBack.cpp

MxciEHCallBack.h

RWInterface.cpp

RWInterface.h

SqlciCSCmd.cpp

SqlciCSCmd.h

SqlciCSSimulator.cpp

SqlciHelp.cpp

SqlciRWCmd.cpp

SqlciRWCmd.h

SqlciRWSimulator.cpp

SqlciUsage.cpp

SqlciUtil.cpp

SqlciUtil.h

UtilInt.cpp

UtilInt.h

UtilMsg.cpp

UtilMsg.h

immudefs.cpp

immudefs.h

  1. … 85 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

LOAD with log error rows failed when ERRORCOUNTER hbase table already exists.

LOAD command now returns the number of error rows skipped.

LOAD with truncate option dumps core when encounters an error.

Similar issue exist with PURGEDATA command too.

  1. … 18 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

Changes to fix the hive failures due to core at

ExHbaseAccessTcb::buildLoggingFileName when the logging location is not set in TDB.

  1. … 2 more files in changeset.
Merge [TRAFODION-2351] PR 876 Bulk load with log error rows enhancements

  1. … 4 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

Fixes to take care of review comments

  1. … 5 more files in changeset.
TRAFODION-1586 Add support for external Traf tables mappedd to HBase tables

This is the initial support for HBase mapped tables.

A doc for external manuals will be added later.

regress/seabase/TEST022 contains various testcases.

  1. … 77 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

The LOAD command now outputs as follows:

Load with log error rows into selva.customer select * from hive.hive.customer ;

Task: LOAD Status: Started Object: TRAFODION.SELVA.CUSTOMER

Task: CLEANUP Status: Started Time: 2016-12-11 00:54:37.642

Task: CLEANUP Status: Ended Time: 2016-12-11 00:54:37.672

Task: CLEANUP Status: Ended Elapsed Time: 00:00:00.030

Logging Location: /bulkload/logs/ERR_TRAFODION.SELVA.CUSTOMER_20161211_005437

Task: LOADING DATA Status: Started Time: 2016-12-11 00:54:37.672

Rows Processed: 99997

Task: LOADING DATA Status: Ended Time: 2016-12-11 00:54:58.296

Task: LOADING DATA Status: Ended Elapsed Time: 00:00:20.624

Task: COMPLETION Status: Started Time: 2016-12-11 00:54:58.296

Task: COMPLETION Status: Ended Time: 2016-12-11 00:54:59.521

Task: COMPLETION Status: Ended Elapsed Time: 00:00:00.756

In addition, currently the "LOADING DATA" task status is shown as it happens. In future,

the task status will be shown as it happens for all tasks.

  1. … 13 more files in changeset.
[Trafodion 2351] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 14 more files in changeset.
[TRAFODION-2356] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 13 more files in changeset.
jira [trafodion-2290] and few other bug fixes

-- trafodion-2290: pcode support for added columns on aligned tables

-- lob conversion error is reported instead of crash

-- alter table rename is restricted to not run under a user xn

-- error during alter table rename are handled correctly

-- error during alter/drop cols on aligned tables are handled correctly.

-- aligned format drop column with delimited names are handled correctly

-- rollup grouping now work correctly on expressions and renamed columns

  1. … 23 more files in changeset.
[TRAFODION-2109] Load with log error rows returns SQL error 2034 at times

Changed the error at the time of logging into error rows into a warning

so that load can continue.

Also, improved the error reporting to display stack trace when java method call

fails at the time of loading and unloading.

  1. … 11 more files in changeset.
[TRAFODION-1918] Remove auto flush concept in Trafodion

The auto flush is on by default in insertRows method. But insertRows was

incorrectly called with the wrong parameter. In fixing this issue, it was

observed that auto flush is deprecated in HBase 1.x. It is also incorrect

to disable auto flush in Trafodion. Hence removed the code related to

auto flush, flush table and flush all tables in Trafodion.

All these are dormant code in Trafodion.

Also disabled executor/TEST106 for now because it seems to cause a hang

at the time of drop table.

  1. … 13 more files in changeset.
update

  1. … 298 more files in changeset.
Merge remote branch 'origin/master' into lob_gc_work

Conflicts:

core/sql/sqlcomp/CmpSeabaseDDLtable.cpp

  1. … 11 more files in changeset.
Merge remote branch 'origin/master' into lob_gc_work

  1. … 13 more files in changeset.
[TRAFODION-1700] Upsert with omitted default value columns leave the aligned format table in corrupted state. [TRAFODION-1847] Upsert with omitted timestamp columns having current_timestamp as default in a non-aligned format table returns wrong value for this column

For TRAFODION-1847, the upsert is transformed into merge.

For TRAFODION-1700, the upsert is transformed into merge when the CQD

TRAF_UPSERT_WITH_INSERT_DEFAULT_SEMANTICS is set to OFF. By default this CQD is

set to OFF. When this CQD is ON, the upsert will add a new row with the omitted

columns filled with default values always.

  1. … 12 more files in changeset.
[TRAFODION-1778] Update command when HBASE_VSBB_ROWSET_OPT 'ON' fails with internal error at times

Direct buffer used in update command was not getting reset correctly when the exact number of

rows as the maximum number of rows set using CQD HBASE_VSBB_ROWSET_SIZE is buffered for rowset

operation. This caused the subsequent rowset operation to fail.

  1. … 1 more file in changeset.
Anoop's delete expresssion change

  1. … 8 more files in changeset.
[TRAFODION-1752] JVM goes out-of-memory for hundreds of upsert statement in one session

Changed to CLOSE_AND_DONE

  1. … 1 more file in changeset.
[TRAFODION-1752] JVM goes out-of-memory for hundreds of upsert statement in one session

It was observed that there are more conditions wherein HTableClient object could be

leaked. Fixed the code in all those places too.

  1. … 2 more files in changeset.
[TRAFODION-1575] Avoid transforming update into delete and insert

commands to improve performance of update statements. When a table has

check constraint the update command is transformed into insert and delete.

Added code to evaluate constraint expressions in all TCBs so that

constraint checking is done correctly. But, the subtask [TRAFODION-1610]

needs to implemented before this change is exercised.

  1. … 6 more files in changeset.
Rework for issues found by Dave and Selva. The flush size is not set in units of KB, with a default of 1024KB. Explain will report it in rows. Size in rows, as sent to executor is capped at 32767.

  1. … 6 more files in changeset.
[TRAFODION-1549] [TRAFODION-1550] TRAFODION-1549 : Extends event_log_reader TMUDF to read DCS logs and overflow files from all components TRAFODION-1550 : Improves performance of bulkloader by determining flush size based on row length Other changes Rework for TRAFODION-1279 suggested by Hans and not addressed in previous pull request Rework for TRAFODION-1474 for a regression found by Weishiun Tsai Fix for regress/hive/TEST020 occasional failure on build machine

  1. … 17 more files in changeset.
[TRAFODION-1456] Merge statement returns error 3241 when there is unique index in the table [TRAFODION-1493] Logs TMDUF does not scan udr logs Also enabled merge with views and merge with RI constraint.

  1. … 15 more files in changeset.
[TRAFODION-9] Non-blocking hbase operation to smoothen the data flow in trafodion engine

Changed the operator steps to COMPLETE_ASYNC_...

Changed the queue length of rowset operator to be based on the HBASE_ROWSET_VSBB_SIZE CQD.

Changes to report "Actual Rows Used" in the get statistics for qid <qid> default correctly.

  1. … 3 more files in changeset.
[TRAFODION-9] Non-blocking hbase operation to smoothen the data flow in trafodion engine

The non-blocking hbase operation all index maintenance operators is now enabled by

default for IUD, merge and update commands.

The hbase operations put, checkAndPut and delete operations from the trafodion engine

for the index maintenance is done using Executor Services to make it non-blocking and the

data flow in the trafodion operators can continue up to the point till it needs to pause for completion.

checkAndDelete is never used for index maintenance now.

Also, ensured that index maintenance uses rowset or single row operators when it should.

  1. … 13 more files in changeset.