ComTdbHbaseAccess.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-2775] Insert does not raise duplicate row error for hbase format table with defaulted first column

There was a problem with column being checked during CheckAndPut call for such tables. We were always checking the first column in the row, which may not have existed in the previous version of this rowid. Please see JIRA for a complete explanation.

  1. … 9 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
[TRAFODION-2438] Unnecessary and sometimes wrong delete during upsert/merge with IM

This change is done by Hans. Design by Selva.

Please see JIRA for an explanation.

  1. … 15 more files in changeset.
TRAFODION-1586 Add support for external Traf tables mappedd to HBase tables

This is the initial support for HBase mapped tables.

A doc for external manuals will be added later.

regress/seabase/TEST022 contains various testcases.

  1. … 77 more files in changeset.
jira TRAFIDION-1435 SQL support for region transactions

This checkin is to add support for sql compiler/executor support

for dtm region transactions.

cqd traf_use_region_xn 'ON' enables it.

Off by default for now until dtm support is complete and tested.

Explain (full and short) will also show if region transactions

are being used.

core/test116 has been enhanced with additional tests

  1. … 19 more files in changeset.
[TRAFODION-1421] Implement parallel scanner primitive new CQD: hbase_dop_parallel_scanner - default 0.0 = disabled - from 0.x to 1.0 -> the DOP will be dynamically assessed as a percentage of region to scan. If 10 region, and cqd is .3 (30%), then 3 threads will be used... - from 2.0 and above, will take ceiling, and use it as fixed number of threads. Advantage of parallel scanner parallelism over ESP is on resource consumption. Parallel scanner is just creating threads, while ESP are full proceses wth high memory consumption. For now, this feature is just a primitive that can be manually exercised. The next obvious step is to make the compiler aware of it and pick it over ESP DOP when appropriate...

  1. … 21 more files in changeset.
[TRAFODION-1918] Remove auto flush concept in Trafodion

The auto flush is on by default in insertRows method. But insertRows was

incorrectly called with the wrong parameter. In fixing this issue, it was

observed that auto flush is deprecated in HBase 1.x. It is also incorrect

to disable auto flush in Trafodion. Hence removed the code related to

auto flush, flush table and flush all tables in Trafodion.

All these are dormant code in Trafodion.

Also disabled executor/TEST106 for now because it seems to cause a hang

at the time of drop table.

  1. … 13 more files in changeset.
update

  1. … 85 more files in changeset.
update

  1. … 298 more files in changeset.
[TRAFODION-1900]

When doing MDAM scans, we are performing interlaced scan for PROBE and for real scan. The probes always return only 1 row, then we close the scanner immediately, therefore should use always small scanner. I will make it conditional on the existing CQD HBASE_SMALL_SCANNER (ether SYSTEM or ON). In addition, caching of blocks retrieved by probe should always we at least receiving one succesfull cache hit on the next MDAM scan, therefore forcing caching ON for MDAM prob is a good idea. Again, will make this forcing conditional on HBASE_SMALL_SCANNER SYSTEM or ON.

Then for the real scan part of MDAM, I will use the following heuristic: If previous scan fitted in one hbase block, then it is likelly than next will also fit in one hbase block, therefore enable small scanner for next scan. Again all this only if CQD above is ON or SYSTEM.

Also includes a fix where SMALL_SCANNER would be turned on for MDAM scan because the compiler for MDAM is not polulating the expected number of rows returned.

Results of using small scanner on MDAM when it make sense showed a 1.39X speed improvement...

  1. … 2 more files in changeset.
Merge remote branch 'origin/master' into lob_gc_work

Conflicts:

core/sql/sqlcomp/CmpSeabaseDDLtable.cpp

  1. … 11 more files in changeset.
Merge remote branch 'origin/master' into lob_gc_work

  1. … 13 more files in changeset.
[TRAFODION-1700] Upsert with omitted default value columns leave the aligned format table in corrupted state. [TRAFODION-1847] Upsert with omitted timestamp columns having current_timestamp as default in a non-aligned format table returns wrong value for this column

For TRAFODION-1847, the upsert is transformed into merge.

For TRAFODION-1700, the upsert is transformed into merge when the CQD

TRAF_UPSERT_WITH_INSERT_DEFAULT_SEMANTICS is set to OFF. By default this CQD is

set to OFF. When this CQD is ON, the upsert will add a new row with the omitted

columns filled with default values always.

  1. … 12 more files in changeset.
Implement TRAFODION-1420 Use ClientSmallScanner for small scans to improve perdormance Hbase implements an optimization for small scan (defined as scanning less than a data block ie 64Kb) resulting in 3X performance improvement. The underlying trick is about cutting down RPC calls from 3 (OpenScan/Next/Close) to 1, and use pread stateless instead of seek/read state-full and locking method to read data. This JIRA is about improving the compiler who can be aware if a scan will be acting on single data block (small) or not, and pass this data to executor so that it can use the right parameter for scan. (scan.setSmall(boolean)). reference: https://issues.apache.org/jira/browse/HBASE-9488 https://issues.apache.org/jira/browse/HBASE-7266

  1. … 27 more files in changeset.
Anoop's delete expresssion change

  1. … 8 more files in changeset.
Merge remote branch 'origin/pr/167/head' into mrg_80

  1. … 4 more files in changeset.
[TRAFODION-1575] Avoid transforming update into delete and insert

commands to improve performance of update statements. When a table has

check constraint the update command is transformed into insert and delete.

Added code to evaluate constraint expressions in all TCBs so that

constraint checking is done correctly. But, the subtask [TRAFODION-1610]

needs to implemented before this change is exercised.

  1. … 6 more files in changeset.
[TRAFODION-1549] [TRAFODION-1550] TRAFODION-1549 : Extends event_log_reader TMUDF to read DCS logs and overflow files from all components TRAFODION-1550 : Improves performance of bulkloader by determining flush size based on row length Other changes Rework for TRAFODION-1279 suggested by Hans and not addressed in previous pull request Rework for TRAFODION-1474 for a regression found by Weishiun Tsai Fix for regress/hive/TEST020 occasional failure on build machine

  1. … 17 more files in changeset.
[TRAFODION-1456] Merge statement returns error 3241 when there is unique index in the table [TRAFODION-1493] Logs TMDUF does not scan udr logs Also enabled merge with views and merge with RI constraint.

  1. … 15 more files in changeset.
Rework for review comments + resolve merge conflict

Merge branch 'master' of github.com:apache/incubator-trafodion into khaled_cif

Conflicts:

core/sql/generator/GenRelScan.cpp

  1. … 4 more files in changeset.
[Trafodion-1424] Enable CIF for Trafodion scan operator When a row is read is read by the scan operator from native HBase or Trafodion tables (either aligned or regular format) a convert expression is applied to convert the row into a form usable by rest of query tree. Output of convert expression is now in aligned format, with varchars not being blank padded. Change can be disabled with cqd COMPRESSED_INTERNAL_FORMAT 'OFF'. Majority of change is due to Khaled Bouaziz.

  1. … 5 more files in changeset.
Part 2 of changes to update license text in Trafodion source

This check-in updates license text in all source files in the

core/sql directory. Remaining directories will be done in

later check-ins.

  1. … 3147 more files in changeset.
Fix for [TRAFODION-14]

upsert or merge into a table with indexes can result

in inconsistency between index and table

This change avoids the HBase issue of rows getting lost when the

delete and put operations are done in the same milliseconds

timestamp dimension for the same rowid.

In the trafodion_delete operation execution, the row is skipped

for deletion if the new and old rowid evaluates to the same value.

This needs change from the compiler sub-task in the JIRA to complete

for it to be effective.

In addition removed yet another memory allocation for the delete

operation.

  1. … 8 more files in changeset.
Merge remote branch 'core/master'

  1. … 108 more files in changeset.
JIRA TRAFODION-9

Non-blocking hbase operation to smoothen the data flow in trafodion engine

A new CQD HBASE_ASYNC_OPERATIONS is introduced to enable non-blocking

hbase operation for checkAndInsert operations. The default value is

'OFF'.

This allows the trafodion engine to insert rows into base table and

indexes at the same time. checkAndInsert operations always done under

a transaction ensuring the data consistency between tables and indexes

for a single row insert operation.

During this process, TM is refactored to enable passing a transaction

id created in a thread to another thread.

Change-Id: I0d09e5b0a3f988808c44bd83463df45fd21c67c0

  1. … 15 more files in changeset.
Move core into subdir to combine repos

    • -0
    • +1028
    ./ComTdbHbaseAccess.h
  1. … 10768 more files in changeset.
Move core into subdir to combine repos

    • -0
    • +951
    ./ComTdbHbaseAccess.h
  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

    • -0
    • +1054
    ./ComTdbHbaseAccess.h
  1. … 10837 more files in changeset.