ExHbaseAccess.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

LOB: Extract lobtofile() to a hdfs file returns 8442 error

Implemented the missing code to support this functionality via the new

implementation.

  1. … 8 more files in changeset.
[TRAFODION-2821] Trafodion core code base needs to be thread safe

Java exceptions thrown while calling the java methods from JNI in

Trafodion are stored in the current context. However in a multi-threaded

ESP environment, these exceptions should be stored in thread specific

variable to enable error handling to be thread safe. Otherwise, the JNI

errors could be overwritten by the JNI errors from another thread.

Also streamlined JNI error handling by extending the getExceptionDetails()

methods to log the errors also.

Incorporated error handling in SequenceFileReader JNI methods.

  1. … 22 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 41 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text format hive tables

Part-6 changes

Removed the dependency of libhdfs while assigning ranges for hdfs scan at runtime

and for sample file creation during bulk load

  1. … 12 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 27 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-1 changes.

Created a new class org.trafodion.sql.HDFSClient. Any direct HDFS access

will be routed to this class via JNI instead of using libhdfs.

Modified the existing code expect for the following to route the HDFS request via this class

1. LOB access

2. Direct HDFS scan of the table

3. Sample data creation during update stats

Added a new class org.trafodio.sql.HdfsScan for scanning one or many ranges of a Hive

text formatted table. This class will be used for Direct HDFS scan in near future.

  1. … 21 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
[TRAFODION-2754] Get statistics cores sqlci or mxosrvr at str_sprintf()

Switched str_sprinf to use the standard sprintf function to ensure

that the format specification and the passed in parameters are consistent

  1. … 65 more files in changeset.
Various fixes, details below

-- max length limited to 16777216 bytes (16M)

for char cols and functions (repeat, concat).

(optimizer/SynthType.cpp, common/ComSmallDefs.h, sqlcomp/nadefaults.cpp)

-- previous max length change requires adding of a new hbase property

called hbase.client.keyvalue.maxsize so large key/values could be

handled in hbase cell.

Following scripts have been updated to handle that.

(sqf/sql/scripts/install_local_hadoop,

install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/

service_advisor.py

install/installer/traf_hortonworks_mods

install/python-installer/configs/mod_cfgs.json)

Developers can also update hbase-site.xml with this property if

they dont want to reinstall local hadoop.

<property>

<name>hbase.client.keyvalue.maxsize</name>

<value>0</value>

</property>

-- while accessing a hive table as an external table, the hive table and

corresponding external table definitions are validated to be the

same. This validates that corresponding columns have the same

data attributes (type, length, scale, etc).

This check causes failures if hive column is of 'string' datatype.

That is because hive 'string' column length can be changed by

a cqd but the corresponding external table has predefined length

set when the table is created.

The validation check now ignores the length attribute if the hive column is

of 'string' datatype.

(optimizer/BindRelExpr.cpp, common/CharType.*, NAType.*)

-- data moved into direct buffer would sometimes cause overflow and crash.

The max direct buffer length used to send/retrieve hbase data

is now limited to 1G (executor/ExHbaseAccess.cpp)

-- errors during vsbb upsert are now handled correctly

(executor/ExHbaseIUD.cpp)

-- support for GET CATALOGS command

(generator/GenRelExeUtil.cpp, executor/ExExeUtilGet.cpp)

-- An incorrect computation would sometimes cause group by rollup to

crash compiler in NAHeap::unlinkLargeFragment()

(generator/GenRelGrby.cpp)

-- sort of a large row that exceeded sort pre-set buffer size would crash

if the row size exceeded the max buffer size.

That has been fixed by allocating space for atleast one row.

(generator/GenRelMisc.cpp)

-- purgedata on a delimited name was failing. That has been fixed.

(optimizer/RelExeUtil.h)

-- regress/tools/runregr_privs1/privs2 fixed to handle running of

subset of tests

-- regress/seabase/TEST031 updated with new tests

  1. … 30 more files in changeset.
Merge [TRAFODION-2438] PR-922 Unnecessary and sometimes wrong delete during upsert/merge with IM

  1. … 2 more files in changeset.
[TRAFODION-2438] Unnecessary and sometimes wrong delete during upsert/merge with IM

This change is done by Hans. Design by Selva.

Please see JIRA for an explanation.

  1. … 15 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

LOAD with log error rows failed when ERRORCOUNTER hbase table already exists.

LOAD command now returns the number of error rows skipped.

LOAD with truncate option dumps core when encounters an error.

Similar issue exist with PURGEDATA command too.

  1. … 18 more files in changeset.
Merge [TRAFODION-2420] PR-898 RMS enhancements

  1. … 4 more files in changeset.
[TRAFODION-2420] RMS enhancements

Introduced a feature to list queries blocked in storage engine

via RMS offender.

This lists queries which are blocked in the storage engine APIs

for more than given number of seconds. The BLOCKED_FOR_SECS denotes how

long the storage engine API is blocked.

The SQL commands to list SE_BLOCKED queries are available at:

$TRAF_HOME/export/limited-support-tools/LSO/se_blocked.sql

Example output is below:

cd $TRAF_HOME/export/limited-support-tools/LSO

./offender -s se_blocked

CURRENT_TIMESTAMP NO_OF_PROCESSES BLOCKED_FOR_SECS QUERY_ID TABLE_NAME

-------------------------- --------------- ---------------- ------------------------------------------------------------ ------------------------------

2016-12-28 10:29:43.941455 1 83 MXID11000030514212349680799580002000000000206U3333300_18_S1 TRAFODION.SCH.T022

  1. … 8 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

Changes to fix the hive failures due to core at

ExHbaseAccessTcb::buildLoggingFileName when the logging location is not set in TDB.

  1. … 2 more files in changeset.
Merge [TRAFODION-2351] PR 876 Bulk load with log error rows enhancements

  1. … 4 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

Fixes to take care of review comments

  1. … 5 more files in changeset.
TRAFODION-1586 Add support for external Traf tables mappedd to HBase tables

This is the initial support for HBase mapped tables.

A doc for external manuals will be added later.

regress/seabase/TEST022 contains various testcases.

  1. … 77 more files in changeset.
[TRAFODION-2351] Bulk load with log error rows enhancements

The LOAD command now outputs as follows:

Load with log error rows into selva.customer select * from hive.hive.customer ;

Task: LOAD Status: Started Object: TRAFODION.SELVA.CUSTOMER

Task: CLEANUP Status: Started Time: 2016-12-11 00:54:37.642

Task: CLEANUP Status: Ended Time: 2016-12-11 00:54:37.672

Task: CLEANUP Status: Ended Elapsed Time: 00:00:00.030

Logging Location: /bulkload/logs/ERR_TRAFODION.SELVA.CUSTOMER_20161211_005437

Task: LOADING DATA Status: Started Time: 2016-12-11 00:54:37.672

Rows Processed: 99997

Task: LOADING DATA Status: Ended Time: 2016-12-11 00:54:58.296

Task: LOADING DATA Status: Ended Elapsed Time: 00:00:20.624

Task: COMPLETION Status: Started Time: 2016-12-11 00:54:58.296

Task: COMPLETION Status: Ended Time: 2016-12-11 00:54:59.521

Task: COMPLETION Status: Ended Elapsed Time: 00:00:00.756

In addition, currently the "LOADING DATA" task status is shown as it happens. In future,

the task status will be shown as it happens for all tasks.

  1. … 13 more files in changeset.
[Trafodion 2351] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 14 more files in changeset.
[TRAFODION-2356] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 13 more files in changeset.
jira [trafodion-2290] and few other bug fixes

-- trafodion-2290: pcode support for added columns on aligned tables

-- lob conversion error is reported instead of crash

-- alter table rename is restricted to not run under a user xn

-- error during alter table rename are handled correctly

-- error during alter/drop cols on aligned tables are handled correctly.

-- aligned format drop column with delimited names are handled correctly

-- rollup grouping now work correctly on expressions and renamed columns

  1. … 23 more files in changeset.
Create NACollection objects on NAHeap. Avoid use of system heap as far as possible

  1. … 50 more files in changeset.
[TRAFODION-2109] Load with log error rows returns SQL error 2034 at times

Changed the error at the time of logging into error rows into a warning

so that load can continue.

Also, improved the error reporting to display stack trace when java method call

fails at the time of loading and unloading.

  1. … 11 more files in changeset.
Merge branch 'master' of github.com:apache/incubator-trafodion into index_issue

Conflicts:

core/sql/bin/SqlciErrors.txt

  1. … 8 more files in changeset.
[TRAFODION-2074] Create index should avoid populating the index within a transaction

Fix for regression failure core/TEST116

Fix to return the error 8034 when an inconsistency due to missing column values

is detected at the time of fetching the row.

  1. … 3 more files in changeset.
[TRAFODION-1700] Upsert with omitted default value columns leave the aligned format table in corrupted state. [TRAFODION-1847] Upsert with omitted timestamp columns having current_timestamp as default in a non-aligned format table returns wrong value for this column

For TRAFODION-1847, the upsert is transformed into merge.

For TRAFODION-1700, the upsert is transformed into merge when the CQD

TRAF_UPSERT_WITH_INSERT_DEFAULT_SEMANTICS is set to OFF. By default this CQD is

set to OFF. When this CQD is ON, the upsert will add a new row with the omitted

columns filled with default values always.

  1. … 12 more files in changeset.
following code review outcome on pull request 255: - for all files, fix tab with white space - ExHbaseAccess.cpp, add comment - Initialize nac in file GenPreCode.cpp, line 11938 and 11417 - Remove checks for ITM_REFERENCE (used only in parser/binder). File GenPreCode.cpp - Remove isFirstAndLayer code in recursive function generating the predicate to push down in RPN - add comments in ValueDesc.cpp - fix indentation in HTableClient.java - Fix 4 buffer overrun vulnerability in GenExplain.cpp - create a reusable isAddedColumnWithNonNullDefault in ValueId class. (ValueDesc.cpp and .h)

  1. … 10 more files in changeset.
First commit for advanced predicate pushdown feature (also known as pushdown V2) associated JIRA TRAFODION-1662 Predicate push down revisited (V2). The JIRA contains a blueprint document, useful to understand what the code is supposed to do. This code is enabled using CQD hbase_filter_preds '2', and bypassed otherwise. Except for the change implemented in ValueDesc.cpp that is a global bug fix whereValueIdSet are supposed to contain set of valueID ANDed together, and should not contain any ValueID with operator ITM_AND.

  1. … 16 more files in changeset.
[TRAFODION-1575] Avoid transforming update into delete and insert

commands to improve performance of update statements. When a table has

check constraint the update command is transformed into insert and delete.

Added code to evaluate constraint expressions in all TCBs so that

constraint checking is done correctly. But, the subtask [TRAFODION-1610]

needs to implemented before this change is exercised.

  1. … 6 more files in changeset.