ExpHbaseInterface.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

  1. … 28 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or errors in

all the utility commands.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

  1. … 14 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or error in

all the utility commands except load. In case of load, the ComDiagsArea

is allocated in advance to report error rows count.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

[TRAFODION-3017] Simplify the hive client access in Trafodion

Hive Client functions are now moved to a new file HiveClient_JNI.h and

HiveClient_JNI.cpp. Most of the HiveClient functions are static functions

allowing to use HiveClient in Trafodion with ease.

  1. … 33 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 27 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-1 changes.

Created a new class org.trafodion.sql.HDFSClient. Any direct HDFS access

will be routed to this class via JNI instead of using libhdfs.

Modified the existing code expect for the following to route the HDFS request via this class

1. LOB access

2. Direct HDFS scan of the table

3. Sample data creation during update stats

Added a new class org.trafodio.sql.HdfsScan for scanning one or many ranges of a Hive

text formatted table. This class will be used for Direct HDFS scan in near future.

  1. … 21 more files in changeset.
[TRAFODION-2775] Insert does not raise duplicate row error for hbase format table with defaulted first column

There was a problem with column being checked during CheckAndPut call for such tables. We were always checking the first column in the row, which may not have existed in the previous version of this rowid. Please see JIRA for a complete explanation.

  1. … 9 more files in changeset.
[TRAFODION-2617] Optionally push estimate row count logic into HBase coproc

  1. … 10 more files in changeset.
[TRAFODION-2654] Change the location of trafodion-site.xml from $TRAF_HOME/etc to config [TRAFODION-2653] Sort operator loops at times

All HBase client connections from Trafodion uses Trafodion configuration

concept. Any client side property can now be set in trafodion-site.xml.

The scratch files are now open in blocking mode. Hence any scratch file

operation shouldn't return EAGAIN.

The internal table TDDL of table is now created without any table

coprocessors attached to it.

  1. … 23 more files in changeset.
[TRAFODION-2440] Add retry to row count estimation logic

  1. … 16 more files in changeset.
[Trafodion 2351] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 14 more files in changeset.
[TRAFODION-2356] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 13 more files in changeset.
jira TRAFIDION-1435 SQL support for region transactions

This checkin is to add support for sql compiler/executor support

for dtm region transactions.

cqd traf_use_region_xn 'ON' enables it.

Off by default for now until dtm support is complete and tested.

Explain (full and short) will also show if region transactions

are being used.

core/test116 has been enhanced with additional tests

  1. … 19 more files in changeset.
[TRAFODION-2129] Trafodion to avoid use of deprecated HBase APIs/Classes

Avoided the caching of Hbase table instances on the client side to avoid

compaction queue size build up while constantly loading the data

  1. … 11 more files in changeset.
[JIRA TRAFODION-2141] Cluster view of hbase stats

-- added column REGION_SERVER to region stats virtual table

-- added cluster view of hbase stats

Syntax: select * from table(cluster stats());

-- added support for 'where' pred in stats queries

Syntax: select * from table(cluster stats()) where schema_name = 'SCH';

  1. … 22 more files in changeset.
[TRAFODION-2074] Create index should avoid populating the index within a transaction Create index is now done in multiple transactions when there is no active transaction. 1) creating the index table in transaction 1. 2) populate the index without any transaction. If it fails, cleanup the index table. 3) If it succeeds complete the rest of the index creation in transaction 2. 4) If there are any failures, cleanup the index table.

When there is an active transaction while the create index is issued, then

all the above steps are done within the user transaction.

Also did some cleanup to return proper errors instead of dumping core

  1. … 7 more files in changeset.
[TRAFODION-1955] JNI optimization at the time of compilation

Removed ByteArrayList class and used array of byte array instead.

Only one JNI to java transistion is done to getRegionStartKeys

and getRegionEndKeys at the time of query compilation

Also fixed an issue with lib_mgmt Makefile.

  1. … 15 more files in changeset.
[TRAFODION-1421] Implement parallel scanner primitive new CQD: hbase_dop_parallel_scanner - default 0.0 = disabled - from 0.x to 1.0 -> the DOP will be dynamically assessed as a percentage of region to scan. If 10 region, and cqd is .3 (30%), then 3 threads will be used... - from 2.0 and above, will take ceiling, and use it as fixed number of threads. Advantage of parallel scanner parallelism over ESP is on resource consumption. Parallel scanner is just creating threads, while ESP are full proceses wth high memory consumption. For now, this feature is just a primitive that can be manually exercised. The next obvious step is to make the compiler aware of it and pick it over ESP DOP when appropriate...

  1. … 21 more files in changeset.
[TRAFODION-1918] Remove auto flush concept in Trafodion

The auto flush is on by default in insertRows method. But insertRows was

incorrectly called with the wrong parameter. In fixing this issue, it was

observed that auto flush is deprecated in HBase 1.x. It is also incorrect

to disable auto flush in Trafodion. Hence removed the code related to

auto flush, flush table and flush all tables in Trafodion.

All these are dormant code in Trafodion.

Also disabled executor/TEST106 for now because it seems to cause a hang

at the time of drop table.

  1. … 13 more files in changeset.
update

  1. … 298 more files in changeset.
[TRAFODION-1868] Compatibility with gcc 4.8

  1. … 3 more files in changeset.
Merge remote branch 'origin/master' into deliver2_dt_br

-- resolved merge conflicts

Conflicts:

core/sql/executor/HBaseClient_JNI.h

core/sql/regress/executor/EXPECTED013.SB

core/sql/regress/seabase/EXPECTED011

core/sql/sqlcomp/CmpSeabaseDDL.h

core/sql/sqlcomp/CmpSeabaseDDLcommon.cpp

core/sql/sqlcomp/CmpSeabaseDDLtable.cpp

  1. … 18 more files in changeset.
JIRA TRAFODION-1798 (ddl xns) and few other fixes, details below.

-- support for sql part of ddl xns. Section 1 of JIRA TRAFODION-1798

-- cqd ddl_transactions to enable or disable ddl xns.

Default is currently off. Once it is tested, it will be turned on.

Dev regressions are run with cqd set to ON

-- get stmts run with read committed to get changes in current xns

-- support for where preds with get stmts

-- scan to pass in transid even if running with read uncommitted access.

This enables rows modified in current xn to be returned.

-- cleanup no longer return multiple duplicate error messages if

objects id is not found.

-- cleanup no longer includes internallay created schemas (_HV_ , _HB_)

during cleanup operations.

-- Correct error msg was not getting returned if an invalid index

existed in table and the same index was created again.

-- init traf, drop md views was giving an error if views didnt exist.

That has been fixed.

-- regressions with -diff option now show original file timestamps

instead of the timestamp when the diff command was run.

  1. … 72 more files in changeset.
JIRA 1844: support for 'alter table alter column datatype'

  1. … 26 more files in changeset.
Implement TRAFODION-1420 Use ClientSmallScanner for small scans to improve perdormance Hbase implements an optimization for small scan (defined as scanning less than a data block ie 64Kb) resulting in 3X performance improvement. The underlying trick is about cutting down RPC calls from 3 (OpenScan/Next/Close) to 1, and use pread stateless instead of seek/read state-full and locking method to read data. This JIRA is about improving the compiler who can be aware if a scan will be acting on single data block (small) or not, and pass this data to executor so that it can use the right parameter for scan. (scan.setSmall(boolean)). reference: https://issues.apache.org/jira/browse/HBASE-9488 https://issues.apache.org/jira/browse/HBASE-7266

  1. … 27 more files in changeset.
JIRA Trafodion-1645 SQL commands to retrieve region stats.

Details in JIRA.

  1. … 23 more files in changeset.
[TRAFODION-9] Non-blocking hbase operation to smoothen the data flow in trafodion engine

The non-blocking hbase operation all index maintenance operators is now enabled by

default for IUD, merge and update commands.

The hbase operations put, checkAndPut and delete operations from the trafodion engine

for the index maintenance is done using Executor Services to make it non-blocking and the

data flow in the trafodion operators can continue up to the point till it needs to pause for completion.

checkAndDelete is never used for index maintenance now.

Also, ensured that index maintenance uses rowset or single row operators when it should.

  1. … 13 more files in changeset.
[TRAFODION-1448] Leak in HTableClient objects

Refactored the code to ensure that HTableClient object in java

is released for every batch of gets.

In the process removed unused code in HTableClient_JNI.cpp and

ExpHbaseInterface.cpp

  1. … 6 more files in changeset.
Merge branch 'master' into trafodion-1444

Conflicts:

core/sql/executor/HBaseClient_JNI.cpp

  1. … 5 more files in changeset.
[TRAFODION-1444] Reduced pathlength for IUD

Reduced the number of JNI to java transitions and

memory allocation by streamlining the Trafodion-HBase

client layer for IUD commands

Also removed unreferenced code and variables in this layer.

  1. … 5 more files in changeset.
fix JIRA 1425 and 1432

  1. … 12 more files in changeset.