selvaganesan <> in Trafodion

Fix for intermittent failure in core/TESTRTS

FILTERRTS can now tolerate the variance in the statistics for


Change-Id: Iccd2f15bbc320fbfc8f4b2fbbf6e7642efb3ac5d


Non-blocking hbase operation to smoothen the data flow in trafodion engine

A new CQD HBASE_ASYNC_OPERATIONS is introduced to enable non-blocking

hbase operation for checkAndInsert operations. The default value is


This allows the trafodion engine to insert rows into base table and

indexes at the same time. checkAndInsert operations always done under

a transaction ensuring the data consistency between tables and indexes

for a single row insert operation.

During this process, TM is refactored to enable passing a transaction

id created in a thread to another thread.

Change-Id: I0d09e5b0a3f988808c44bd83463df45fd21c67c0

    • -2
    • +1
    • -73
    • +209
    • -16
    • +67
    • -4
    • +10
    • -39
    • +100
    • -55
    • +90
Reworked fix for 1452424 VSBB scan cause query to return wrong result

The assumption that rowID should be exactly equal to the calculated

key length is not correct. Trafodion SQL engine uses a null extra byte

to ensure that the row ID is not found in case of data conversion errors

Hence direct ROWID buffer format is changed to accommodate this.

Now the format is

numRowIds + rowIDSuffix + rowId + rowIDSuffix + rowId + …

rowIDSuffix is '0' then the subsequent rowID is of length

= passed rowID len

'1' then the subsequent rowID is of length

= passed rowID len+1

Change-Id: I07a283895f6f9c652b3f933bcf0330b69ee2d300

Additional fix for 1452424 VSBB scan cause query to return wrong result

VSBB scan was returning wrong results randomly even for the same query.

An extra null byte was added in the row id at times while setting up the

unique key. This caused a shift in row id parsing at java layer leading

to row not found for all row ids after this extra byte.

In addition, VSSB select was not getting GET_EOD or GET_NOMORE queue

entries at the end of the query. Hence, the rowset was never getting

closed causing a resource leak.

Also, retained the other changes that helped in debugging this issue.

Change-Id: I6a807bddc8edaff2f4140931d4f228e94badcc05

RMS and other related changes

The process level statistics at ESP level can now be obtained

using RMS while the query is running. The SQL command is

select * from table(statistics(null, 'QID=<qid>,DETAIL=1'))

To get a list of ESPs processes participating in the query execution, issue

select distinct tdb_id, text process_name from

table(statistics(null, 'QID=<qid>,DETAIL=1')) order by tdb_id

Explain information can now be obtained using RMS. To do this

CQD EXPLAIN_IN_RMS 'ON' should be issued in the session before the

query is prepared.

EXPLAIN OPTIONS 'F' FOR QID <qid> from a different session can give

the explain info.

TESTRTS is now incorporated into core test suite.

AQR-ed queries were not getting garbage collected from RMS shared segment.

This has been fixed.

Missing code in IpcGuardinServer::serverDied method is now added.

Change-Id: Ib591ee5c9251ab6778e3dec9450f5b0466041e9b

    • -0
    • +3669
    • -0
    • +110
    • -0
    • +258
    • -0
    • +20
    • -0
    • +38
Fix for 1452424 vsbb scan/delete cause query to return wrong result

VSBB update/delete were not tracking the number of rows in the

buffer.This has been corrected.

Change-Id: I2a89ccd9a84832c4771481de2ee8503e912ce0d8

    • -4
    • +171
Enabling Bulk load and Hive Scan error logging/skip feature

Also Fixed the hanging issue with Hive scan (ExHdfsScan operator) when there

is an error in data conversion.

ExHbaseAccessBulkLoadPrepSQTcb was not releasing all the resources when there

is an error or when the last buffer had some rows.

Error logging/skip feature can be enabled in

hive scan using CQDs and in bulk load using the command line options.

For Hive Scan


CQD TRAF_LOAD_LOG_ERROR_ROWS ‘ON’ to log the error rows in Hdfs files.

For Bulk load

LOAD WITH CONTINUE ON ERROR [TO <location>] – to skip error rows

LOAD WITH LOG ERROR ROWS – to log the error rows in hdfs files.

The default parent error logging directory in hdfs is /bulkload/logs. The error

rows are logged in subdirectory ERR_<date>_<time>. A separate hdfs file is

created for every process/operator involved in the bulk load in this directory.

Error rows in hive scan are logged in


Error rows in bulk upsert are logged in


Bulk load can also aborted after a certain number of error rows are seen using


Change-Id: Ief44ebb9ff74b0cef2587705158094165fca07d3

    • -369
    • +413
    • -198
    • +380
  1. … 19 more files in changeset.
Changes to enable Rowset select - Fix for bug 1423327

HBase always returns an empty result set when the row is not found. Trafodion

is changed to exploit this concept to project no data in a rowset select.

Now optimizer has been enabled to choose a plan involving Rowset Select

where ever possible. This can result in plan changes for the queries -

nested join plan instead of hash join,

vsbb delete instead of delete,

vsbb insert instead of regular insert.

A new CQD HBASE_ROWSET_VSBB_SIZE is now added to control the hbase rowset size.

The default values is 1000

Change-Id: Id76c2e6abe01f2d1a7b6387f917825cac2004081

    • -59
    • +158
    • -21
    • +21
  1. … 5 more files in changeset.
Fixes in T2 driver to enable OE performance run

Following errors are ignored at the T2 jdbc driver to conform

to JDBC/ODBC Standard.

ERROR[8605] Committing a transaction which has not started.

ERROR[8609] Waited rollback performed without starting a transaction.

Memory corruption causing java core in T2 OE run.

Row count is treated as 32 bit integer while SQL expects 64 bit

numeric value to be passed to SQL_EXEC_GetDiagnosticsStmtInfo2.

This was causing the corruption.

There was a possibility that the row count buffer was used

after de-allocation. Fixed this code in Type2 JDBC driver

Change-Id: If0ae5475ed9986c8996cb324e679a615e62cd9b1

    • -4
    • +16
Changes to reduce the memory growth/leak in mxosrvr and T2 driver

Memory growth in mxosrvr

It was observed that the memory growth comes from JNIHandle::alloc_block

within JVM in mxosrvr. PushLocalFrame and PopLocalFrame is now called

in Trafodian JNI methods calling java functions to ensure this memory

growth is avoided.

CommonLogger was allocating and deallocating buffer to log the message

every time. Instead a thread safe variable is used to allocate only

once per thread.

CommonLogger was constructing the message even when the category is NOT

set. The message construction involves string creation and memory allocation.

Changes in T2.

The buffer to hold the output row was allocated every time. It is now

allocated once and reused till the statement is dropped.

The row count buffer was also allocated for every fetch but was used

after deallocation. Cleaned up the row count allocation code.

Change-Id: I7aad93beb03dc42a85b78da720737e2eb46a7080

    • -13
    • +11
    • -54
    • +500
    • -11
    • +7
Tracking java calls from JNI

A thread local variable tsRecentJMFromJNI is introduced to track

the recent java method called from the JNI. When the core dumps

are analyzed, this variable can be printed out to figure out the

recent hbase call that possibly didn't respond.

Change-Id: Ie0dcea636829f98d60e741e98bec1a7f46ac50b8

    • -0
    • +20
Avoid unnecessary memory allocation while logging via log4cpp

log4cpp.getInstance method takes in std::string argument. Wven the caller

passes in const char *, a std::string instance is created. Changed

code to pass std::string object from all callers to avoid memory

allocation from system heap

Change-Id: Id46e084b8eff8dacd31251a8a23ef013cf886a91

    • -1
    • +2
Improving the path length for get or multi-row get operations

Reduced Java Object creation and JNI to java transition

for get or multi-row get operations.

Change-Id: I7d637c9d45027b20e6f1ffe28a82bb4fb1860d9e

    • -47
    • +227
Reducing JNI to java transition for single or multiple key selects

The ResultSet for single or multiple key selects is now pushed

from java to JNI as part of startGet and startGets methods respectively.

Also reduced java object creation further for single key selects.

Also fixed runtime statistics accounting issue happened in an

earlier merge

Change-Id: I0eae08af93f91e1115d616942b6f5de4573b302c

    • -80
    • +103
Using one thread pool for all HTables access

Earlier we had a ThreadPool for every HTable. Now we have one

thread pool for all HTables across all JDBC/ODBC connections

to improve thread resource usage.

Also cleaned up code to ensure the configuration is read at the

lower layers. Removed the need to pass host name and port

number because it will be picked up from hbase-site.xml

Fix for bug 1429882

Disabled pre-fetch in case of snapshot scan

Change-Id: Ic6b0e6a04d2e4fd26e4290302b6229aa2367ffef

Reduced Java objects allocation in the client side

jmap on the client side revealed that it is possible to avoid some

java object creation on the client side. Reworked to avoid Callable

object creation with every pre-fetch operation. Removed

QualifiedColumn creation to parse family name and qualifier name.

Uses CachedThreadPool instead of FixedThreadPool to reduce thread

creation and share thread pool across HTableClients.

Change-Id: I05bf8f4f3f0aaa24c749a535b7586f7da799dd88

Reworked the fix for bug 1418757

Interrupting the thread to cancel the pending IO was causing the

client to retry the connection and pause when the HTable was used

next time.

This fix allows the pending IO to complete normally. If it can't

complete normally within 30 seconds, the thread is interrupted

and the table is removed from cache.

Change-Id: I404ec7a91952301ef2bcb1a8d6448f8534a421d8

Fix to take care of internal statements leak

Internal statements in Trafodion engine use ExeCliInterface. When

this interface is instantiated in a scope of method, statements

and other related objects can be leaked.

Added a destructor in ExeCliInterface so that these objects

are destroyed correctly when the interface goes out of scope.

This fix possibly takes care of memory growth when many statements

are compiled.

Change-Id: Ic47c912ae153e9f83cca298b66da40bf214ef3e7

Fix for bug 1404430 Phoenix T2 tests core dumping in CliStatement::execute

Phoneix T2 tests involve multiple JDBC connections in a single thread and

hence multiple CLI contexts in a single thread. With cancel

feature enabled, there is an outstanding nowaited query started message.

In the master process, SQL always wait on a set of IPC connections pertaining

to the current JDBC/ODBC connection for the nowaited IOs to complete in every

SQL thread.

With fast completion enabled in master, the SQL engine was completing the

IO for all the thread specific nowait IO calls without limiting itself

to a specific JDBC connection.

Disabled fast completion in Type 2 JDBC driver that enables set of

connection based nowait completion.

Change-Id: Ia1b6323acb3aeb1edc6bbd9eb3e242f4bedade63

    • -1
    • +2
Changes to destroy more elements in NATable destructor

Changed to use CmpStatement Heap instead of NATable Heap

while creating the table descrptor.

With these changes, the NATable Heap allocated size

is around 22K at the end of seabase/TEST020 with no table

entries in the cache. Earlier, the allocated size was

around 1.5MB.

Patch Set #5:Addressed all the review comments

Change-Id: I61777dcb02f67f87092340d69086ba0ec0de2493

Change that possibly fixes the bug 1418757

If a scanner is closed before the next call completes, it

results in OutOfOrderScannerNextException. Trafodion engine can

close the scanner when the next is active in a different thread

with pre-fetch concept. The fix is to cancel the pending next

before the scanner is closed.

Thanks to Oliver and Joanie for tracking this down. Ran

catman1/TEST137 many times successfully

Change-Id: I3b2872e0ca4fc6586e7e486600748c1102994e7b

Change that possibly fixes the bug 1418757

If a scanner is closed before the next call completes, it

results in OutOfOrderScannerNextException. Trafodion engine can

close the scanner when the next is active in a different thread

with pre-fetch concept. The fix is to cancel the pending next

before the scanner is closed.

Thanks to Oliver and Joanie for tracking this down. Ran

catman1/TEST137 many times successfully

Change-Id: Ifb080abdd9bbe4042c88e2df1fa07b17f8eba2b5

More memory leak Changes

Changes to delete more elements in NATable. LIST or collection type

elements in NATable that collect references are now deleted.

Also removed some defensive code in RMS because it is not valid.

Added code to delete the internal statements when the row

is not found

PatchSet 3 contains the fix for T4 Phoenix test failures

Change-Id: Ib43da75582dd56d2562cb753738abed0ea5842d3

Fix for more leaks in NATable

Added code to delete more elements in NATable destructor. This

fixes leaks when NATable is destroyed.

Changed the NATable Cache Management to use allocated size rather

than the total size.

This also enables seabase/TEST020 to pass in debug mode.

Change-Id: Ifebc20e602c0149f73f9cefcd11b37c69d9eec74

Fix for NATable Heap leak

Earlier, each NATable entry had its own heap. Now, there is a NATable Heap

created for each compiler context. All the cached NATable object is

created from this heap.

This is done to reduce the memory usage in the compiler context and to

take care of leak of NATable heap.

Change-Id: I3ec362c0b8e5b1466956c3773c2b5dc7dccf0d52

Changes to defend RMS memory segment corruption

Added code to avoid memory corruption in the RMS shared segment by making

the rogue process to dump core if it doesn't adhere to rules of access to the

RMS shared segment.

Enabled seabase/TEST024

Change-Id: Ibb0bbbc9f4617da30bca9a7129d016079ac2fdbe

    • -1
    • +1
Fix for an issue with drop schema statement

Drop schema was not removing the NATable cached entry for the tables

in the schema because catalog name and object name got switched in an

internal statement.

When the SSCP process dies while it had locked the RMS semaphore,

SSMP didn't take control of the semaphore to release it. There was no activity

in SSMP process and hence SSMP didn't take control. Code changed to register

SSCP process in the shared segment and this can result in death message to

be processed by SSMP process. Processing of death message of SSCP will allow

SSMP to take control and release the semaphore held by SSCP when it died.

Change-Id: Ib4bc93b9614d5222033487091e3a2461f72e11f0

Fix for compGeneral/TEST011 failure

Removed the code in switchToUserTransid call. For reasons unknown this

code was returning stale transid for GETTRANSID call. The fix for the

bug 1355038 doesn't require BACTIVATERECEIVETRANSID to be called

Also ESPs and other processes were inheriting the transid without this


Change-Id: I7a0b9a99adf3f756793b93ab155a0a5e8d23fcdb

Fix for bug 1409128

Reworked the code to delete Compiler Contexts (CmpContext) from

the Trafodion SQL Engine CLI contexts (ContextCli). Now all the

CmpContexts created in a ContextCli are deallocated when it is deleted

as part of JDBC disconnect call.

Earlier, the code was deallocating the current CmpContext alone. However

the current CmpContext may not be from the current ContextCli. This was

causing the deallocated memory to be referenced again.

Change-Id: Ia957f17a8733e528495107b097d29bdc92d7bf95

Reworked Transaction handling in SQL

Fixed to inherit the transaction from the received message in ESPs, MXUDR or

any other process when it is passed from master processes.

Reworked switching transaction when the context is switched to handle more than

one connection in a thread. This fixed the following bugs

Bug 1356193 SPJ with resultsets not working thru JDBC T2 driver

Bug 1355038 SPJ with result set failed with ERROR[8841]

ESPs will particpate in a transaction with this change

Change-Id: If91f70060f718818eec25aef206f318262cf47aa