Clone Tools
Constraints: committers
Constraints: files
Constraints: dates
[TRAFODION-2821] Trafodion core code base needs to be thread safe

Fixes as per review comments and missed out changes in the commit 073cf68dd4d3

[TRAFODION-2821] Trafodion core code base needs to be thread safe

Java exceptions thrown while calling the java methods from JNI in

Trafodion are stored in the current context. However in a multi-threaded

ESP environment, these exceptions should be stored in thread specific

variable to enable error handling to be thread safe. Otherwise, the JNI

errors could be overwritten by the JNI errors from another thread.

Also streamlined JNI error handling by extending the getExceptionDetails()

methods to log the errors also.

Incorporated error handling in SequenceFileReader JNI methods.

  1. … 22 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 15 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 27 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-1 changes.

Created a new class org.trafodion.sql.HDFSClient. Any direct HDFS access

will be routed to this class via JNI instead of using libhdfs.

Modified the existing code expect for the following to route the HDFS request via this class

1. LOB access

2. Direct HDFS scan of the table

3. Sample data creation during update stats

Added a new class org.trafodio.sql.HdfsScan for scanning one or many ranges of a Hive

text formatted table. This class will be used for Direct HDFS scan in near future.

  1. … 21 more files in changeset.
[TRAFODION-2801] select count(*) with cqd hbase_coprocessors 'off' and cqd traf_table_snapshot_scan 'latest' fail

Changed to pass two more arrays references - familyArray and QualifierArray

from Java to JNI to extract the family and the qualifier names correctly

on the native side

  1. … 3 more files in changeset.
[TRAFODION-2775] Insert does not raise duplicate row error for hbase format table with defaulted first column

There was a problem with column being checked during CheckAndPut call for such tables. We were always checking the first column in the row, which may not have existed in the previous version of this rowid. Please see JIRA for a complete explanation.

  1. … 9 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
[TRAFODION-2617] Optionally push estimate row count logic into HBase coproc

  1. … 10 more files in changeset.
[TRAFODION-2681] Repeated execution of prepared SQL select statement causes memory leak

This leak can be seen with the mdam plans when it attempts to do multiple scans

of different key ranges

  1. … 1 more file in changeset.
Fix in the commit e458290

Merge branch 'master' of git://git.apache.org/incubator-trafodion into trafodion-2654

  1. … 1 more file in changeset.
ra TRAFODION-2584 Add support to register hive schemas in traf MD

This checkin extends jira TRAFODION-2584 to add support to register

hive schemas(referred to as databases in hive) in trafodion metadata.

Until now, only hive tables and views were registered.

Registering hive schemas would allow certain operations, like

granting/revoking privileges, to be done on a hive schema/database.

Syntax to register/unregister hive schema

-- register/unregister hive schema hive.sch;

Syntax to show registred hive schemas

-- get hive registered schemas in catalog trafodion

"showddl schema hive.sch" will now show the ddl for hive database

creation and whether this schema is registered.

showddl and 'get tables in schema' will now return error if that

schema doesn't exist.

regress/hive/TEST007 has been extended with additional tests.

  1. … 15 more files in changeset.
[TRAFODION-2654] Change the location of trafodion-site.xml from $TRAF_HOME/etc to config [TRAFODION-2653] Sort operator loops at times

All HBase client connections from Trafodion uses Trafodion configuration

concept. Any client side property can now be set in trafodion-site.xml.

The scratch files are now open in blocking mode. Hence any scratch file

operation shouldn't return EAGAIN.

The internal table TDDL of table is now created without any table

coprocessors attached to it.

  1. … 23 more files in changeset.
[TRAFODION-2596] Improve the log4j and log4cxx infrastructure in Trafodion

The following changes are made in the way Trafodion logs the messages:

Writes to a log file based on a component or set of components.

C++ part of the code base:

Component Default log file Configuration File

TM tm_<nid>.log log4cxx.trafodion.tm.config

SSCP sscp_<nid>.log log4cxx.trafodion.sscp.config

SSMP ssmp_<nid>.log log4cxx.trafodion.ssmp.config

All SQL processes trafodion.sql_<nid>.log log4cxx.trafodion.sql.config






Java part of the code base

TM trafodion.dtm.log log4j.dtm.config

SQL trafodion.sql.java.log log4j.sql.config

By default, the log level is set to INFO for most of the cases. When a message

dominates the log file and if doesn't add value in the current level, it will be

changed to the higher level in the hierarchy. This should help to improve the

usefulness of the log file at the default INFO level.

The existence of an environment variable TRAF_MULTIPLE_SQL_LOG_FILE will revert back

to the old way of logging into multiple files. Then, the configuration file

log4cxx.trafodion.masterexe.config will be used.

Currently, RollingFileAppender appender is used for both modes. Different config files

are used to change this appender when the need arises.

log4cxx.trafodion.udr.config and log4cxx.trafodion.lob.config are removed.

log4j.hdfs.config used as the config file for SQL is renamed to log4j.sql.config

Foundation components logging will be revamped later.

This closes #1070

  1. … 19 more files in changeset.
Rework based on Selva's comments

[TRAFODION-2455] More refinements to row count estimation retry logic

  1. … 7 more files in changeset.
[TRAFODION-2440] Add retry to row count estimation logic

  1. … 16 more files in changeset.
TRAFODION-1586 Add support for external Traf tables mappedd to HBase tables

This is the initial support for HBase mapped tables.

A doc for external manuals will be added later.

regress/seabase/TEST022 contains various testcases.

  1. … 77 more files in changeset.
[Trafodion 2351] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 14 more files in changeset.
[TRAFODION-2356] HBase snapshot concept needs to be contained within HBaseClient in Trafodion

Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.

  1. … 13 more files in changeset.
[TRAFODION-2356] Trafodion process can dump core at times because JNIEnv is not initialized

  1. … 2 more files in changeset.
jira TRAFIDION-1435 SQL support for region transactions

This checkin is to add support for sql compiler/executor support

for dtm region transactions.

cqd traf_use_region_xn 'ON' enables it.

Off by default for now until dtm support is complete and tested.

Explain (full and short) will also show if region transactions

are being used.

core/test116 has been enhanced with additional tests

  1. … 19 more files in changeset.
[TRAFODION-2129] Trafodion to avoid use of deprecated HBase APIs/Classes

Avoided the caching of Hbase table instances on the client side to avoid

compaction queue size build up while constantly loading the data

  1. … 11 more files in changeset.
[JIRA TRAFODION-2141] Cluster view of hbase stats

-- added column REGION_SERVER to region stats virtual table

-- added cluster view of hbase stats

Syntax: select * from table(cluster stats());

-- added support for 'where' pred in stats queries

Syntax: select * from table(cluster stats()) where schema_name = 'SCH';

  1. … 22 more files in changeset.
[TRAFODION-2129] Trafodion to avoid use of deprecated HBase APIs/Classes

References to the deprecated classes HConnection, HConnectionManager and

HBaseAdmin are reworked to use Connection, ConnectionFactory and Admin classes


A connection object is created and it is passed around to ensure that the

same connection object is used within the process. Connection object is thread safe.

Admin object is created and closed after its use immediately.

Creation of admin object is lightweight and the object is not thread safe

Also fixed the random hang at the time of rollback transaction waiting for the response from the TM process.

The TM process hung because it assumed that RS didn't respond back when there is an exception in DDL

rollback or end-point coprocessor call.

  1. … 28 more files in changeset.
[TRAFODION-1955] JNI optimization at the time of compilation Additional changes and bug fix Removed ByteArrayList JNI class

  1. … 2 more files in changeset.
[TRAFODION-1955] JNI optimization for the methods used at the time of query compilation Changes as per review comments. Added back ByteArrayList JNI but without any references to it to ensure other branches are not broken. Contributors are advised to modify their code to avoid use of this class.

  1. … 5 more files in changeset.
[TRAFODION-1955] JNI optimization at the time of compilation

Removed ByteArrayList class and used array of byte array instead.

Only one JNI to java transistion is done to getRegionStartKeys

and getRegionEndKeys at the time of query compilation

Also fixed an issue with lib_mgmt Makefile.

  1. … 15 more files in changeset.
[TRAFODION-1421] Implement parallel scanner primitive new CQD: hbase_dop_parallel_scanner - default 0.0 = disabled - from 0.x to 1.0 -> the DOP will be dynamically assessed as a percentage of region to scan. If 10 region, and cqd is .3 (30%), then 3 threads will be used... - from 2.0 and above, will take ceiling, and use it as fixed number of threads. Advantage of parallel scanner parallelism over ESP is on resource consumption. Parallel scanner is just creating threads, while ESP are full proceses wth high memory consumption. For now, this feature is just a primitive that can be manually exercised. The next obvious step is to make the compiler aware of it and pick it over ESP DOP when appropriate...

  1. … 21 more files in changeset.