Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-3272] Change installer settings for TRAF_LOG, TRAF_CONF, TRAF_VAR

Changes python-installer and ambari-installer.

Installers set TRAF_CONF to /etc/trafodion/conf

Installers allow user to specify TRAF_VAR, but defaults to /var/lib/trafodion

Installers allow user to specify TRAF_LOG, but defaults to /var/log/trafodion

Changes DCS conf directory from DCS_INSTALL_DIR/conf to TRAF_CONF/dcs

Changes REST conf directory from REST_INSTALL_DIR/conf to TRAF_CONF/rest

  1. … 53 more files in changeset.
[TRAFODION-2626] Change path of sqconf, ms.env, sqcert, udr and related files

This changeset covers the remaining instances of hard-coded paths based

on TRAF_HOME for dynamic files that should be in specific paths such as

TRAF_CONF or TRAF_VAR.

All files referenced via TRAF_HOME should be read-only software, not written

during run-time.

  1. … 59 more files in changeset.
[TRAFODION-3234] Refactor hive meta calls

Hive similarity check was detecting if there is a change in the hive

directory timestamp only. It has been changed to detect the DDL changes too.

Cleaned up code further and removed all the unwanted code.

  1. … 16 more files in changeset.
[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

Fixed a bug where createTime is not converted into millisecs before comparing with directory timestamp in millisecs

  1. … 1 more file in changeset.
[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

Possible fix for check-PR failures.

  1. … 2 more files in changeset.
[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

There was yet another method that called HiveMetaStoreClient.listPartitions API. This method is

called whenever the hive table entry is picked from the NATable cache.

Replaced listParitions call with Trafodion specific HDFSClient.getHiveTableModificationTS call

that is less expensive.

HiveMetaStoreClient is now made as ThreadLocal variable to ensure that HiveMetaStoreClient methods

can be invoked in a thread safe manner.

  1. … 6 more files in changeset.
[TRAFODION-3234] Add support for hive partitioned tables

Refactored the interaction with hive metastore to use the efficient APIs to obtain the necessary

information to construct the hive table descriptors. This refactoring is expected to reduce the

memory requirements, efficient and high performing even when there are thousands of partitions.

  1. … 14 more files in changeset.
[TRAFODION-3223] Don't scale down for non-Puts when estimating row counts

  1. … 2 more files in changeset.
[TRAFODION-2626] Make logs directory location configurable

Makes path to log directory independent of TRAF_HOME. The default location

is still $TRAF_HOME/logs. The DCS and REST log directories are now

sub-directories of the main TRAF_LOG directory.

  1. … 71 more files in changeset.
[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Fix for hive/TEST006 failure in daily build

Merge PR 1702 [TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Fix for the following exception seen while accessing hive sequence file with the new implementation

*** ERROR[8447] An error occurred during hdfs access. Error Detail: SETUP_HDFS_SCAN java.util.concurrent.ExecutionException: java.nio.BufferOverflowException

java.util.concurrent.FutureTask.report(FutureTask.java:122)

java.util.concurrent.FutureTask.get(FutureTask.java:192)

org.trafodion.sql.HDFSClient.trafHdfsReadBuffer(HDFSClient.java:424)

org.trafodion.sql.HdfsScan.trafHdfsRead(HdfsScan.java:215) Caused by

java.nio.BufferOverflowException

java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)

org.trafodion.sql.HDFSClient.sequenceFileRead(HDFSClient.java:301)

org.trafodion.sql.HDFSClient$HDFSRead.call(HDFSClient.java:217)

java.util.concurrent.FutureTask.run(FutureTask.java:266)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:748) [2018-08-21 15:56:00]

Merge branch 'master' of github.com:apache/trafodion into trafodion-3110

[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

LOB: Extract lobtofile() to a hdfs file returns 8442 error

Implemented the missing code to support this functionality via the new

implementation.

  1. … 8 more files in changeset.
[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Ensured that split of non-compressed sequence file works. In case of compressed

sequence filee, an exception is thrown when the file is split.

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Changes possibly fix the hive/TEST006 failure in daily build.

[TRAFODION-3180] At times establishing a JDBC/ODBC connection takes observably long time

Analysis revealed that the mxosrvr process in connecting state was attempting to open the

ssmp process on the node for a non-unique query as part of establishing connection.

The ssmp process has many ports in CLOSE_WAIT state. It looks like the client happens

to hit on a port that is in CLOSE_WAIT state. The port transitions to ESTABLISHED state

after some time. Hence the connection was taking a longer time.

The mxssmp process keeps the port in CLOSE_WAIT because the socket wasn't closed on the

server side when client exits gracefully as well as abruptly. The seabed layer in

Trafodion doesn't handle more than one open to a process in a correct way. I have changed

the IPC infrastructure in SQL to ensure that the ssmp process is opened only once

in mxosrvr process.

The API msg_get_phandle opens the process with the given name to obtain the handle. This API

is now replaced with XFILENAME_TO_PROCESSHANDLE_

  1. … 3 more files in changeset.
[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Fix to resolve the issue highlighted in the review comment

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

  1. … 3 more files in changeset.
TRAFODION-3086 DDL on Hive objs: use database and msck support

-- traf "set schema hive.<sch>" now behaves the same as Hive "use <db>".

1) <sch> is validated to exist at set schema time.

2) <sch> is set in Hive environment for that session. Any unqualified

object reference in ddl uses that default schema (database)

-- added support for msck (meta store check) command.

This command repairs database by refreshing untracked files/partitions

in hive metadata.

  1. … 14 more files in changeset.
[TRAFODION-3126] Refactored HDFS client implementation should also support Alluxio file system

Alluxio doesn't support direct ByteBuffer access. Circumvented

this problem by using non-direct ByteBuffer to read hdfs files

when it belongs to Alluxio file system.

No need to change the default setting of USE_LIBHDFS for Alluxio to work.

  1. … 3 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

Used a single CQD USE_LIBHDFS to switch to older implementation of

using libhdfs for both hdfs scan and LOB access. The CQD USE_LIBHDS_SCAN is

retired.

In addition fixed the following:

1. Fixed a memory leak of LOB Heap structures

2. Possible fix for memory corruption at times

3. Avoid unnecessary creation of LOB threads when not needed

  1. … 23 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

This feature is enabled by default. To disable, set a variable USE_LIBHDFS=1 in

$TRAF_HOME/etc/ms.env and restart the trafodion cluster.

This feature includes the following:

1. Uses single FSDataInputStream for each LOB column in a query as

opposed to the opening the hdfs file for every row.

2. Uses FSDataOutputStream to write the lob data but closes it

immediately to allow concurrent writes to the hdfs file. HDFS supports

a single writer at a time. Need to conform if multiple writes can

be done without the need for RMS lock feature.

3. Improved error messaging that displays the java exception stack to the

end user.

4. LOB worker threads are no longer created

  1. … 12 more files in changeset.
nanosecs precision for timestamp/interval datatypes

  1. … 31 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 90 more files in changeset.
  1. … 2 more files in changeset.
[TRAFODION-2982] JNI HDFS interface should support varied sized large buffers for read/write

Changed the interim byte array size to be of int type to accoumdate

at least 64MB as per review comments.

  1. … 10 more files in changeset.
[TRAFODION-3065] Trafodion to support compressed Hive Text formatted tables

Compressed text files are now supported via the new implementation using

HDFS java APIs. When the hadoop is not configured to support a particular type

of compression, an error is thrown.

[TRAFODION-2982] JNI HDFS interface should support varied sized large buffers for read/write

A new CQD HDFS_IO_INTERIM_BYTEARRAY_SIZE_IN_KB is introduced to chunk

the read and write when byteArray is involved.

  1. … 15 more files in changeset.
use coprocessor to set storage policy

  1. … 6 more files in changeset.
[TRAFODION-3059] remove "mt_" column family