JavaObjectInterface.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-2626] Make logs directory location configurable

Makes path to log directory independent of TRAF_HOME. The default location

is still $TRAF_HOME/logs. The DCS and REST log directories are now

sub-directories of the main TRAF_LOG directory.

  1. … 72 more files in changeset.
Disabled INSERT thru 'process hive statement'

It return an error on HDP platform. Once the issue is fixed,

it will be enabled.

Also, merged remote branch 'origin/master' into ansharma_hiveddl_br

Conflicts:

core/sql/executor/JavaObjectInterface.cpp

core/sql/executor/JavaObjectInterface.h

  1. … 7 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 90 more files in changeset.
[TRAFODION-3095] Trafodion SQL process should abort when it encounters OOM condition in java

  1. … 1 more file in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or error in

all the utility commands except load. In case of load, the ComDiagsArea

is allocated in advance to report error rows count.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

[TRAFODION-3017] Simplify the hive client access in Trafodion

Hive Client functions are now moved to a new file HiveClient_JNI.h and

HiveClient_JNI.cpp. Most of the HiveClient functions are static functions

allowing to use HiveClient in Trafodion with ease.

  1. … 33 more files in changeset.
[TRAFODION-2821] Trafodion core code base needs to be thread safe

Java exceptions thrown while calling the java methods from JNI in

Trafodion are stored in the current context. However in a multi-threaded

ESP environment, these exceptions should be stored in thread specific

variable to enable error handling to be thread safe. Otherwise, the JNI

errors could be overwritten by the JNI errors from another thread.

Also streamlined JNI error handling by extending the getExceptionDetails()

methods to log the errors also.

Incorporated error handling in SequenceFileReader JNI methods.

  1. … 22 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 41 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text format hive tables

Part-6 changes

Removed the dependency of libhdfs while assigning ranges for hdfs scan at runtime

and for sample file creation during bulk load

  1. … 12 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 27 more files in changeset.
[TRAFODION-2836] Enable Trafodion SQL processes to configure garbage collector in its embedded JVM

Provided a way to configure the garbage collector when the JVM is created

in EsgynDB processes.

[TRAFODION-2682] JVM startup options like heap are not passed correctly

Character arrays used to build the options went out of scope before

they were passed to the JNI layer to create the JVM.

Selva also requested that we use the default heap size defined in

our code instead of the JVM default method of 1/4 the physical

memory when a value <= 0 is passed in env var JVM_MAX_HEA_SIZE_MB.

[TRAFODION-2637] Library management with Cloudera parcels

Also in this commit:

[TRAFODION-2581] Combine JVM startup code of SQL executor and UDR server

We have library management SPJs that allow a user to deploy library

files from a client machine. These SPJs make use of the pdcp command

that is installed on many clusters, but not always. The new method

uses temporary HDFS files and a TMUDF to distribute the library

file to all nodes of the cluster.

For now I left the old PUT method in place and added a new PUTFILE

method. We can remove the old method later.

A few other things fixed in the process:

- Make sure that when a UDR server calls the executor and uses a JVM,

we share the same JVM. To do this, we need to use the same startup

method, and I needed to change the executor startup method to accept

the extra options used for the UDR server.

- Got rid of the "HBase compatibility mode" in JavaObjectInterface.

- A common method to execute SQL in the library management code.

- removed the obsolete init_libmgmt.sh file.

- We have special code that makes sure a UDR can request to run in

parallel, including a request to run on all nodes. Expanded this

special code from predefined UDFs to any UDF.

  1. … 25 more files in changeset.
[TRAFODION-2566] Reduce the virtual memory allocated in Trafodion processes with JDK1.8

Added two new configurable JVM options -XX:CompressedClassSpaceSize=128m and

-XX:MaxMetaSpaceSize=128m.

You can modify the default value of these options using environment variables

JVM_COMPRESSED_CLASS_SPACE_SIZE and JVM_METASPACE_SIZE respectively.

With this change the virtual memory of any SQL processes and tm is reduced by 1GB.

In JDK1.8 this space is used in lieu of permanent generation. Hence, I reduced the

default Java Objects Max heap size for SQL master, ESP and arkcmp processes to 256 MB from 512MB.

The udrserv process continues to have 512MB of java objects space.

  1. … 3 more files in changeset.
convert all MY_SQROOT to TRAF_HOME

  1. … 294 more files in changeset.
[TRAFODION-2356] Trafodion process can dump core at times because JNIEnv is not initialized

  1. … 2 more files in changeset.
[TRAFODION-1988] Better java exception handling in the java/JNI layer

Changes to fix the regression failure

  1. … 1 more file in changeset.
[TRAFODION-1988] Better java exception handling in the java/JNI layer

The stack trace of the exception displayed from JNI side now contains the

cause of the exception displayed in a nested way.

  1. … 3 more files in changeset.
Changes to move hdfFs handing to the context globals level and remove all disconnects from hdfs from the various components.

  1. … 27 more files in changeset.
AFODION-1492] Allow specification of MaxHeapSize for JVM started by UDR server

This is a fix for the second of the two issues described in this JIRA. The first

has been addressed by narendra in a separate checkin.

In this checkin we allow multiple methods to specify JVM startup options for the

UDR server process. There are three methods applied in this order with the last

specification for each property superceeding previous specifications.

1. A default default of 512MB for UDR JVM's max heap size

2. Specify any desired property including -Xmx in a new configuration file

3. Specify any desired property using TRAF_UDR_JAVA_OPTIONS env variable

The new configuration file is at $MY_SQROT/conf/trafodion.udr.config. It has

detailed comments on how properties should be specified in the file. The location

(including name) of the configuration file can be overridden with the env variable

TRAFUDRCFG.

The TRAF_UDR_JAVA_OPTIONS env variable must be specified in $MY_SQROOT/etc/ms.env

for it to be visible to the UDR server. Multiple properties can be separated by a

'\t' (TAB) chracater or the '\n' (newline) character

  1. … 4 more files in changeset.
Part 2 of changes to update license text in Trafodion source

This check-in updates license text in all source files in the

core/sql directory. Remaining directories will be done in

later check-ins.

  1. … 3147 more files in changeset.
TRAFODION-38 rework for comments and release build failure.

Sorry for taking so long, Steve first had to give me a crash course

on how to navigat through the Jenkins logs to find the cause of the

failure.

  1. … 1 more file in changeset.
TRAFODION-38 First working TMUDFs in Java

Also addresses:

TRAFODION-7 Have a .project file to be able to import source code into Eclipse

TRAFODION-35 Add a method that allows UDR writers to debug their UDRs on a release build

Summary of changes:

TRAFODION-7:

Eclipse project files can now be generated with the following make target in the core directory:

cd core

make eclipse

This will print out instructions on how to import the generated projects into Eclipse.

TRAFODION-35:

Java UDRs are debugged with the following CQDs:

cqd UDR_JAVA_DEBUG_PORT

cqd UDR_JAVA_DEBUG_TIMEOUT

The best way to do this is with the sqlci tool. Set UDR_JAVA_DEBUG_PORT before executing any Java UDR. Then, after executing the

first one, attach a remote Java debugger like jdb or Eclipse. To debug the very first invocation, set UDR_JAVA_DEBUG_TIMEOUT to

a time in milliseconds and attach the debugger within that time.

This debugs only the runtime part of the UDR. To debug the compiler interface, set this environment variable before starting

sqlci:

export JVM_DEBUG_PORT=12345

For both cases, here are two more things to know: First, this only works if you are connected as the DB__ROOT user (running

sqlci without special options runs as DB__ROOT). Second, if you want to debug parallel instances of UDRs, choose a multiple of

1000 for the port number. Trafdion with then add <process id> mod 1000 to the port and put all the parallel instances into Java

debugging mode on these different ports.

TRAFODION-38:

Integrating Java code written by Suresh and Pavani with the engine code for Java TMUDFs.

Addressing a few Java issues flagged by Eclipse.

Adding Java TMUDF tests to core/sql/regress/udr/TEST001.

  1. … 70 more files in changeset.
Move core into subdir to combine repos

    • -0
    • +513
    ./JavaObjectInterface.cpp
  1. … 10768 more files in changeset.
Move core into subdir to combine repos

    • -0
    • +507
    ./JavaObjectInterface.cpp
  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

    • -0
    • +513
    ./JavaObjectInterface.cpp
  1. … 10837 more files in changeset.