Clone
 

selvaganesang <selva.govindarajan@esgyn.com> in Trafodion

[TRAFODION-3260] SSMP may wait 3 seconds before handling requests

Encapsulated the changes in the commit 2ee272b325d021 to Ipc layer.

[TRAFODION-3274] At times sqlci or any other SQL process fails to come up and dumps core

It looks like there is a race condition in C++ main function prologue while

initializing the embedded global objects and the stdin, stdout and stderr file

descriptors.

File descriptor of value 2 is returned for epoll_create(). But, the error redirection

code possibly used this fd to redirect overriding the epoll fd.

This caused epoll_ctl to return EINVAL resulting in core dump of sql process.

Changed the global object gv_sock_ctlr to a global object pointer.

    • -22
    • +36
    /core/sqf/src/seabed/src/sock.cpp
    • -4
    • +0
    /core/sql/executor/ex_split_bottom.cpp
[TRAFODION-3234] Refactor hive meta calls

Hive similarity check was detecting if there is a change in the hive

directory timestamp only. It has been changed to detect the DDL changes too.

Cleaned up code further and removed all the unwanted code.

    • -58
    • +1
    /core/sql/executor/ExExeUtilMisc.cpp
    • -22
    • +0
    /core/sql/executor/ExFastTransport.cpp
    • -1
    • +13
    /core/sql/generator/GenFastTransport.cpp
Fix for hive/TEST003 failure. Created a known failure to accommodate the costing change between Jenkins and contributor's work space environments.

    • -46
    • +4
    /core/sql/regress/hive/DIFF003.KNOWN
Fix for hive/TEST003 failure seen in recent builds.

[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

Fixed a bug where createTime is not converted into millisecs before comparing with directory timestamp in millisecs

[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

Removed some more unneeded data modification timestamp check code

    • -68
    • +1
    /core/sql/executor/ExFastTransport.cpp
[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

Fixes for hive/TEST005 failure.

This also made some of the earlier failed test cases in this test suite to return correct results.

Data modification check was unnecessarily done in the scan operator. This has been removed.

    • -41
    • +26
    /core/sql/optimizer/HDFSHook.cpp
    • -72
    • +56
    /core/sql/regress/hive/EXPECTED005
[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

Possible fix for check-PR failures.

    • -16
    • +18
    /core/sql/executor/HiveClient_JNI.cpp
[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

There was yet another method that called HiveMetaStoreClient.listPartitions API. This method is

called whenever the hive table entry is picked from the NATable cache.

Replaced listParitions call with Trafodion specific HDFSClient.getHiveTableModificationTS call

that is less expensive.

HiveMetaStoreClient is now made as ThreadLocal variable to ensure that HiveMetaStoreClient methods

can be invoked in a thread safe manner.

[TRAFODION-3234] Add support for hive partitioned tables

Refactored the interaction with hive metastore to use the efficient APIs to obtain the necessary

information to construct the hive table descriptors. This refactoring is expected to reduce the

memory requirements, efficient and high performing even when there are thousands of partitions.

    • -119
    • +757
    /core/sql/executor/HiveClient_JNI.cpp
    • -10
    • +49
    /core/sql/executor/HiveClient_JNI.h
    • -521
    • +105
    /core/sql/executor/hiveHook.cpp
    • -0
    • +61
    /core/sql/executor/org_trafodion_sql_HiveClient.h
    • -28
    • +18
    /core/sql/regress/hive/EXPECTED003
    • -56
    • +70
    /core/sql/regress/hive/EXPECTED005
    • -1
    • +125
    /core/sql/sqlcomp/CmpDescribe.cpp
[TRAFODION-3225] Obscure cores seen in RMS and logger related code when Trafodion is stressed

Process de-registration is now logged to understand the spate of cores seen during

stress testing

Cleanup dangling semaphore incorrectly detected the problem due to pid recycling and

hence de-registered the process and its query fragment from the shared segment

When the RMS semaphore is held for more than 10 seconds, it is assumed to be in a

deadlock situation and hence the process holding the semaphore is core dumped for

further analysis.

Create a global logger instance instead of a function static variable

to help in debugging when problems with QRlogger is reported.

    • -1
    • +5
    /core/sql/executor/ex_esp_frag_dir.cpp
    • -37
    • +76
    /core/sql/runtimestats/SqlStats.cpp
[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Fix for hive/TEST006 failure in daily build

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Fix for the following exception seen while accessing hive sequence file with the new implementation

*** ERROR[8447] An error occurred during hdfs access. Error Detail: SETUP_HDFS_SCAN java.util.concurrent.ExecutionException: java.nio.BufferOverflowException

java.util.concurrent.FutureTask.report(FutureTask.java:122)

java.util.concurrent.FutureTask.get(FutureTask.java:192)

org.trafodion.sql.HDFSClient.trafHdfsReadBuffer(HDFSClient.java:424)

org.trafodion.sql.HdfsScan.trafHdfsRead(HdfsScan.java:215) Caused by

java.nio.BufferOverflowException

java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)

org.trafodion.sql.HDFSClient.sequenceFileRead(HDFSClient.java:301)

org.trafodion.sql.HDFSClient$HDFSRead.call(HDFSClient.java:217)

java.util.concurrent.FutureTask.run(FutureTask.java:266)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:748) [2018-08-21 15:56:00]

[TRAFODION-3110] LOB: Extract lobtofile() to a hdfs file returns 8442 error

Fixes as per the review comments for the commit ba00576e1f47a9f0e0b3f344da8742aeecbc3ce

Also took care of truncate option.

Merge branch 'master' of github.com:apache/trafodion into trafodion-3110

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Added a known diff file till TRAFODION-3185 is taken care of/

    • -0
    • +114
    /core/sql/regress/hive/DIFF006.KNOWN
Fix for failure with check-PR of commit 185ff850ac815d68712ee58ded493fff7f8ae8d3

SYSKEY can be 16 digits long

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Disabled parallel plan in hive/TEST006 that could cause spliting a compressed sequence file

[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

LOB: Extract lobtofile() to a hdfs file returns 8442 error

Implemented the missing code to support this functionality via the new

implementation.

    • -3
    • +53
    /core/sql/executor/HdfsClient_JNI.cpp
    • -37
    • +27
    /core/sql/optimizer/HDFSHook.cpp
    • -14
    • +29
    /core/sql/regress/executor/EXPECTED130
[TRAFODION-3180] At times establishing a JDBC/ODBC connection takes observably long time

When the sock stream is in use, socket close is kept pending in a queue till it can

be closed. But, the sockets from this queue is never closed. Added code

to close the sockets from this queue at the time of creating the socket stream next

time.

    • -0
    • +1
    /core/sqf/src/seabed/src/sockstream.cpp
[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Ensured that split of non-compressed sequence file works. In case of compressed

sequence filee, an exception is thrown when the file is split.

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Changes possibly fix the hive/TEST006 failure in daily build.

[TRAFODION-3180] At times establishing a JDBC/ODBC connection takes observably long time

Fixes as per the review comments

[TRAFODION-3180] At times establishing a JDBC/ODBC connection takes observably long time

Analysis revealed that the mxosrvr process in connecting state was attempting to open the

ssmp process on the node for a non-unique query as part of establishing connection.

The ssmp process has many ports in CLOSE_WAIT state. It looks like the client happens

to hit on a port that is in CLOSE_WAIT state. The port transitions to ESTABLISHED state

after some time. Hence the connection was taking a longer time.

The mxssmp process keeps the port in CLOSE_WAIT because the socket wasn't closed on the

server side when client exits gracefully as well as abruptly. The seabed layer in

Trafodion doesn't handle more than one open to a process in a correct way. I have changed

the IPC infrastructure in SQL to ensure that the ssmp process is opened only once

in mxosrvr process.

The API msg_get_phandle opens the process with the given name to obtain the handle. This API

is now replaced with XFILENAME_TO_PROCESSHANDLE_

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

Fix to resolve the issue highlighted in the review comment

[TRAFODION-3171] Refactor Hive sequence file reading to use the new implementation

[TRAFODION-2430] RMS enhancements

RMS now displays 3 numeric values separated by '|' in "Details"

columns in the default output based on the type of stats entry.

These numeric values are VAL2, VAL3, VAL4 of the statistics virtual

table.

Stats Type VAL2 VAL3 VAL4

ROOT_OPER_STATS MessageBytes MessageCount MemoryAllocated

SE_STATS SE_IO_KBytes SE_IO_SumTime ActRowsAccessed

BMO_STATS Scr_IO_Count bmoHeapAllocated ScrFileCount

VAL1 is operCpuTime and TEXT column contains the PID of the process

ExFastExtractTcb is now instrumented with ExHdfsScanStats.

Hdfs scan and access opearations are now properly instrumented with ExHdfsScanStats

counters.

Lob access is now instrumented and is part of the statistics

entry(ExHdfsScanStats or ExHbaseAccessStats) attached with tcb.

Additionally, the following changes are done

1. Removed obsolete ExLobStats

2. Optimized space usage in RMS shared segments for some of the operators

3. A new class ExStorageEngineStats is now created. This class is

type-defed as ExHbaseAccessStats and ExHdfsScanStats.

    • -4
    • +1
    /core/sql/executor/ExExeUtilGetStats.cpp
    • -28
    • +66
    /core/sql/executor/ExExeUtilLoad.cpp
    • -3
    • +18
    /core/sql/executor/ExFastTransport.cpp
    • -594
    • +146
    /core/sql/executor/ExStats.cpp
  1. … 14 more files in changeset.
[TRAFODION-3126] Refactored HDFS client implementation should also support Alluxio file system

Alluxio doesn't support direct ByteBuffer access. Circumvented

this problem by using non-direct ByteBuffer to read hdfs files

when it belongs to Alluxio file system.

No need to change the default setting of USE_LIBHDFS for Alluxio to work.

[TRAFODION-2963] RMS infrastructure now reads the configured pid Max from /proc/sys/kernel/pid_max

Changes to avoid log storm when there is a pid violation

    • -6
    • +2
    /core/sql/executor/ex_esp_frag_dir.cpp
    • -0
    • +15
    /core/sql/runtimestats/SqlStats.cpp
Merge branch 'master' of github.com:apache/trafodion into trafodion-2963

    • -8
    • +12
    /core/sql/executor/ExExeUtilGetStats.cpp
    • -1
    • +21
    /core/sql/runtimestats/SqlStats.cpp