qmscommon

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-3280] Reduce path length in Trafodion for improved performance and scalability

There are many places in Trafodion code getenv is called repeatedly. Cleaned up code so that

repeated getenv calls are avoided during prepare and execute operations.

  1. … 11 more files in changeset.
[TRAFODION-3225] Obscure cores seen in RMS and logger related code when Trafodion is stressed

Process de-registration is now logged to understand the spate of cores seen during

stress testing

Cleanup dangling semaphore incorrectly detected the problem due to pid recycling and

hence de-registered the process and its query fragment from the shared segment

When the RMS semaphore is held for more than 10 seconds, it is assumed to be in a

deadlock situation and hence the process holding the semaphore is core dumped for

further analysis.

Create a global logger instance instead of a function static variable

to help in debugging when problems with QRlogger is reported.

  1. … 6 more files in changeset.
Merge remote branch 'origin/pr/1503/head' into merge_1503

  1. … 8 more files in changeset.
Performance changes for LOB and increasing limit for larger varchars/bobs to use with batch input.

  1. … 17 more files in changeset.
Merge PR 1417 [TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 8 more files in changeset.
Reworks in response to concerns from Hans

  1. … 3 more files in changeset.
[TRAFODION-2927] Refactor and improve UPDATE STATISTICS logging

  1. … 12 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 26 more files in changeset.
JIRA TRAFODION-2731 CodeCleanup: Phase 6. Cleanup of obsolete/unused cqds.

  1. … 25 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 4. Remove legacy/obsolete pragmas

  1. … 391 more files in changeset.
lcov: commit #1

  1. … 324 more files in changeset.
code cleanup, commit #1

  1. … 128 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 459 more files in changeset.
[TRAFODION-2727] Memory leak in the compiler part of the code in Trafodion

Compiler objects were refactored to use the heap infrastructure to ensure

that it is accounted and allocated within the trafodion memory management.

The commonly used NAType and derived classes are mostly created via heap,

but the heap was not passed in the constructor. The embedded objects

used the system heap though the container object is allocated in the compiler

heap. This could result in memory leak when the container object is

destroyed.

  1. … 63 more files in changeset.
[TRAFODION-2596] Improve the log4j and log4cxx infrastructure in Trafodion

Removed setting the root logger to be the appender logger in most of

the configuration files. This would allow T2 driver applications

to have control over the logging in their layer to its preferred way.

But, the logging in SQL engine layer will be controlled by the Trafodion

engine itself.

Changed the code so that the logging location and the file name is

picked up from the config file.

  1. … 8 more files in changeset.
[TRAFODION-2596] Improve the log4j and log4cxx infrastructure in Trafodion

The following changes are made in the way Trafodion logs the messages:

Writes to a log file based on a component or set of components.

C++ part of the code base:

Component Default log file Configuration File

TM tm_<nid>.log log4cxx.trafodion.tm.config

SSCP sscp_<nid>.log log4cxx.trafodion.sscp.config

SSMP ssmp_<nid>.log log4cxx.trafodion.ssmp.config

All SQL processes trafodion.sql_<nid>.log log4cxx.trafodion.sql.config

mxosrvr

sqlci

tdm_arkesp

tdm_arkcmp

tdm_udrserv

Java part of the code base

TM trafodion.dtm.log log4j.dtm.config

SQL trafodion.sql.java.log log4j.sql.config

By default, the log level is set to INFO for most of the cases. When a message

dominates the log file and if doesn't add value in the current level, it will be

changed to the higher level in the hierarchy. This should help to improve the

usefulness of the log file at the default INFO level.

The existence of an environment variable TRAF_MULTIPLE_SQL_LOG_FILE will revert back

to the old way of logging into multiple files. Then, the configuration file

log4cxx.trafodion.masterexe.config will be used.

Currently, RollingFileAppender appender is used for both modes. Different config files

are used to change this appender when the need arises.

log4cxx.trafodion.udr.config and log4cxx.trafodion.lob.config are removed.

log4j.hdfs.config used as the config file for SQL is renamed to log4j.sql.config

Foundation components logging will be revamped later.

This closes #1070

  1. … 18 more files in changeset.
[TRAFODION-2594] Trafodion logs the same message twice in most of its log4j and log4cxx logs [TRAFODION-2420] RMS enhancements [TRAFODION-2478] Memory monitor Thread enhancements

a) Changed the configuration to have log4j.additive property set to false

b) ScratchIOSize is initialized so that this counter doesn't show invalid

value

c) Cleaned up memory monitor concept and added code to log the memory monitoring

counters to sccp log file every 5 minutes or when there is a pressure difference

  1. … 9 more files in changeset.
Merge remote branch 'origin/master' into TRAFODION-2001

Conflicts:

core/sqf/monitor/linux/config.cxx

core/sqf/monitor/linux/pnodeconfig.cxx

core/sqf/monitor/linux/pnodeconfig.h

core/sqf/monitor/linux/reqqueue.cxx

core/sqf/monitor/linux/shell.cxx

core/sqf/sql/scripts/gensq.pl

core/sqf/sql/scripts/sqconfig

core/sqf/sql/scripts/sqconfig.sample

core/sqf/sql/scripts/sqgen

core/sqf/sql/scripts/sqnodestatus

  1. … 28 more files in changeset.
[TRAFODION-2422] populateSortCols was flagged as major perf offender during profiling the reason being an unbound iterative string search on huge string. Added a minor fix: When not setting category in log conf file, extra code is executed, potentially impacting performance

  1. … 1 more file in changeset.
convert all MY_SQROOT to TRAF_HOME

  1. … 294 more files in changeset.
Merge remote branch 'origin/master' into TRAFODION-2001

Conflicts:

install/installer/addNode_step2

  1. … 8 more files in changeset.
[TRAFODION-2296] Consistent error reporting in abort, commit transaction. Changes to wait for all RS to respond when there is an exception in abort and commit transaction requests.

When there is an error returned from commit or rollback transaction, the details of the error can be obtained in the following log files

a) $MY_SQROOT/logs/tm_.log in the node that issued this request contains the error message as seen by the TM process in the JNI side. This event may not have transaction id.

b) $MY_SQROOT/logs/trafodion.dtm.log contains more info about this error with the transaction id. These events are logged from java side of TM.

c) In the region server logs of all the regions that participated in the transaction.

These exceptions are visible as error code on the client side. To get the details about the exception, the above logs need to be browsed.

Fixes to avoid tm core upon double delete of incoming messages to it

This closes #768

  1. … 7 more files in changeset.
Merge remote branch 'origin/master' into TRAFODION-2001

  1. … 3 more files in changeset.
[TRAFODION-2262] Mxosrvr or java core with the stack trace pointing to log4Cxx functions

The Logger repository needs to be initialized. The method QRLogger::initLog4cxx

initialized the commonLogger infrastructure also.

PR #745 merged to fix this issue exposed a problem in SSMP. This

caused SSMP to dump core when a node is brought down.

SSMP should have create the connection to SSCP on that node.

  1. … 3 more files in changeset.
Post merge commit. All files here relate to NAList

  1. … 63 more files in changeset.
[TRAFODION-2261] Mxosrvr or java core with starting from Java_org_trafodion_jdbc_t2_SQLMXConnection_close [TRAFODION-2262] Mxosrvr or Java core with the stack trace pointing to log4Cxx functions

IpcServerClass was not getting destructed correctly. When the CLI context is deleted, the ESP

server class object an IpcServerClass instance that manages the ESP server started

in the CLI context are also destroyed. Fixed the IpcServerClass and IpcServer destructors

so that there is no memory corruption.

Log4Cxx infrastructure in SQL was recursively attempting to log the messages when there is an

issue with it. This was causing stack corruption.

Also added a code to initialize the log4cxx infrastructure if it is not done already upon

the first CLI call.

  1. … 9 more files in changeset.
Fix to sqlci abort on added node.

Removed unneeded files lacking copyrights Fixing hive/EXPECTED009, a result changed sdue to new heap sort algo.

  1. … 50 more files in changeset.
Most of the Trafodion Java source files are built through Maven, using projects DCS, REST, HBase-trx and SQL. A few files remain in the core/sql/executor and core/sql/ustat directories that are built through javac commands in the Makefile and helper files. Also, these files don't conform to common naming standards and therefore are a bit harder to debug with tools like Eclipse.

I would like to do the following:

1. Move these remaining Java source files under core/sql/src,

where the UDR-related files already are.

2. Include the class files in trafodion-sql-1.2.0.jar, the

deliverable of the core/sql Maven project.

3. Change the package name to org.trafodion.sql for all these files.

4. Remove the trafodion-HBaseAccess-1.2.0.jar file (content

will be included in trafodion-sql-1.2.0.jar). This jar file

used to be built with a Makefile.

5. Make some source code changes to reflect the new package names

and to avoid calling private method of class

org.apache.hadoop.hive.ql.io.orc.OrcStruct.

6. Remove Makefile rules for compiling Java files.

Also, this change includes a change of the component name for the

event logger. We use SQL.COMP now, to make analyzing log messages

for Trafodion easier. Venkat asked for this change.

  1. … 44 more files in changeset.
JIRA TRAFODION-1488

Fixed several issues related to logging:

log4cxx.trafodion.masterexe.config changes:

- The timestamp was not being reported in the log4cxx files

- Messages were being displayed twice

- Turned on debug logging for privilege manager events

SQL logging mechanism:

- Messages were not being displayed when expected

Regression test logging:

- changed TEST009 to log hive requests to a rundir log. This test is

failing during daily runs and this change will help isolate the issue.

  1. … 8 more files in changeset.