Clone Tools
Constraints: committers
Constraints: files
Constraints: dates
[TRAFODION-3281] Remove virtual nodes configuration

Changed workstation mode (aka install_local_hadoop aka ILH) to use

single-node configuration instead of virtual nodes.

* ILH script generates sqconfig for current node.

* template sqconfig file moved to config template dir.

* Updated expected files accordingly.

** Filed jira [TRAFODION-3283] for bug found.

Removed setting and use of SQ_VIRTUAL_NODES variable.

Removed virtual nodes support in sqgen.

  1. … 18 more files in changeset.
RMS infrastructure now reads the configured pid Max from /proc/sys/kernel/pid_max but limits it to a maximum of 128K. It can be overridden by "PID_MAX" environment variable in ms.env.

If the PID_MAX variable is set to 0, 64K is assumed to be configured pid max.

If the PID_MAX variable is set to less than 32K, then 32K is assumed to be configured pid max.

When a SQL process has a pin greater than the configured pid max, an error message

"Pid , is higher than the configured pid max <configured_pid>" is written to

the SQL log file and disables RMS for those processes. Node won't be brought down.

"Get Statistics for rms " shows the the configured pid max.

The environment variable to set the size of RMS shared segment size is now


genms change for controlling pid_max and rms shared segment size

  1. … 13 more files in changeset.
JIRA TRAFODION-2731 CodeCleanup: Phase 6. Cleanup of obsolete/unused cqds.

  1. … 25 more files in changeset.
[TRAFODION-2420] RMS Enhancements

Refactored the ESP memory management so that all the SQL memory both

heap and space objects are accounted correctly in RMS

With this change, the total SQL memory used by all processes involved

in the query is represented in the counter "SQL Heap WM"

  1. … 4 more files in changeset.
[TRAFODION-2754] Get statistics cores sqlci or mxosrvr at str_sprintf()

str_sprintf is Trafodion specific formatting function. Replaced

str_sprintf with sprint that provides better checking with parameters.

str_sprintf is replaced with sprintf in the statistics output related functions

only. It is advised that the new code should start using the standard formatting


  1. … 3 more files in changeset.
[TRAFODION-2420] RMS enhancements

Added four new counters to provide information about multiple executes

of a prepared statement.

No. of times executed

Min. Execute Time

Max. Execute Time

Avg. Execute Time

  1. … 5 more files in changeset.
[TRAFODION-2631] Disk IO counter is not populated for hdfs/hive IOs

  1. … 5 more files in changeset.
[TRAFODION-2420] RMS enhancements

Fixed a possible corruption issue in Type 2 JDBC driver while

getting the accumulated statistics.

The SE_IO_Bytes is now changed to SE_IO_KBytes. It used to return

in MB and now it is changed to KB.

  1. … 2 more files in changeset.
[TRAFODION-2420] RMS enhancements

New counters introduced as part of BMO

InterimRowCount - Counter to keep track intermediate rows

produced by hash join and for intermediate

runs in sort

phase - BMO Phase for hash-join and sort

get statistics for statement s1, options 'cs' now works.

Removed attaching BMO stats to Non-root Paritial and leaf partition hash group by operators.

These operators are non-blocking BMOs.

Introduced 3 phases for hash grby operator to be visible in RMS Stats

options 'sl' now works with get statistics for qid.

The relevant new counters are now part of accumulated statistics type.

Extended the support upto 512 opeartors from 256 operators in RMS

Fixed both Type 2 and mxosrvr code to obtain all the externalized counters in accumualted stats correctly.

  1. … 27 more files in changeset.
[TRAFODION-2594] Trafodion logs the same message twice in most of its log4j and log4cxx logs [TRAFODION-2420] RMS enhancements [TRAFODION-2478] Memory monitor Thread enhancements

a) Changed the configuration to have log4j.additive property set to false

b) ScratchIOSize is initialized so that this counter doesn't show invalid


c) Cleaned up memory monitor concept and added code to log the memory monitoring

counters to sccp log file every 5 minutes or when there is a pressure difference

  1. … 9 more files in changeset.
[TRAFODION-1906] Dop is reported incorrectly in RMS operator stats

DOP should be set to 1 in the ExOperStats constructors when TDB is passed

in. DOP is reset to 0, when the statistics area in the ESP processes

are shipped to higher tier process to avoid double counting.

DOP is restored back to 1 before the next execution again.

scratchIOSize is now initialized to 0 in the BMOStats constructor

to avoid junk value seen at times.

  1. … 5 more files in changeset.
[TRAFODION-2420] RMS enhancements

Improved BMO operation accounting. Two new counters are added

a) scratchIOSize - Size in KB of the scratch IO

b) scratchIOMaxTime - Time in microseconds taken by any long pole

ESP instance to do scratch IO operations

The ScrBufSize, ScrBufRead and ScrBufWritten are no longer displayed

in the formatted get statistics command outputs.

Reduced the default value of CQD HBASE_NUM_CACHE_ROWS_MAX to 1024 from

10000 to reduce stress on the client JVM memory.

Disabled the memory pressure triggering in the BMO operators by increasing

the threshold GEN_MEM_PRESSURE_THRESHOLD to 10000 from 100. The memory

pressure detection code was inadvertently enabled when the cap to limit

memory pressure constant to 100 was removed.

Memory pressure triggers in BMO operators will be enabled later when

we understand the memory pressure detection better.

  1. … 14 more files in changeset.
[TRAFODION-2420] RMS enhancements Introduced 2 new counters HeapWM: - "SQL Heap WM" - Tracks the high water mark of the used memory including space buffer and bmo operator memory. It is enough to monitor this counter alone to track the memory utilization of the query.

scrIOTime: "Scr. IO Time" - Tracks the IO time taken to read/write to

the scratch files at BMO operator level

Introduced a 'Single line Format' option to display BMO stats and SE stats

in a single line format to enable exporting the output to Excel easily.


Removed and cleaned up the code further

Also fixed memory corruption issues seen with GET STATISTICS command.

  1. … 16 more files in changeset.
[TRAFODION-2854] Improve SET STATISTICS command to display the information in new format

Currently SET STATISTICS ON | OFF alone is supported. It is improved to take some more options




This will display pertable, progress, operator level statistics for the these options respectively.

SET STATISTICS ALL will display both progress and operator level statistics.

These commands are valid both in sqlci and trafci.

  1. … 6 more files in changeset.
Merge branch 'master' of git://git.apache.org/incubator-trafodion into topn



  1. … 1 more file in changeset.
[TRAFODION-2259] Add TopN stats counter and realign BMO stats.

  1. … 7 more files in changeset.
Squashed commit of the following:

commit f924372b8a8d1f395e227ace804bf32ea18e2437

Author: selvaganesang <selva.govindarajan@esgyn.com>

Date: Fri Jan 6 03:27:53 2017 +0000

[TRAFODION-2420] RMS enhancements

Fix for core/TESTRTS failure

commit 461cdc879b887de437adee253fbeb74eef3204e3

Author: selvaganesang <selva.govindarajan@esgyn.com>

Date: Fri Jan 6 00:02:31 2017 +0000

[TRAFODION-2420] RMS enhancements

Fixed the issue pointed by Steve

This closes #PR 904

  1. … 1 more file in changeset.
[TRAFODION-2400] RMS enhancements

Removed FsDp2MsgStats and ExeDp2Stats

Introduced a new class ExeSEStats to cater the different

storaage engine supported by Trafodion.

The counters pertaining to storage engine were not getting

in GET STATISTICS FOR QID ... ACCUMULATED mode. Fixed this issuue

Removed references to obsolete Measuree concepts

  1. … 15 more files in changeset.
[TRAFODION-2420] RMS enhancements and cleanup

Removed the following classes pertaining to obsolete storage engine DP2.




And also removed some counters belonging to this obsolete storage engine

in other classes.

  1. … 7 more files in changeset.
jira TRAFODION-2234 turn aligned format on, phase 1

This is phase 1 of aligned format change.

Dev regressions now run with aligned format tables.

To test both hbase and aligned format, some tests have explicit

specification to create hbase format tables.

Tests that test for features that are currently only available

with hbase format (like pushdown sel expr) create hbase format tables.

Many expected files have been updated to reflect aligned format,

mostly in showddl and explain output.

  1. … 35 more files in changeset.
[TRAFODION-2134] Change the overflow_mode to 'DISK' by default

  1. … 1 more file in changeset.
[TRAFODION-1823] ESP idle timeout does not kick in, leading to too many ESPs on the system

Close message from the master was not sent to ESP by the platform

agnostic messaging layer in Trafodion. Fixed this bug and then ESP idle time

works as expected.

ESP_IDLE_TIMEOUT can now be given as a CQD. Internally, it would be

changed to SET SESSION DEFAULT and used by the IPC layer of Trafodion.

ESP_IDLE_TIMEOUT is set to 30 minutes by default in the CQD too.

Added test in core/TESTRTS to test ESP_IDLE_TIMEOUT

  1. … 5 more files in changeset.
[TRAFODION-1771] Enabled TESTRTS again TESTRTS was failing at times with error 8926. This is due to absence of ROOT_OPER_STATS in statistics area, but the stats descriptor was incorrectly indicating that there is a ROOT_OPER_STATS.

In future, I will get rid of setting the accumulated statistics as part of

cleanup effort of the statistics collection.

  1. … 2 more files in changeset.
Fix issue where optimization on key column addition should be turned off for MDAM scans, and update EXPECTEDTESTRTS to showcase the new value expected as byte read showing a 53% improvement over previous code

  1. … 1 more file in changeset.
ESP-colocation fixes (Ravisha and Qifan)

  1. … 6 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.