Clone
 

tharaknath capirala <capirala.tharaknath@hp.com> in Trafodion

Repository support in MT server #2

This is the second installment of code for the repository functionality

in the DCS multi-threaded server/T2. Adding query statistics code.

Note:

This code is for compile only for now and not part of any active code path.

Most of the this is existing code from MXOSRVR side and is made to fit

into T2.

There are some temporary to-do comments in the code which shall be

removed eventually.

Change-Id: Iff4a3ec9d87ad6d3ce7c2e1882126368ce9e6d6b

    • -0
    • +18
    /conn/jdbc_type2/native/CSrvrConnect.cpp
    • -908
    • +930
    /conn/jdbc_type2/native/CSrvrStmt.cpp
    • -0
    • +22
    /conn/jdbc_type2/native/CSrvrStmt.h
    • -1
    • +97
    /conn/jdbc_type2/native/CoreCommon.h
    • -3
    • +1
    /conn/jdbc_type2/native/ResStatisticsSession.cpp
    • -0
    • +2537
    /conn/jdbc_type2/native/ResStatisticsStatement.cpp
    • -0
    • +458
    /conn/jdbc_type2/native/ResStatisticsStatement.h
    • -0
    • +125
    /conn/jdbc_type2/native/SrvrCommon.cpp
    • -1334
    • +1339
    /conn/jdbc_type2/native/SrvrOthers.cpp
Repository support in MT server #1

This is the first installment of code to start including repository

functionality in the DCS multi-threaded server/T2. Includes only session statistics code for now.

Note: This code is for compile only for now and not part of any active

code path.

There are some temporary to-do comments in the code which shall be removed eventually.

===================================================================

Patch set #2: Updated with review comments

Change-Id: Ida3689d9c0f029ebb47e6d87e670d0a4a1653480

    • -385
    • +438
    /conn/jdbc_type2/native/CSrvrConnect.cpp
    • -139
    • +159
    /conn/jdbc_type2/native/CSrvrStmt.h
    • -259
    • +363
    /conn/jdbc_type2/native/CoreCommon.h
    • -0
    • +66
    /conn/jdbc_type2/native/PubInterface.h
    • -0
    • +291
    /conn/jdbc_type2/native/PubQueryStats.h
    • -0
    • +57
    /conn/jdbc_type2/native/ResStatistics.cpp
    • -0
    • +87
    /conn/jdbc_type2/native/ResStatistics.h
    • -0
    • +491
    /conn/jdbc_type2/native/ResStatisticsSession.cpp
    • -0
    • +428
    /conn/jdbc_type2/native/ResStatisticsSession.h
    • -1842
    • +1849
    /conn/jdbc_type2/native/SrvrCommon.cpp
    • -147
    • +151
    /conn/jdbc_type2/native/SrvrCommon.h
Partial fix for bug 1449343 and fixes driver core with pyODBC

Fixes issue where the last few bytes of data are lost for nullable columns that

are greater than 32K. This also fixes a core seen on the ODBC driver

side from a query run by HPDSM via pyODBC.

Partially fixes 1449343

Change-Id: I61c1f5a00ffd7da54467e3be288433045fa2951b

Partial fix for bug 1449343 and fixes driver core with pyODBC

Fixes issue where the last few bytes of data are lost for nullable columns that

are greater than 32K. This also fixes a core seen on the ODBC driver

side from a query run by HPDSM via pyODBC.

Partially fixes 1449343

Change-Id: I61c1f5a00ffd7da54467e3be288433045fa2951b

Fix for bug 1438775

The fetch buffer size calculation did not account for varchar indicator

length for columns greater than 32K. The indicator length in this case is 4 bytes instead of 2. Currently, the buffer length calculation was using 2, which resulted in allocating insufficient memory leading to a corruption.

Fixes bug 1438775

Change-Id: Ib16b6644ca3c7f36d96687a33ea36ad4f0ffe903

Fix for bug 1443688

Explain plan is now collected only for non-unique query types and queries that generate stats.

Also, fixed a bug where explain plan was being collected even though the

statistics feature is disabled.

Fixes bug 1443688

Change-Id: I67433083758044e1da0071241e00c4a09e701dbd

    • -0
    • +11
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
Fix for bug 1443561

SQL plan collection enabled by default now.

Fixes bug 1443561

Change-Id: Ibc9cb9e41d37a384a6f0bf4cb34c625ea0a98dd7

Fix for bug 1443561

SQL plan collection enabled by default now.

Fixes bug 1443561

Change-Id: Ibc9cb9e41d37a384a6f0bf4cb34c625ea0a98dd7

Writing packed explain plan to metric_query_table

The packed explain is retrieved using the SQL_EXEC_GetExplainData() CLI

call. This is picked up by the repository watchdog thread and then written to

the metric_query_table using SQL_EXEC_StoreExplainData().

The packed explain could have binary data within it so it could not be

used as is to formulate the insert statement the way we do today with a

stringstream. So, Anoop provided the new CLI call which takes in the plan

and updates the row in the metric_query_table table.

Also, removed writing of sql text and plan during the updates to

metric_query_table since they don't change and is redundant.

Change-Id: I15efc2134b78d0d43000b1b05094aface4d9d0e0

    • -1
    • +7
    /conn/odbc/src/odbc/Common/PubQueryStats.h
    • -23
    • +100
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +1
    /conn/odbc/src/odbc/nsksrvrcore/CSrvrStmt.h
Removing old version 1.0 of log4cpp

Replaced by log4cpp version 1.1.1

Change-Id: I77ee24391e5d9dd737fcf04d6e3065ca7ddd09a6

    • -1
    • +0
    /log4cpp/log4cpp-1.0/include/Makefile.am
    • -527
    • +0
    /log4cpp/log4cpp-1.0/include/Makefile.in
    • -98
    • +0
    /log4cpp/log4cpp-1.0/include/config.h.in
    • -150
    • +0
    /log4cpp/log4cpp-1.0/include/log4cpp/Appender.hh
    • -675
    • +0
    /log4cpp/log4cpp-1.0/include/log4cpp/Category.hh
  1. … 98 more files in changeset.
New and updated repository columns

Repository column changes...

"_REPOS_".METRIC_QUERY_TABLE

Added:

QUERY_STATUS

QUERY_SUB_STATUS --> for future use

----------------------------------------------------

"_REPOS_".METRIC_QUERY_AGGR_TABLE;

Added:

SESSION_START_UTC_TS

AGGREGATION_LAST_UPDATE_UTC_TS

AGGREGATION_LAST_ELAPSED_TIME

TOTAL_DDL_STMTS --> Falls under OTHER category since no

corresponding sql type exists today

TOTAL_UTIL_STMTS

TOTAL_CATALOG_STMTS

TOTAL_OTHER_STMTS

TOTAL_INSERT_ERRORS

TOTAL_DELETE_ERRORS

TOTAL_UPDATE_ERRORS

TOTAL_SELECT_ERRORS

TOTAL_DDL_ERRORS

TOTAL_UTIL_ERRORS

TOTAL_CATALOG_ERRORS

TOTAL_OTHER_ERRORS

DELTA_DDL_STMTS

DELTA_UTIL_STMTS

DELTA_CATALOG_STMTS

DELTA_OTHER_STMTS

DELTA_INSERT_ERRORS

DELTA_DELETE_ERRORS

DELTA_UPDATE_ERRORS

DELTA_SELECT_ERRORS

DELTA_DDL_ERRORS

DELTA_UTIL_ERRORS

DELTA_CATALOG_ERRORS

DELTA_OTHER_ERRORS

Deleted:

AGGREGATION_START_UTC_TS

Updated:

DELTA_NUM_ROWS_IUD

Note: These columns were already added to the insert/update statements as

part of Anoop's earlier commit.

Packed explain plan to follow soon.

Change-Id: I268d1d24a8886ba1f0dc6181e1f0a65e53143fac

    • -7
    • +35
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -0
    • +3
    /conn/odbc/src/odbc/nsksrvrcore/csrvrstmt.cpp
    • -0
    • +2
    /conn/odbc/src/odbc/nsksrvrcore/srvrothers.cpp
Fix for bug #1409229

Changed the error messages reported from verifyPortAvailable() and

listenToPort as warning instead of errors.

Fix for bug #1409229

After discussing with Arvind felt that we should keep the error logging

intact in ListenToPort() otherwise we would be masking any genuine

errors. So, have revertted back the changes in that method.

Change-Id: I445e0784be9d9c6edeb1a002449ae114186056bb

    • -38
    • +75
    /conn/odbc/src/odbc/Common/TransportBase.cpp
    • -25
    • +28
    /conn/odbc/src/odbc/Common/TransportBase.h
Fix core during dcsstop when repository is enabled

When the repository feature is enabled at time MXOSRVR may core during a

dcsstop/sqstop. This is because the repository based thread does get

terminated properly before the sqstop is invoked. Fix is to call the

SyncPublicationThread() during terminateThreads() where the rest of the

threads in MXOSRVR are being stopped.

Moved SyncPublicationThread() before the check for tcp/ip thread.

Change-Id: Ib7b5de187ff02da49974b206832e25218891119c

    • -3
    • +0
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
Fix core during dcsstop when repository is enabled

When the repository feature is enabled at time MXOSRVR may core during a

dcsstop/sqstop. This is because the repository based thread does get

terminated properly before the sqstop is invoked. Fix is to call the

SyncPublicationThread() during terminateThreads() where the rest of the

threads in MXOSRVR are being stopped.

Moved SyncPublicationThread() before the check for tcp/ip thread.

Change-Id: Ib7b5de187ff02da49974b206832e25218891119c

    • -3
    • +0
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
Added option to disable SQL plan collection for user queries

Added a new option that works on top of the new DCS property -SQLPLAN to disable

collection of query plan for user queries.

Cleaned up SessionWatchDog method to alloc/dealloc statement handles

between writing of stats, which may contribute to memory leak.

Change-Id: I6ded905c0b8197047f36268ae34f4a5f308d9e17

(cherry picked from commit 159c02117cd3b491c3c04936fbc395567680faa6)

    • -52
    • +88
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +29
    /conn/odbc/src/odbc/nsksrvr/SrvrMain.cpp
Added option to disable SQL plan collection for user queries

Added a new option that works on top of the new DCS property -SQLPLAN to disable

collection of query plan for user queries.

Cleaned up SessionWatchDog method to alloc/dealloc statement handles

between writing of stats, which may contribute to memory leak.

Change-Id: I6ded905c0b8197047f36268ae34f4a5f308d9e17

    • -52
    • +88
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +29
    /conn/odbc/src/odbc/nsksrvr/SrvrMain.cpp
Fix for bug #1410928. Mxosrvr coring during performance tests.

The core seems to be occurring because of a lack of synchronization

between the timer thread and main thread accessing a global statement

object. When the statement is dropped the associated pSrvrStmt

is also deleted and hence the global pointer is invalidated and causes

the core when accessed. The fix now also nulls the global statement

pointer so that the timer thread ignores the dropped

statement.

Fixes bug #1410928

Change-Id: I06b15b90325a7b405d4adcca871b58c9dba51729

    • -0
    • +5
    /conn/odbc/src/odbc/nsksrvrcore/srvrcommon.cpp
Fix for bug 1404108 where mxosrvrs do not exit on a sqstop

When sqstop or dcsstop is called some mxosrvrs do not exit when the

statistics collection feature is enabled. The statistics collection

occurs in a separate thread and that thread sometimes does not

terminate and also holds on to a mutex and does not get released during

exit. The fix is to retry the thread termination during exit time.

Fixes bug 1404108

Change-Id: Ia5764a4e3672da8dbade00f8e4b5bdacbb0d5ac5

    • -2
    • +27
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
Fix for bug #1409223 MXOSRVR coring during startup

This fixes bugs #1409223 and 1409224 where some of the MXOSRVRs core

during dcsstart since the new thread for repository continues to run and

invokes SQL calls even after the main thread terminated for some

reason.

Fixes bug 1409223

Fixes bug 1409224

Change-Id: I4cd2404c04e5b967ae27ec6580d8649b058aa0f6

    • -25
    • +28
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
Fixes memory leak in MXOSRVR Zookeeper processing

The code was calling the Zookeeper API zoo_get_children() but the

returned children list was not being freed using free_String_vector().

Change-Id: I73b7ed37c859f5cb01adeb1ba287d19a7369da12

Closes-Bug: 1361371

    • -0
    • +12
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
Fix for bug #1315537 - Dialogue ID does not match

The clients can sometimes get a Dialogue ID does not match error when

trying to connect. This was happening whenever both the

TerminateDialogue and BreakDialogue were being invoked, during the

previous client disconnect call, where the TerminateDialogue would make

the MXOSRVR state to AVAILABLE, invalidate the dialogue ID and

BreakDialogue would also attempt to do the same. The issue arises when

the latter call tries to change the state again but DCSMaster has

already allocated the MXOSRVR object reference to a new client.

Change-Id: I8e5a0bd5fd6a0827b4bd483d522f283c6d34348f

    • -0
    • +11
    /conn/odbc/src/odbc/Common/Global.h
    • -9
    • +20
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
Fix for bug - Zookeeper not in connecting state

This is a fix for bug #1252790. The issue was in the timer that was

acting on the timed out CONNECTING state to move back to AVAILABLE was

actually processing an old connection. The fix now looks at the dialogue

IDs to ascertain that it is processing the current connection request.

Added checks for strtok output per Arvind's comment.

Change-Id: If0fad0da305258120157b9c9e3998be4225bd589

    • -29
    • +53
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp