Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Add new PCode Expression Cache feature.

This new cache is maintained by the SQL Compiler. The purpose of this

cache is to avoid the fairly expensive logic involved in transforming

unoptimized PCode to optimized PCode and, where applicable, to also

avoid

the logic involved in transforming optimized PCode to a Native

Expression. This cache is accessed ONLY by the SQL Compiler code.

NOTES:

* This is second attempt to check in this code. First attempt had to

be abandoned as other developers made changes which prevented

automatic merging.

* This code has been pre-reviewed by Justin, Qifan, Selva, Mike,

Ravisha, Suresh, and Dave B. Many thanks to them for various

suggestions. Most of those suggestions have been incorporated into

this delivery. A few are left for future improvements.

* There is one instance of this new cache per CmpContext.

* There are 5 new CQDs used to control this cache. To be effective for

all instances of the cache, these need to be set in the system

defaults table. The CQD command given to sqlci will affect only the

instance of the cache for the current CmpContext.

The 5 CQDs are:

PCODE_EXPR_CACHE_ENABLED - set to 0 to disable the cache. Default is 1

PCODE_EXPR_CACHE_SIZE - max size in bytes. Default is 2,000,000.

PCODE_EXPR_CACHE_CMP_ONLY - Compare Only mode - useful to QA and

Development only.

PCODE_EXPR_CACHE_DEBUG - set to 1 to enable debug mode. Default is 0

PCODE_DEBUG_LOGDIR - pathname of existing directory where debug log

files will be placed -- one log file per cache

instance. Log files are designed to be easily

imported into an Excel Spreadsheet. No default.

* Also included are a small number of changes to the Native Expressions

feature to (a) Use the new PCODE_DEBUG_LOGDIR cqd to specify where to

put the Native Expressions debug log files, (b) measure cpu-time

rather than wall-clock time for measuring how long it took to produce

a Native Expression, and (c) add a CQD named PCODE_NE_ENABLED so we

can easily disable the Native Expressions feature [though there is

currently no known reason for doing so.]

Change-Id: I58f833f63099743ff6c1107acdff94fe8aef4b70

    • -22
    • +889
    /sql/exp/ExpPCodeOptimizations.cpp
    • -11
    • +10
    /sql/exp/ExpPCodeOptsNativeExpr.cpp
    • -0
    • +329
    /sql/exp/PCodeExprCache.h
Corrects problem in setting Java garbage collection switched in bin/dcs

The file conf/dcs-env.sh was missing some basic params for

analysing GC. Also the script bin/dcs didn't correctly set

the switches in SERVER_GC_OPTS variable.

Closes bug: 1429344

Change-Id: Iabdc0443900990c24c6a13b634eeccf00ea3b630

client-death stress.

Change-Id: Id50332474efe186c41162b7dd2e27fe3b093889a

    • -0
    • +20
    /sqf/src/seabed/test/go282
    • -0
    • +363
    /sqf/src/seabed/test/t282ms.cpp
Fixed bug with default & previous values for build tar files

If a build file was specified (as opposed to a package file) by the user

the values from previous runs was not being picked up as the

default values on subsequent runs. This fixes that for the

trafodion build and DCS build tar files.

Change-Id: I7bc941a0789a2619268d2dfb439daf7f593e0c94

Snapshot Scan changes

The changes in this delivery include:

-decoupling the snapshot scan from the bulk unload feature. Setup of the

temporary space and folders before running the query and cleanup afterwards

used to be done by the bulk unload operator because snapshot scan was specific

to bulk unload. In order the make snapshot scan indepenednt from bulk unload

and use it in any query the setup and cleanup tasks are now done by the query

itself at run time (the scan and root operators).

-caching of the snapshot information in NATable to optimize compilation time

Rework for chaching: when the user sets TRAF_TABLE_SNAPSHOT_SCAN to LATEST

we flush the metadata and then we set the caching back to on so that metadata

get cached again. If newer snapshots are created after setting the cqd they

won't be seen if they are already cached unless the user issue a command/cqd

to invalidate or flush the cache. One way for doing that can be to issue

"cqd TRAF_TABLE_SNAPSHOT_SCAN 'latest';" again

-code cleanup

below is a description of the CQds used with snapshot scan:

TRAF_TABLE_SNAPSHOT_SCAN

this CQD can be set to :

NONE--> (default)Snapshot scan is disabled and regular scan is used ,

SUFFIX --> Snapshot scan is enabled for the bulk unload (bulk unload

behavior is not changed)

LATEST --> Snapshot Scan is enabled independently from bulk unload and

the latest snapshot is used if it exists. If no snapshot exists

the regular scan is used. For this phase of the project the user

needs to create the snapshots using hbase shell or other tools.

And in the next phase of the project new comands to create,

delete and manage snapshots will be add.

TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX

This CQD is used with bulk unload and its value is used to build the

snapshot name as the table name followed by the suffix string

TRAF_TABLE_SNAPSHOT_SCAN_TABLE_SIZE_THRESHOLD

When the estimated table size is below the threshold (in MBs) defined by

this CQD the regular scan is used instead of snapshot scan. This CQD

does not apply to bulk unload which maintains the old behavior

TRAF_TABLE_SNAPSHOT_SCAN_TIMEOUT

The timeout beyond which we give up trying to create the snapshot scanner

TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION

Location for temporary links and files produced by snapshot scan

Change-Id: Ifede88bdf36049bac8452a7522b413fac2205251

    • -13
    • +37
    /sql/comexe/ComTdbHbaseAccess.cpp
    • -1
    • +107
    /sql/executor/HBaseClient_JNI.cpp
    • -44
    • +1
    /sql/executor/SequenceFileReader.cpp
  1. … 30 more files in changeset.
Using one thread pool for all HTables access

Earlier we had a ThreadPool for every HTable. Now we have one

thread pool for all HTables across all JDBC/ODBC connections

to improve thread resource usage.

Also cleaned up code to ensure the configuration is read at the

lower layers. Removed the need to pass host name and port

number because it will be picked up from hbase-site.xml

Fix for bug 1429882

Disabled pre-fetch in case of snapshot scan

Change-Id: Ic6b0e6a04d2e4fd26e4290302b6229aa2367ffef

Merge "Added new Trafodion REST server infrastructure"

NODE_LIST, removed hostname checks, scanner

Changed how NODE_LIST is created. A list of nodes is entered by the user

and that list is used to ssh to each node given which then the

"hostname" command is used to get hostname which is used to create

NODE_LIST that is used to generate sqconfig. Users can now enter in IP

addresses or node names. All hostname checks are now removed.

The scanner can now be called from trafodion installer.

Edit 1: Removed code used for testing

Edit 2: Corrected TRAF_VERSION (for when a debug build is used) and

added changes to how scanner is called.

Change-Id: I5671be3566c1fc3af0d7441254d9fa52150db2ba

partialy fix for query cancel

1. prepared DcsMaster for accepting request sent by T4 Driver to cancel

the query

2. Cleanup the code. Changing tabs to spaces.

3. Added processing unknown requests.

===============================================

Patch

1. To servermt added processing EndTransaction

2. To Master Listener added processing requests in LITTLE_ENDIAN

3. To servermt added fix for Logging

4. Fixed Exception message in RequestCancelQuery

5. In servermt fixed bugs reported in Phoenix Test.

===================================================

Patch

1. Fixed wrong text exception

2. Added checking if LOG is on

3. Removed listing LOG properties

4. Fixed precision for TIME and TIMESTAMP

5. Changed TYPE_EXPLAIN to return result set

6. Changed in SqlFetch process EOD

7. To SqlPrepare added more LOG info

8. Changed TConnection/TStatement closing

================================================

Patch

1. added query cancel procedure.

2. to pom.xml changed hadoop version to 2.5.0-cdh5.3.0 to make T2 driver

work

3. to servermt added SqlClose Api

4. fixed more problems in servermt

================================================

Patch

1. in pom.xml changed hadoop version to 2.6.0 and removed cloudera

repository.

2. fixed connecting timeout.

3. added connecting server state.

4. cleanup debug log messages.

5. restored in ClientData accidently deleted code.

================================================

Patch

1. fixed TAB-SPAC issues

===============================================

Patch

Fixed issues reported in code review

Change-Id: Ifd0b7835c87dbf84d5ea0dde05219a0651f19a74

  1. … 12 more files in changeset.
Merge "Fix HQC Bugs: LP1421374 LP1409863 LP1409830"

Enable authorization by default for regress, plus

Patch 1:

Added TEST138 to catman1 - skipped files

Fixed wording in the traf_authentication_setup script from reviewer comments.

Original delivery:

change 1 - Enable authorization during development regression tests

change 2 – Added support for create schema IF NOT EXISTS and drop schema IF EXISTS

change 3 - Changed traf_authentication_setup script to support a new installation option

change 1 - Enable authorization during development regression tests

Authorization will be enabled during regressions runs

Since regressions run mostly as DB__ROOT, there should be few visible differences.

Developers may see GRANT statements displayed as part of SHOWDDL requests.

This can be controlled by a new CQD:SHOWDDL_DISPLAY_PRIVILEGE_GRANTS

SHOWDDL_DISPLAY_PRIVILEGE_GRANTS

ON - display GRANTS if authorization is enabled

OFF - do not display GRANTS

SYSTEM

if running with SQLMX_REGRESS set, do not display grants

otherwise, display grants

regress/tools/init_sb_regr_sql -- execute initialize authorization

regress/tools/runregr_catman1.ksh -- turn on TEST138

regress/catman1 -- various test and expected files to set the new SHOWDDL CQD

"Initialize authorization, drop;" can be performed to disable authorization

files:

sql/regress/catman1/EXPECTED135

sql/regress/catman1/EXPECTED137

sql/regress/catman1/EXPECTED138

sql/regress/catman1/TEST133

sql/regress/catman1/TEST135

sql/regress/catman1/TEST136

sql/regress/catman1/TEST137

sql/regress/catman1/TEST138

sql/regress/catman1/TEST139

sql/regress/tools/init_sb_regr.sql

sql/regress/tools/runregr_catman1.ksh

sql/sqlcomp/CmpDescribe.cpp

sql/sqlcomp/CmpSeabaseDDLauth.cpp

sql/sqlcomp/DefaultConstants.h

sql/sqlcomp/nadefaults.cpp

change 2: Added support for create schema IF NOT EXISTS and drop schema IF EXISTS

Added support for new schema syntax. Change update stats for HIVE tables to use this syntax

files:

sql/parser/StmtDDLCreate.cpp

sql/parser/StmtDDLCreateSchema.h

sql/parser/StmtDDLDrop.cpp

sql/parser/StmtDDLDropSchema.h

sql/parser/sqlparser.y

sql/sqlcomp/CmpSeabaseDDL.h

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLschema.cpp

sql/ustat/hs_globals.cpp

change 3: Changed traf_authentication_setup script

This file was changed to support a new option "--setup" that only enables authentication

This will be used by the installation script when the customer chooses not to

initialize trafodion.

sqf/sql/scripts/traf_authentication_setup

traf_authentication_setup --help

This script enables or disables security features for Trafodion

Usage: traf_authentication_setup [options]

Options:

--file <loc> Optional location of the OpenLDAP configuration file

--help Prints this message

--off Disables authentication and authorization

--on Enables authentication and authorization

--setup Enables authentication

--status Returns status of authentication enablement

Change-Id: Ia9a66364a6d74955a0833088874e0aaca044eae3

  1. … 10 more files in changeset.
Fix regression from 1225 - abend from hive scan

This change fixes a problem merge in Mar 4 which will

cause an abend and corefile when scanning hive tables.

Change-Id: I8e115d85e1dbedb11e3f286fe4735301feb3c21b

Changes for Bug #1403380

This is a re-commit.

Including cleanups for unused java references in native,

and made the native driver global objects(gJNICache, gSrvrGlobal)

be global for the process, gDescItems will be thread local.

Change-Id: I6ec2334e06c1bd346757fcf1e2816add99d9503f

    • -69
    • +4
    /conn/jdbc_type2/native/SQLMXCommonFunctions.cpp
    • -6
    • +0
    /conn/jdbc_type2/native/SQLMXDriver.cpp
    • -1
    • +1
    /conn/jdbc_type2/native/SrvrCommon.cpp
Merge "LP 1390261 odbc does not return any value from hbase table"

Merge "Bug 1425650 mxosrvr cores after initial connection."

Merge "Cleanup hive reader thread when canceling scan"

Merge "Fix for bug #1415936"

Merge "Bug 1425650 Partial fix"

Bug 1415661 and 1421373 fixed

Bug 1415661: Check to make sure trafodion user id to install trafodion

with (normally 'Trafodion') is not the same as the sudo user.

Bug 1421373: Install gzip if not installed

Change-Id: Ib9e0c080a4c3aaa95ef230045437aafbe9d1be10

Merge "fix #1395212 With multiple statements,statement seems to be closed."

Cleanup hive reader thread when canceling scan

Cancel processing, whether triggered by [FIRST N], error

handling, SQL cancel, or any other reason, can cause the

main executor thread to abruptly stop interacting with

the reader threads. This change fixes a hang caused by

a reader thread waiting for the main thread to give it

an empty buffer, after the main thread has finished the

canceled query.

Change-Id: Ib41a8a0036b7aab8dedf7d10ee55eb2007f7c265

Closes-Bug: #1425661

Merge "Fixed catman1 TEST137 failure"

OSIM (Optimizer Simulator) redesign 1.

Simulate query plan generation of production cluster on dev workstation,

by collecting information from production cluster, and restore it on dev worksation.

--runnig on production clusters, collect table DDLs, statistics, CQD, to osim-directory,

--the directory path can either full(absolute) or relative.

osim capture location '<osim-directory>'[, force];

--runing queries on cluster

osim capture stop;

--restore DDLs, CQDs, statistics and cluster information.

osim load from '<osim-directory>';

--setup runtime information, like cpu number, node number.

osim simulate start|continue '<osim-directory>';

Change-Id: I30882e87a6ea0f08c9aa64685705eebebcbb3bf0

  1. … 24 more files in changeset.
Merge "fix NDCS build - version build."

Merge "Reduced Java objects allocation in the client side"

Convert DSC documentation from docbooks to asciidoc

Docbooks was very difficult for contributors to easily add

updates. More time was spent fixing xml tags than writing

the needed documentation. Asciidoc while very feature rich in

its tag set is much much easier to learn.

Change-Id: Ib23ba6c11c8b0f13023119e181ea16efddd07ac6

    • -0
    • +118
    /src/main/asciidoc/_chapters/architecture.adoc
    • -0
    • +416
    /src/main/asciidoc/_chapters/configuration.adoc
    • -0
    • +33
    /src/main/asciidoc/_chapters/external_apis.adoc
    • -0
    • +43
    /src/main/asciidoc/_chapters/faq.adoc
  1. … 20 more files in changeset.
Reordered the way HDFS and HBase are restarted for Hortonworks

Previously, components were restarted in this order:

restart HBase

restart HDFS

This might have been causing intermittent issues on the Jenkins

gate machines resulting in hanging sqstart.

This change reorders it to this:

stop HBase

stop HDFS

start HDFS

start HBase

Change-Id: I3fe21b69d33f9b74cd0859e4400fa02b93f7a371

    • -46
    • +54
    /installer/traf_hortonworks_mods98
Fixed catman1 TEST137 failure

And some minor changes from a previous review.

Change-Id: I01319b40e3ddf1bdba88c70d5a4b62057698491e

Merge "Fix for bug 1329361 and minor improvements in hive scan"

Merge "LP 1390261 odbc does not return any value from hbase table"