Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fixed bug with default & previous values for build tar files

If a build file was specified (as opposed to a package file) by the user

the values from previous runs was not being picked up as the

default values on subsequent runs. This fixes that for the

trafodion build and DCS build tar files.

Change-Id: I7bc941a0789a2619268d2dfb439daf7f593e0c94

Snapshot Scan changes

The changes in this delivery include:

-decoupling the snapshot scan from the bulk unload feature. Setup of the

temporary space and folders before running the query and cleanup afterwards

used to be done by the bulk unload operator because snapshot scan was specific

to bulk unload. In order the make snapshot scan indepenednt from bulk unload

and use it in any query the setup and cleanup tasks are now done by the query

itself at run time (the scan and root operators).

-caching of the snapshot information in NATable to optimize compilation time

Rework for chaching: when the user sets TRAF_TABLE_SNAPSHOT_SCAN to LATEST

we flush the metadata and then we set the caching back to on so that metadata

get cached again. If newer snapshots are created after setting the cqd they

won't be seen if they are already cached unless the user issue a command/cqd

to invalidate or flush the cache. One way for doing that can be to issue

"cqd TRAF_TABLE_SNAPSHOT_SCAN 'latest';" again

-code cleanup

below is a description of the CQds used with snapshot scan:

TRAF_TABLE_SNAPSHOT_SCAN

this CQD can be set to :

NONE--> (default)Snapshot scan is disabled and regular scan is used ,

SUFFIX --> Snapshot scan is enabled for the bulk unload (bulk unload

behavior is not changed)

LATEST --> Snapshot Scan is enabled independently from bulk unload and

the latest snapshot is used if it exists. If no snapshot exists

the regular scan is used. For this phase of the project the user

needs to create the snapshots using hbase shell or other tools.

And in the next phase of the project new comands to create,

delete and manage snapshots will be add.

TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX

This CQD is used with bulk unload and its value is used to build the

snapshot name as the table name followed by the suffix string

TRAF_TABLE_SNAPSHOT_SCAN_TABLE_SIZE_THRESHOLD

When the estimated table size is below the threshold (in MBs) defined by

this CQD the regular scan is used instead of snapshot scan. This CQD

does not apply to bulk unload which maintains the old behavior

TRAF_TABLE_SNAPSHOT_SCAN_TIMEOUT

The timeout beyond which we give up trying to create the snapshot scanner

TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION

Location for temporary links and files produced by snapshot scan

Change-Id: Ifede88bdf36049bac8452a7522b413fac2205251

    • -13
    • +37
    /sql/comexe/ComTdbHbaseAccess.cpp
    • -1
    • +107
    /sql/executor/HBaseClient_JNI.cpp
    • -44
    • +1
    /sql/executor/SequenceFileReader.cpp
  1. … 30 more files in changeset.
Using one thread pool for all HTables access

Earlier we had a ThreadPool for every HTable. Now we have one

thread pool for all HTables across all JDBC/ODBC connections

to improve thread resource usage.

Also cleaned up code to ensure the configuration is read at the

lower layers. Removed the need to pass host name and port

number because it will be picked up from hbase-site.xml

Fix for bug 1429882

Disabled pre-fetch in case of snapshot scan

Change-Id: Ic6b0e6a04d2e4fd26e4290302b6229aa2367ffef

Merge "Added new Trafodion REST server infrastructure"

NODE_LIST, removed hostname checks, scanner

Changed how NODE_LIST is created. A list of nodes is entered by the user

and that list is used to ssh to each node given which then the

"hostname" command is used to get hostname which is used to create

NODE_LIST that is used to generate sqconfig. Users can now enter in IP

addresses or node names. All hostname checks are now removed.

The scanner can now be called from trafodion installer.

Edit 1: Removed code used for testing

Edit 2: Corrected TRAF_VERSION (for when a debug build is used) and

added changes to how scanner is called.

Change-Id: I5671be3566c1fc3af0d7441254d9fa52150db2ba

partialy fix for query cancel

1. prepared DcsMaster for accepting request sent by T4 Driver to cancel

the query

2. Cleanup the code. Changing tabs to spaces.

3. Added processing unknown requests.

===============================================

Patch

1. To servermt added processing EndTransaction

2. To Master Listener added processing requests in LITTLE_ENDIAN

3. To servermt added fix for Logging

4. Fixed Exception message in RequestCancelQuery

5. In servermt fixed bugs reported in Phoenix Test.

===================================================

Patch

1. Fixed wrong text exception

2. Added checking if LOG is on

3. Removed listing LOG properties

4. Fixed precision for TIME and TIMESTAMP

5. Changed TYPE_EXPLAIN to return result set

6. Changed in SqlFetch process EOD

7. To SqlPrepare added more LOG info

8. Changed TConnection/TStatement closing

================================================

Patch

1. added query cancel procedure.

2. to pom.xml changed hadoop version to 2.5.0-cdh5.3.0 to make T2 driver

work

3. to servermt added SqlClose Api

4. fixed more problems in servermt

================================================

Patch

1. in pom.xml changed hadoop version to 2.6.0 and removed cloudera

repository.

2. fixed connecting timeout.

3. added connecting server state.

4. cleanup debug log messages.

5. restored in ClientData accidently deleted code.

================================================

Patch

1. fixed TAB-SPAC issues

===============================================

Patch

Fixed issues reported in code review

Change-Id: Ifd0b7835c87dbf84d5ea0dde05219a0651f19a74

  1. … 12 more files in changeset.
Merge "Fix HQC Bugs: LP1421374 LP1409863 LP1409830"

Enable authorization by default for regress, plus

Patch 1:

Added TEST138 to catman1 - skipped files

Fixed wording in the traf_authentication_setup script from reviewer comments.

Original delivery:

change 1 - Enable authorization during development regression tests

change 2 – Added support for create schema IF NOT EXISTS and drop schema IF EXISTS

change 3 - Changed traf_authentication_setup script to support a new installation option

change 1 - Enable authorization during development regression tests

Authorization will be enabled during regressions runs

Since regressions run mostly as DB__ROOT, there should be few visible differences.

Developers may see GRANT statements displayed as part of SHOWDDL requests.

This can be controlled by a new CQD:SHOWDDL_DISPLAY_PRIVILEGE_GRANTS

SHOWDDL_DISPLAY_PRIVILEGE_GRANTS

ON - display GRANTS if authorization is enabled

OFF - do not display GRANTS

SYSTEM

if running with SQLMX_REGRESS set, do not display grants

otherwise, display grants

regress/tools/init_sb_regr_sql -- execute initialize authorization

regress/tools/runregr_catman1.ksh -- turn on TEST138

regress/catman1 -- various test and expected files to set the new SHOWDDL CQD

"Initialize authorization, drop;" can be performed to disable authorization

files:

sql/regress/catman1/EXPECTED135

sql/regress/catman1/EXPECTED137

sql/regress/catman1/EXPECTED138

sql/regress/catman1/TEST133

sql/regress/catman1/TEST135

sql/regress/catman1/TEST136

sql/regress/catman1/TEST137

sql/regress/catman1/TEST138

sql/regress/catman1/TEST139

sql/regress/tools/init_sb_regr.sql

sql/regress/tools/runregr_catman1.ksh

sql/sqlcomp/CmpDescribe.cpp

sql/sqlcomp/CmpSeabaseDDLauth.cpp

sql/sqlcomp/DefaultConstants.h

sql/sqlcomp/nadefaults.cpp

change 2: Added support for create schema IF NOT EXISTS and drop schema IF EXISTS

Added support for new schema syntax. Change update stats for HIVE tables to use this syntax

files:

sql/parser/StmtDDLCreate.cpp

sql/parser/StmtDDLCreateSchema.h

sql/parser/StmtDDLDrop.cpp

sql/parser/StmtDDLDropSchema.h

sql/parser/sqlparser.y

sql/sqlcomp/CmpSeabaseDDL.h

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLschema.cpp

sql/ustat/hs_globals.cpp

change 3: Changed traf_authentication_setup script

This file was changed to support a new option "--setup" that only enables authentication

This will be used by the installation script when the customer chooses not to

initialize trafodion.

sqf/sql/scripts/traf_authentication_setup

traf_authentication_setup --help

This script enables or disables security features for Trafodion

Usage: traf_authentication_setup [options]

Options:

--file <loc> Optional location of the OpenLDAP configuration file

--help Prints this message

--off Disables authentication and authorization

--on Enables authentication and authorization

--setup Enables authentication

--status Returns status of authentication enablement

Change-Id: Ia9a66364a6d74955a0833088874e0aaca044eae3

  1. … 10 more files in changeset.
Fix regression from 1225 - abend from hive scan

This change fixes a problem merge in Mar 4 which will

cause an abend and corefile when scanning hive tables.

Change-Id: I8e115d85e1dbedb11e3f286fe4735301feb3c21b

Changes for Bug #1403380

This is a re-commit.

Including cleanups for unused java references in native,

and made the native driver global objects(gJNICache, gSrvrGlobal)

be global for the process, gDescItems will be thread local.

Change-Id: I6ec2334e06c1bd346757fcf1e2816add99d9503f

    • -69
    • +4
    /conn/jdbc_type2/native/SQLMXCommonFunctions.cpp
    • -6
    • +0
    /conn/jdbc_type2/native/SQLMXDriver.cpp
    • -1
    • +1
    /conn/jdbc_type2/native/SrvrCommon.cpp
Merge "LP 1390261 odbc does not return any value from hbase table"

Merge "Bug 1425650 mxosrvr cores after initial connection."

Merge "Cleanup hive reader thread when canceling scan"

Merge "Fix for bug #1415936"

Merge "Bug 1425650 Partial fix"

Bug 1415661 and 1421373 fixed

Bug 1415661: Check to make sure trafodion user id to install trafodion

with (normally 'Trafodion') is not the same as the sudo user.

Bug 1421373: Install gzip if not installed

Change-Id: Ib9e0c080a4c3aaa95ef230045437aafbe9d1be10

Merge "fix #1395212 With multiple statements,statement seems to be closed."

Cleanup hive reader thread when canceling scan

Cancel processing, whether triggered by [FIRST N], error

handling, SQL cancel, or any other reason, can cause the

main executor thread to abruptly stop interacting with

the reader threads. This change fixes a hang caused by

a reader thread waiting for the main thread to give it

an empty buffer, after the main thread has finished the

canceled query.

Change-Id: Ib41a8a0036b7aab8dedf7d10ee55eb2007f7c265

Closes-Bug: #1425661

Merge "Fixed catman1 TEST137 failure"

OSIM (Optimizer Simulator) redesign 1.

Simulate query plan generation of production cluster on dev workstation,

by collecting information from production cluster, and restore it on dev worksation.

--runnig on production clusters, collect table DDLs, statistics, CQD, to osim-directory,

--the directory path can either full(absolute) or relative.

osim capture location '<osim-directory>'[, force];

--runing queries on cluster

osim capture stop;

--restore DDLs, CQDs, statistics and cluster information.

osim load from '<osim-directory>';

--setup runtime information, like cpu number, node number.

osim simulate start|continue '<osim-directory>';

Change-Id: I30882e87a6ea0f08c9aa64685705eebebcbb3bf0

  1. … 24 more files in changeset.
Merge "fix NDCS build - version build."

Merge "Reduced Java objects allocation in the client side"

Convert DSC documentation from docbooks to asciidoc

Docbooks was very difficult for contributors to easily add

updates. More time was spent fixing xml tags than writing

the needed documentation. Asciidoc while very feature rich in

its tag set is much much easier to learn.

Change-Id: Ib23ba6c11c8b0f13023119e181ea16efddd07ac6

    • -0
    • +118
    /src/main/asciidoc/_chapters/architecture.adoc
    • -0
    • +416
    /src/main/asciidoc/_chapters/configuration.adoc
    • -0
    • +33
    /src/main/asciidoc/_chapters/external_apis.adoc
    • -0
    • +43
    /src/main/asciidoc/_chapters/faq.adoc
  1. … 20 more files in changeset.
Reordered the way HDFS and HBase are restarted for Hortonworks

Previously, components were restarted in this order:

restart HBase

restart HDFS

This might have been causing intermittent issues on the Jenkins

gate machines resulting in hanging sqstart.

This change reorders it to this:

stop HBase

stop HDFS

start HDFS

start HBase

Change-Id: I3fe21b69d33f9b74cd0859e4400fa02b93f7a371

    • -46
    • +54
    /installer/traf_hortonworks_mods98
Fixed catman1 TEST137 failure

And some minor changes from a previous review.

Change-Id: I01319b40e3ddf1bdba88c70d5a4b62057698491e

Merge "Fix for bug 1329361 and minor improvements in hive scan"

Merge "LP 1390261 odbc does not return any value from hbase table"

Bug 1425650 mxosrvr cores after initial connection.

1. Recursive mutex for maintaining SRVR_SESSION_HDL list

2. Disabled invocation of ESPs for repository queries

3. Updated WHERE clause for METRIC_QUERY_AGGR_TABLE.

Cherry picked Change-Id: If55f9de8f1541ba0a06b13f92750b64ff4676e0b

Took care of review comments by Tharak.

Change-Id: Ibf69b5659e47cf2607297195351fda4772068cfb

    • -7
    • +29
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -4
    • +52
    /conn/odbc/src/odbc/nsksrvrcore/srvrcommon.cpp
Added logic to wait for HDFS to leave safemode before continuing.

When HDFS Namenode restarts, it is temporarily in safemode which

makes HDFS read-only for a time. This caused the hdfs mkdir/setfacl

commands to sometimes fail.

Added some additional error checks for HDFS commands and corrected some

message texts.

Change-Id: I026ef87f715edbdabfd0115d93ac0638dfdf0147

Added new Trafodion REST server infrastructure

This commit provides the infrastructure for a Trafodion REST

server. The code was taken from stand alone DCS rest server and

slightly modified to retain just the REST server components and

its supporting scripting and JDBC T2/T4 capabilities. Additional

work is needed by installation group but it currently can be

started in standalone mode while requirements are fully fleshed out.

Change-Id: I1e7a07a426a2b2c21bbb8c9fdb62222bc542329d

    • -0
    • +113
    /rest/bin/rest-config.sh
    • -0
    • +202
    /rest/bin/rest-daemon.sh
    • -0
    • +67
    /rest/bin/scripts/servers.py
    • -0
    • +59
    /rest/bin/scripts/sys_shell.py
    • -0
    • +32
    /rest/bin/start-rest.sh
    • -0
    • +31
    /rest/bin/stop-rest.sh
    • -0
    • +62
    /rest/conf/log4j.properties
    • -0
    • +1
    /rest/conf/realm.properties
    • -0
    • +91
    /rest/conf/rest-env.sh
    • -0
    • +21
    /rest/conf/rest-site.xml
  1. … 55 more files in changeset.