Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
LP and other fixes.

-- LP 1414074. Added 'cleanup obsolete volatile tables' command

-- Added support for 'get all volatile tables' command

-- LP 1411864. Alter Sequence now correctly returns an error if value

specified exceeds the max value

-- LP 1413743. Error indicating sequence number max is reached is now

being returned instead of a numeric overflow error ff max largeint value

is reached while generating sequence numbers.

-- LP 1418685. Partition information is now being set up correctly for

delimited salted tables.

-- changed copyright message

Change-Id: Ic9e532204890a68ea0616b99a3170a0cc735ad53

Signed-off-by: Anoop Sharma <anoop.sharma@hp.com>

    • -386
    • +51
    /sql/executor/ExExeUtilVolTab.cpp
Merge "Changes in T2 driver to support new NA server."

Fix for hang encountered during hive tests.

Added proper copyright.

Added comment to make it clear to readers of the code.

Hive tests have been hanging for quite a while on the official slave

machines.

The cause for this particular hang was that the main thread was

destroying the cursor (ExLobCursor) object . The worker thread was kind

of slow and it continues to access the cursor after the main thread had

destroyed it. pthread calls exhibit undefined behavior when

uninitialized mutex or condition variables are accessed. Added code to

prevent this timing issue in the worker threads.

Added trace utlitlty to diagnose hangs and execution issues.

The trace messages look like this and get logged into trace file on the

local directory named trace_threads.<pid>. It is controlled by an

environment variable TRACE_HDFS_THREAD_ACTIONS.The envvar is checked

only ones and the file handle is checked each time the trace needs to be

done. If file handle is NULL, no message is logged.

Change-Id: I95519b61d71339c719e37500bccae111ce070a15

Fix for more leaks in NATable

Added code to delete more elements in NATable destructor. This

fixes leaks when NATable is destroyed.

Changed the NATable Cache Management to use allocated size rather

than the total size.

This also enables seabase/TEST020 to pass in debug mode.

Change-Id: Ifebc20e602c0149f73f9cefcd11b37c69d9eec74

Merge "Fix memory leak when ComDiagsArea is not deleted" into stable/1.0

Merge "HBaseClient now logs correctly. Updated copyright"

Merge "Added new property to disable sqlplan for user queries"

Disable seabase/TEST024

Change-Id: Ife00eedd8b09aae17ada84959930847bdf5684a0

    • -2
    • +2
    /sql/regress/tools/runregr_seabase.ksh
Merge "Fix memory leak when ComDiagsArea is not deleted"

Bulk unload optimization using snapshot scan

resubmitting after facing git issues

The changes consist of:

*implementing the snapshot scan optimization in the Trafodion scan operator

*changes to the bulk unload changes to use the new snapshot scan.

*Changes to scripts and permissions (using ACLS)

*Rework based on review

Details:

*Snapshot Scan:

----------------------

**Added support for snapshot scan to Trafodion scan

**The scan expects the hbase snapshots themselves to be created before running

the query. When used with bulk unload the snapshots can created by bulk unload

**The snapshot scan implementation can be used without the bulk-unload. To use

the snapshot scan outside bulk-unload we need to use the below cqds

cqd TRAF_TABLE_SNAPSHOT_SCAN 'on'; --

-- the snapshot name will the table name concatenated with the suffix-string

cqd TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX 'suffix-string';

-- temp dir needed for the hbase snapshotsca

cqd TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION '/bulkload/temp_scan_dir/'; n

**snapshot scan can be used with table scan, index scans etc…

*Bulk unload utility :

-------------------------------

**The bulk unload optimization is due the newly added support for snapshot scan.

By default bulk unload uses the regular scan. But when snapshot scan is

specified it will use snapshot scan instead of regular scan

**To use snapshot scan with Bulk unload we need to specify the new options in

the bulk unload syntax : NEW|EXISTING SNAPHOT HAVING SUFFIX QUOTED_STRING

***using NEW in the above syntax means the bulk unload tool will create new

snapshots while using EXISTING means bulk unload expect the snapshot to

exist already.

***The snapshot names are based on the table names in the select statement. The

snapshot name needs to start with table name and have a suffix QUOTED-STRING

***For example for “unload with NEW SNAPSHOT HAVING SUFFIX ‘SNAP111’ into ‘tmp’

select from cat.sch.table1; “ the unload utiliy will create a snapshot

CAT.SCH.TABLE1_SNAP111; and for “unload with EXISTING SNAPSHOT HAVING SUFFIX

‘SNAP111’ into ‘tmp’ select from cat.sch.table1; “ the unload utility will

expect a snapshot CAT.SCH.TABLE1_SNAP111; to be existing already. Otherwise

an error is produced.

***If this newly added options is not used in the syntax bulk unload will use

the regular scan instead of snapshot scan

**The bulk unload queries the explain plan virtual table to get the list of

Trafodion tables that will be scanned and based on the case it either creates

the snapshots for those tables or verifies if they already exists or not

*Configuration changes

--------------------------------

**Enable ACLs in hdfs

**

*Testing

--------

**All developper regression tests were run and all passed

**bulk unload and snapshot scan were tested on the cluster

*Examples:

**Example of using snapshot scan without bulk unload:

(we need to create the snapshot first )

>>cqd TRAF_TABLE_SNAPSHOT_SCAN 'on';

--- SQL operation complete.

>>cqd TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX 'SNAP777';

--- SQL operation complete.

>>cqd TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION '/bulkload/temp_scan_dir/';

--- SQL operation complete.

>>select [first 5] c1,c2 from tt10;

C1 C2

--------------------- --------------------

.00 0

.01 1

.02 2

.03 3

.04 4

--- 5 row(s) selected.

**Example of using snapshot scan with unload:

UNLOAD

WITH PURGEDATA FROM TARGET

NEW SNAPSHOT HAVING SUFFIX 'SNAP778'

INTO '/bulkload/unload_TT14_3' select * from seabase.TT20 ;

Change-Id: Idb1d1807850787c6717ab0aa604dfc9a37f43dce

    • -187
    • +528
    /sql/executor/ExExeUtilLoad.cpp
    • -11
    • +39
    /sql/executor/HBaseClient_JNI.cpp
    • -102
    • +266
    /sql/executor/HTableClient.java
    • -10
    • +282
    /sql/executor/SequenceFileReader.cpp
  1. … 21 more files in changeset.
Added option to disable SQL plan collection for user queries

Added a new option that works on top of the new DCS property -SQLPLAN to disable

collection of query plan for user queries.

Cleaned up SessionWatchDog method to alloc/dealloc statement handles

between writing of stats, which may contribute to memory leak.

Change-Id: I6ded905c0b8197047f36268ae34f4a5f308d9e17

(cherry picked from commit 159c02117cd3b491c3c04936fbc395567680faa6)

    • -52
    • +88
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +29
    /conn/odbc/src/odbc/nsksrvr/SrvrMain.cpp
Added option to disable SQL plan collection for user queries

Added a new option that works on top of the new DCS property -SQLPLAN to disable

collection of query plan for user queries.

Cleaned up SessionWatchDog method to alloc/dealloc statement handles

between writing of stats, which may contribute to memory leak.

Change-Id: I6ded905c0b8197047f36268ae34f4a5f308d9e17

    • -52
    • +88
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +29
    /conn/odbc/src/odbc/nsksrvr/SrvrMain.cpp
The following Launchpad bugs are fixed in this change:

Bug 1370749: Now using MAX_USERNAME_LEN instead of hardcoded value

Bug 1413760: CREATE TABLE LIKE was failing in some circumstances because

SHOWDDL was including the BY clause. Ownership rules changes in

CREATE TABLE changed when ANSI schemas was implemented, so the BY clause

is no longer needed.

Bug 1392107: Privileges granted on a view are no longer lost if the

view is replaced via CREATE OR REPLACE VIEW.

Bug 1370740: A potential memory corruption problem is now avoided

by reworking the authorization name lookup functions.

Bug 1413767: Previously DROP SCHEMA CASCADE would fail to drop a

table with an IDENTITY column.

Bug 1413758: Previously DROP TABLE CASCADE did not drop nested views.

Bug 1412891: Previously DROP TABLE CASCADE failed if a dependent object

contained a delimited name.

Changes are present for 1392086, but the work is not yet completed.

This problem is related to roles and security keys.

Code changes are also present for giving ownership of an object to

another authorization ID, but these changes are not complete. A

description of

the changes is included.

The GIVE command transfers ownership of a SQL item from one

authorization ID to another. Implemented in this delivery is

GIVE SCHEMA and GIVE ALL.

GIVE ALL transfers all SQL items owned by an authorization ID to another

authorization ID. Current or new owner can be a user or a role. The

GIVE ALL command requires the ALTER privilege.

GIVE ALL FROM authID TO authID

GIVE SCHEMA behavior depends on the type of schema and whether RESTRICT

or CASCADE is specified. For private schemas, all the objects in the

schema are given, as well as the schema itself. For shared schemas,

only the

schema is given, unless the CASCADE option is specified. In that case,

ownership

of all the objects in the shared schema is given to the new owner. Use

of

the CASCADE option requires the ALTER_SCHEMA privilege. Otherwise, GIVE

SCHEMA only requires the user to be the owner of the schema.

GIVE SCHEMA schema-name TO authID [RESTRICT|CASCADE]

NOTE: RESTRICT and CASCADE are not applicable to private schemas and are

ignored.

GIVE OBJECT is added to the syntax but is not implemented and may not

be implemented.

A more detailed blueprint will be provided prior to the final delivery

of GIVE.

Change-Id: I7449da599dc80de1c0659164e684841cda4647c8

    • -0
    • +272
    /sql/parser/StmtDDLGive.h
  1. … 20 more files in changeset.
Merge "Fix for NATable Heap leak"

Fix memory leak when ComDiagsArea is not deleted

In ExControlTcb, the allocated ComDiagsArea for the embedded compiler

to use did not get freed, resulting 336-byte block leak in the

ContextCli heap each time a set query default statement was executed.

This fix gets the memory for the Statement heap and frees it after use.

Change-Id: I0ef68e6d3c9372a8963ff8da375479af8d64c198

Fix memory leak when ComDiagsArea is not deleted

In ExControlTcb, the allocated ComDiagsArea for the embedded compiler

to use did not get freed, resulting 336-byte block leak in the

ContextCli heap each time a set query default statement was executed.

This fix gets the memory for the Statement heap and frees it after use.

Change-Id: I0ef68e6d3c9372a8963ff8da375479af8d64c198

HBaseClient now logs correctly. Updated copyright

HbaseClient failed to setup log4j properly due to its

inability to locate the correct properties file.

Updated copyright

Closes-Bug: #1391918

Change-Id: Ifed74ca8879431254b8a8e6586e717eb588a924f

Added new property to disable sqlplan for user queries

Changed the property name isolated to statistics

Change-Id: I378270984ef22afc699063341cb30901e19611ab

Added new property to disable sqlplan for user queries

Changed the property name isolated to statistics

Change-Id: I378270984ef22afc699063341cb30901e19611ab

(cherry picked from commit f14119cdcb82d59cb8d83a2af7f7485a13fc5f2b)

Fix memory leaks when EXPLAIN is re-executed

In cases where an EXPLAIN statement is prepared once and

executed many times, each reexecution will allocate more

memory from the Statement heap. Cleanup will only happen

when the Statement heap is deallocated. This change fixes

the problem by deallocating between executions.

Change-Id: I515bde53348384302f040bb9391a3a3530c7a1d2

Fix memory leaks when EXPLAIN is re-executed

In cases where an EXPLAIN statement is prepared once and

executed many times, each reexecution will allocate more

memory from the Statement heap. Cleanup will only happen

when the Statement heap is deallocated. This change fixes

the problem by deallocating between executions.

Change-Id: I515bde53348384302f040bb9391a3a3530c7a1d2

Merge the private shared single script repo into master.

This is a checkin of the single script project which basically

wraps all the various scripts into one single script that the user

runs. The script now has the logic to determine which Hadoop distro

is installed as well as to determine if it is a full install or only

an upgrade and many other changes.

This was developed and tested in a private shared repo so

a review of the code has already been done. QA has approved this

to be checked in.

edit1: fixed shift problem in parsing parameters

edit2: Fixed problems with user config file

Added more error checking for config file

edit3: Fixed config directory creation

edit4: Fixed copyright and OpenStack VM package installation

edit5: Changed Java version detection

No longer do yum removes if package is there already

edit6: Refixed OpenStack VM package installation that got

overlaid in a merge

Added chown command to handle case where SQ_ROOT was not

a sub-directory of the Trafodion userid's home dir

Change-Id: I37a331c89c311feee430541b69f5f7dfe2177e60

    • -1
    • +1
    /installer/tools/hortonworks_uninstall
    • -2
    • +2
    /installer/tools/traf_cloudera_uninstall
  1. … 13 more files in changeset.
port skew buster to Trafodion

1. add simplified TEST062

2. reuse cached partitioning expression when only doVarCharCast

is the same as the when the expression is created in

TableHashPartitioningFunction::createPartitioningExpressionImp().

3. rework

4. fix a time monitor bug reporting incorrect processor time (CPU time

computed from clock() calls)

5. comment out the assert on the size of NATable cache not decreasing.

This is to fix regression failure with seabase/TEST020. Selva will check

in a formal fix that allocates space for NATable objects from a single

heap, and then reenables this assert check.

Change-Id: I9eeee4f36ba8678e90e0ac68a85bfc733599d932

    • -1
    • +131
    /sql/executor/ex_split_bottom.cpp
    • -1
    • +110
    /sql/generator/GenRelEnforcer.cpp
    • -2
    • +270
    /sql/optimizer/OptPhysRelExpr.cpp
  1. … 12 more files in changeset.
Increase RMS timeout in executor/TEST106

This test has failed on some daily builds. Analysis of logs

showed that a GET STATISTICS statement was timing out. This

change increases the time value to try to make the test

more deterministic.

Change-Id: I3bfb5f2101e998fc2c7766f1ecaf45f87edf510a

Merge "Update release version number on master branch"

Add dependencies and fix build.

Change-Id: I9b7c0fd6b79c9bb92776a17180dd6cd8cdc0ad3f

Fix for NATable Heap leak

Earlier, each NATable entry had its own heap. Now, there is a NATable Heap

created for each compiler context. All the cached NATable object is

created from this heap.

This is done to reduce the memory usage in the compiler context and to

take care of leak of NATable heap.

Change-Id: I3ec362c0b8e5b1466956c3773c2b5dc7dccf0d52

Merge "SQL support for phandle verifier, etc."

Update release version number on master branch

Next minor feature release will be 1.1.0

Change-Id: I325e5a16fdcbe88a96d1a31bb2e51852f8018a5d

Update version to 1.1.0 on master branch

Next minor feature release.

TestBasic updated to match latest DB version number (1.1.0) in core.

Change-Id: I101060deb2fc1dfd68cb6681d12ef5f40eec52af