Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Changed "chown" to "-chown" for the hdfs dfs commands.

Change-Id: Iab3dd3b0914eebcf2d0d08821cf968966bab0062

Reduced Java objects allocation in the client side

jmap on the client side revealed that it is possible to avoid some

java object creation on the client side. Reworked to avoid Callable

object creation with every pre-fetch operation. Removed

QualifiedColumn creation to parse family name and qualifier name.

Uses CachedThreadPool instead of FixedThreadPool to reduce thread

creation and share thread pool across HTableClients.

Change-Id: I05bf8f4f3f0aaa24c749a535b7586f7da799dd88

Query cancel privilege checking

This change integrates the ControlRunningQuery::isUserAuthorized

method (change 1183) to work with ControlRunningQuery::bindNode.

It also adds new test cases to regress/catman1/TEST131.

Change-Id: I0c6c2357b7ff928863d8511702fdf6e9e2411411

Closes-Bug: #1414231

    • -1
    • +160
    /sql/regress/catman1/EXPECTED131
    • -82
    • +162
    /sql/regress/executor/EXPECTED106
Reworked the fix for bug 1418757

Interrupting the thread to cancel the pending IO was causing the

client to retry the connection and pause when the HTable was used

next time.

This fix allows the pending IO to complete normally. If it can't

complete normally within 30 seconds, the thread is interrupted

and the table is removed from cache.

Change-Id: I404ec7a91952301ef2bcb1a8d6448f8534a421d8

Merge "Misleading error when Hivestats can't be created"

Merge "Adding exception checks/reconnect for tm scanner"

Merge "SHOWDDL, QUERY Cancel, rework"

Merge "Removed qpid dependency from connectivity"

Adding exception checks/reconnect for tm scanner

This change along with a change from SQL in HTableClient.java

will fix our thread cancel issue which is causing a

coprocessor connection retry. Also adding exception logging

and a POM change to remove a warning.

Change-Id: Iabbc84193ca56546add8797f1a864df65e527799

Misleading error when Hivestats can't be created

When Update Statistics is performed on a Hive table, the

trafodion.hivestats schema is created if it doesn't already

exist. If the user issuing the ustat statement does not

have the create_schema component privilege, this will fail, but

the code as it existed ignored the error, assuming it was

a "schema already exists" error, which is expected in most

cases. If the schema does not exist, an error is then given

when an attempt is made to create the histograms table in

the non-existing schema.

This has been changed to check the error when the schema

creation attempt fails, so the correct error is generated.

This is a partial fix, because a user with manage_statistics

privilege should not be prevented from executing Update

Statistics just because the trafodion.hivestats schema did

not already exist. The likely full solution would be to

create the schema automatically at Trafodion initialization.

Change-Id: I26561100bab9f6a59f3f5d0d56ce9d2aab380fa3

Partial-Bug: #1425748

Closes-Bug: 1420490

Fixes node down of pnid > 63.

Change-Id: I583aa6a9bf948697be0d8e929834f20801bce339

    • -143
    • +162
    /sqf/monitor/linux/pnode.cxx
Merge "Move 'private' Seabed utilatomic.h to 'public' atomic.h."

Merge "Fix to take care of internal statements leak"

Merge "Fixes for security gaps"

Adding more run-time memory allocations from NAHeap

This set of changes moves some of the string vector variables in HBase

access operators from standard string template to our NAList and

NAString (or HbaseStr for row IDs). In the process, allocationis of the

objects will be from our HAHeap instead of the system heap. This would

help us tracking memory usage and detecting leaks easier.

In addition, a change in ExHbaseAccessTcb::setupListOfColNames()

prevents unnecessary allocations to populate the columns list unless it

is empty. The Google profiling tools helped us on identifying this

problem.

also, removed ExHbaseAccessDeleteTcb operator which was not used.

Change-Id: I87ab674ab8e3d291f2fc9563718d88de537ae96b

    • -29
    • +61
    /sql/executor/HBaseClient_JNI.cpp
    • -17
    • +78
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
    • -2
    • +25
    /sql/sqlcomp/CmpSeabaseDDLtable.cpp
Bug 1425650 Partial fix

1. Recursive mutex for maintaining SRVR_SESSION_HDL list

2. Disabled invocation of ESPs for repository queries

3. Updated WHERE clause for METRIC_QUERY_AGGR_TABLE.

Change-Id: If55f9de8f1541ba0a06b13f92750b64ff4676e0b

    • -25
    • +72
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +36
    /conn/odbc/src/odbc/nsksrvrcore/srvrcommon.cpp
Bug 1425650 Partial fix

1. Recursive mutex for maintaining SRVR_SESSION_HDL list

2. Disabled invocation of ESPs for repository queries

3. Updated WHERE clause for METRIC_QUERY_AGGR_TABLE.

Change-Id: If55f9de8f1541ba0a06b13f92750b64ff4676e0b

    • -25
    • +72
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +36
    /conn/odbc/src/odbc/nsksrvrcore/srvrcommon.cpp
Removing any hard coded "trafodion"

Removed all hard coded "trafodion" and replaced with $TRAF_USER

Change-Id: I78c5df3830b3264be91f98df3b5aa83c1689a2d6

Merge "Full backup and restore utilities"

Move 'private' Seabed utilatomic.h to 'public' atomic.h.

Change-Id: I438642536419fbd0ba762c98da256e11f2cc7657

    • -0
    • +104
    /sqf/export/include/seabed/atomic.h
    • -104
    • +0
    /sqf/src/seabed/src/utilatomic.h
SHOWDDL, QUERY Cancel, rework

This delivery addresses security issues with SHOWDDL, adds initial

support for security in query cancel, and implements part of the

proposed GIVE commands.

Bug 1414234: SHOWDDL command now check component privileges.

SHOW is granted to PUBLIC by default, so effectively there are

no new restrictions unless SHOW is revoked from PUBLIC.

SHOWDDL COMPONENT now checks for MANAGE_COMPONENTS or SHOW privilege.

SHOWDDL ROLE now checks for MANAGE_ROLES or SHOW privilege.

SHOWDDL SCHEMA now checks for SHOW privilege.

SHOWDDL USER now checks for MANAGE_USERS or SHOW privilege.

SHOWDDL LIBRARY is implemented. A user must have the USAGE

privilege on the library, or the MANAGE_LIBRARY or SHOW privilege.

New function to determine if the user canceling the query has

the authority: either DB__ROOT, or the user owns the query, or

the user has the QUERY_CANCEL privilege. Note, the code is

delivered in an inactive state pending future integration.

Three new component privileges are added: QUERY_ACTIVATE,

QUERY_CANCEL, and QUERY_SUSPEND. These will be added if

authorization is dropped and reinitialized. A future

delivery will add an INITIALIZE AUTHORIZATION,UPDATE

command that will add these privileges to an existing

instance with authorization enabled.

Support for library objects was added to NATable, but the code

is currently not used. May be integrated into CREATE ROUTINE

and GRANT for libraries in the future.

Also included is minor rework from delivery 1082, and the

GIVE SCHEMA command now updates associated privileges when object

ownership is changed. Note, GIVE commands are still prototype.

A detailed blueprint for GIVE will be released shortly.

This patch merges with changes from 1177 and addresses a couple of

minor comments from the initial submittal.

Change-Id: I60419228f886555ed0e066441bb824c5246ee498

  1. … 14 more files in changeset.
Errors reported for Update Stats on Hive tables

Attempts to execute Update Statistics statements on Hive

tables fail with error 1002 (catalog does not exist). The

catalog used to store Hive stats was no longer being

successfully created on demand. The Trafodion catalog is

now used instead, with a schema for Hive stats created

in it on first use. The tables for histograms and

histogram intervals for Hive tables are created in that

schema on first use.

Change-Id: Ib57d0af4a3da6f52f0544d6c2fce3e77d2e823c1

Closes-Bug: #1320397

Fix to take care of internal statements leak

Internal statements in Trafodion engine use ExeCliInterface. When

this interface is instantiated in a scope of method, statements

and other related objects can be leaked.

Added a destructor in ExeCliInterface so that these objects

are destroyed correctly when the interface goes out of scope.

This fix possibly takes care of memory growth when many statements

are compiled.

Change-Id: Ic47c912ae153e9f83cca298b66da40bf214ef3e7

Merge "Use Seabed atomics for platform compatibility"

Merge "DDL transactions TM changes"

scanner improvements

+ added new check YumAndInternetConnection to check that Yum is

configured properly and Internet connectivity is available

+ removed unixODBCVersion check to avoid confusion

Change-Id: I9be9f725baa72a71548a3174db014de59b622bf2

    • -8
    • +20
    /installer/tools/trafodion_scanner.cfg
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

    • -274
    • +271
    /sql/optimizer/BindRelExpr.cpp
  1. … 25 more files in changeset.
Added configuration support for snapshot scans and other misc settings

Set HDFS dfs.namenode.acls.enabled=true and added several setfacl commands

as configuration support for snapshot scans.

Set HBase hbase.regionserver.lease.period to 10 minutes, this setting

is synonymous with hbase.regionserver.lease.period

Fixed a few temp file paths

edit 1: added mkdir and chown for the acls directories for the build machines

which didn't seem to have them.

Change-Id: I3a00018ed3980be06b3c4886d015338d4c4cb11f

    • -1
    • +113
    /installer/traf_hortonworks_mods98
Merge "several scanner fixes and improvements"

Use Seabed atomics for platform compatibility

Change-Id: If333ac5d7a3cc06e41a462f99d69d9a16159fe5c