Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Misleading error when Hivestats can't be created

When Update Statistics is performed on a Hive table, the

trafodion.hivestats schema is created if it doesn't already

exist. If the user issuing the ustat statement does not

have the create_schema component privilege, this will fail, but

the code as it existed ignored the error, assuming it was

a "schema already exists" error, which is expected in most

cases. If the schema does not exist, an error is then given

when an attempt is made to create the histograms table in

the non-existing schema.

This has been changed to check the error when the schema

creation attempt fails, so the correct error is generated.

This is a partial fix, because a user with manage_statistics

privilege should not be prevented from executing Update

Statistics just because the trafodion.hivestats schema did

not already exist. The likely full solution would be to

create the schema automatically at Trafodion initialization.

Change-Id: I26561100bab9f6a59f3f5d0d56ce9d2aab380fa3

Partial-Bug: #1425748

Closes-Bug: 1420490

Fixes node down of pnid > 63.

Change-Id: I583aa6a9bf948697be0d8e929834f20801bce339

    • -143
    • +162
    /sqf/monitor/linux/pnode.cxx
Merge "Move 'private' Seabed utilatomic.h to 'public' atomic.h."

Merge "Fix to take care of internal statements leak"

Merge "Fixes for security gaps"

Adding more run-time memory allocations from NAHeap

This set of changes moves some of the string vector variables in HBase

access operators from standard string template to our NAList and

NAString (or HbaseStr for row IDs). In the process, allocationis of the

objects will be from our HAHeap instead of the system heap. This would

help us tracking memory usage and detecting leaks easier.

In addition, a change in ExHbaseAccessTcb::setupListOfColNames()

prevents unnecessary allocations to populate the columns list unless it

is empty. The Google profiling tools helped us on identifying this

problem.

also, removed ExHbaseAccessDeleteTcb operator which was not used.

Change-Id: I87ab674ab8e3d291f2fc9563718d88de537ae96b

    • -29
    • +61
    /sql/executor/HBaseClient_JNI.cpp
    • -17
    • +78
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
    • -2
    • +25
    /sql/sqlcomp/CmpSeabaseDDLtable.cpp
Bug 1425650 Partial fix

1. Recursive mutex for maintaining SRVR_SESSION_HDL list

2. Disabled invocation of ESPs for repository queries

3. Updated WHERE clause for METRIC_QUERY_AGGR_TABLE.

Change-Id: If55f9de8f1541ba0a06b13f92750b64ff4676e0b

    • -25
    • +72
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +36
    /conn/odbc/src/odbc/nsksrvrcore/srvrcommon.cpp
Bug 1425650 Partial fix

1. Recursive mutex for maintaining SRVR_SESSION_HDL list

2. Disabled invocation of ESPs for repository queries

3. Updated WHERE clause for METRIC_QUERY_AGGR_TABLE.

Change-Id: If55f9de8f1541ba0a06b13f92750b64ff4676e0b

    • -25
    • +72
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -1
    • +36
    /conn/odbc/src/odbc/nsksrvrcore/srvrcommon.cpp
Removing any hard coded "trafodion"

Removed all hard coded "trafodion" and replaced with $TRAF_USER

Change-Id: I78c5df3830b3264be91f98df3b5aa83c1689a2d6

Merge "Full backup and restore utilities"

Move 'private' Seabed utilatomic.h to 'public' atomic.h.

Change-Id: I438642536419fbd0ba762c98da256e11f2cc7657

    • -0
    • +104
    /sqf/export/include/seabed/atomic.h
    • -104
    • +0
    /sqf/src/seabed/src/utilatomic.h
SHOWDDL, QUERY Cancel, rework

This delivery addresses security issues with SHOWDDL, adds initial

support for security in query cancel, and implements part of the

proposed GIVE commands.

Bug 1414234: SHOWDDL command now check component privileges.

SHOW is granted to PUBLIC by default, so effectively there are

no new restrictions unless SHOW is revoked from PUBLIC.

SHOWDDL COMPONENT now checks for MANAGE_COMPONENTS or SHOW privilege.

SHOWDDL ROLE now checks for MANAGE_ROLES or SHOW privilege.

SHOWDDL SCHEMA now checks for SHOW privilege.

SHOWDDL USER now checks for MANAGE_USERS or SHOW privilege.

SHOWDDL LIBRARY is implemented. A user must have the USAGE

privilege on the library, or the MANAGE_LIBRARY or SHOW privilege.

New function to determine if the user canceling the query has

the authority: either DB__ROOT, or the user owns the query, or

the user has the QUERY_CANCEL privilege. Note, the code is

delivered in an inactive state pending future integration.

Three new component privileges are added: QUERY_ACTIVATE,

QUERY_CANCEL, and QUERY_SUSPEND. These will be added if

authorization is dropped and reinitialized. A future

delivery will add an INITIALIZE AUTHORIZATION,UPDATE

command that will add these privileges to an existing

instance with authorization enabled.

Support for library objects was added to NATable, but the code

is currently not used. May be integrated into CREATE ROUTINE

and GRANT for libraries in the future.

Also included is minor rework from delivery 1082, and the

GIVE SCHEMA command now updates associated privileges when object

ownership is changed. Note, GIVE commands are still prototype.

A detailed blueprint for GIVE will be released shortly.

This patch merges with changes from 1177 and addresses a couple of

minor comments from the initial submittal.

Change-Id: I60419228f886555ed0e066441bb824c5246ee498

  1. … 14 more files in changeset.
Errors reported for Update Stats on Hive tables

Attempts to execute Update Statistics statements on Hive

tables fail with error 1002 (catalog does not exist). The

catalog used to store Hive stats was no longer being

successfully created on demand. The Trafodion catalog is

now used instead, with a schema for Hive stats created

in it on first use. The tables for histograms and

histogram intervals for Hive tables are created in that

schema on first use.

Change-Id: Ib57d0af4a3da6f52f0544d6c2fce3e77d2e823c1

Closes-Bug: #1320397

Fix to take care of internal statements leak

Internal statements in Trafodion engine use ExeCliInterface. When

this interface is instantiated in a scope of method, statements

and other related objects can be leaked.

Added a destructor in ExeCliInterface so that these objects

are destroyed correctly when the interface goes out of scope.

This fix possibly takes care of memory growth when many statements

are compiled.

Change-Id: Ic47c912ae153e9f83cca298b66da40bf214ef3e7

Merge "Use Seabed atomics for platform compatibility"

Merge "DDL transactions TM changes"

scanner improvements

+ added new check YumAndInternetConnection to check that Yum is

configured properly and Internet connectivity is available

+ removed unixODBCVersion check to avoid confusion

Change-Id: I9be9f725baa72a71548a3174db014de59b622bf2

    • -8
    • +20
    /installer/tools/trafodion_scanner.cfg
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

    • -274
    • +271
    /sql/optimizer/BindRelExpr.cpp
  1. … 25 more files in changeset.
Added configuration support for snapshot scans and other misc settings

Set HDFS dfs.namenode.acls.enabled=true and added several setfacl commands

as configuration support for snapshot scans.

Set HBase hbase.regionserver.lease.period to 10 minutes, this setting

is synonymous with hbase.regionserver.lease.period

Fixed a few temp file paths

edit 1: added mkdir and chown for the acls directories for the build machines

which didn't seem to have them.

Change-Id: I3a00018ed3980be06b3c4886d015338d4c4cb11f

    • -1
    • +113
    /installer/traf_hortonworks_mods98
Merge "several scanner fixes and improvements"

Use Seabed atomics for platform compatibility

Change-Id: If333ac5d7a3cc06e41a462f99d69d9a16159fe5c

DDL transactions TM changes

Added functionality at TM startup time where TM0 can

create a table and set a region that will be used for the coprocessor

work and used for servicing the DDL requests from SQL.

TDDL table will be created at trafodion startup time, if the table

is already created, nothing happens. If it has not been created,

TM0 will be the only one creating this table since only one table

per cluster will be needed.

This table will only be deleted when uninstall trafodion happens.

Changed table name to TRAFODION._DTM_.TDDL

Copyright date updated.

Added details to try/catch methods.

Change-Id: Ie48b7a0807f939e5b4072d9695d6ba93467734c8

    • -2
    • +5
    /sqf/src/seatrans/tm/hbasetmlib2/Makefile
    • -0
    • +75
    /sqf/src/seatrans/tm/hbasetmlib2/TmDDL.java
Fix for bug 1404430 Phoenix T2 tests core dumping in CliStatement::execute

Phoneix T2 tests involve multiple JDBC connections in a single thread and

hence multiple CLI contexts in a single thread. With cancel

feature enabled, there is an outstanding nowaited query started message.

In the master process, SQL always wait on a set of IPC connections pertaining

to the current JDBC/ODBC connection for the nowaited IOs to complete in every

SQL thread.

With fast completion enabled in master, the SQL engine was completing the

IO for all the thread specific nowait IO calls without limiting itself

to a specific JDBC connection.

Disabled fast completion in Type 2 JDBC driver that enables set of

connection based nowait completion.

Change-Id: Ia1b6323acb3aeb1edc6bbd9eb3e242f4bedade63

    • -1
    • +2
    /conn/jdbc_type2/native/SQLMXCommonFunctions.cpp
Fix for bug 1329361 and minor improvements in hive scan

This is a joint checkin with Sandhya Sundaresan.

PreOpens during hive scan were not working correctly, cause the same range

to be read twice. The read triggered by the preopen was not used since it had

incorrect cursor name. This is now fixed.

Reduced number of LOB threads to 2

Removed multi cursor code since it is not used

Skip reading the first range if it has 0 bytes to be read.

Change-Id: I91cff41134490435165da7d59c955c7215b3c6b8

    • -229
    • +139
    /sql/executor/ExHdfsScan.cpp
Removed qpid dependency from connectivity

Change-Id: Icaffc13ca8b8e3699dadcebc3e26b1e9796f2507

    • -1
    • +1
    /conn/odbc/src/odbc/nsksrvrcore/Makefile
Script name change, Java path and uninstaller bug

Changed name of traf_installer to traf_start to avoid confusion with new

script trafodion_install.

Checking to make sure there is a correct Java path in Java version

check.

Added fix to trafodion_uninstaller to all --instance option to be ran by

root or sudo user.

Change-Id: I263f5d9969f3e53e745dc497ceead0c33ece873c

    • -0
    • +282
    /installer/traf_start
DcsServer checks if Trafodion is running before mxosrvr restart

DcsServer aggresively tried to restart its child mxosrvrs.

In scenarios where Trafodion had been stopped abruptly e.g.,

when "ckillall" command was executed, it would cause many

mxosrvr processes to start then quickly die. This change

adds check to ensure that Trafodion is running before ever

attempting a restart of any mxosrvr.

Change-Id: If430ce94e6071e402786d73d544d2ab23306d1fe

Closes-Bug: #1411475

Full backup and restore utilities

Shell scripts to perform full offline backup and restore

of all trafodion tables including meta data tables. Trafodion

needs to be shutdown before doing the backup or restore operations.

The Backup operation takes snapshots of all the trafodion tables

and exports them using the ExportSnapshot MapReduce job to a HDFS

location. The restore operations also use the ExportSnapshot

MapReduce job to import the snapshots back from the HDFS location

and the restore the tables.

The backup and restore were tested on clusters with Cloudera and

Horton Works distributions. it was also tested on the development

workstations

The Blueprint can be found at:

https://blueprints.launchpad.net/trafodion/+spec/full-backup-restore

Change-Id: I120a34eb4eb94e286577b4e6dc529ca528f0b846

Cloud code removal, fix Java check, uninstaller fix

Removed all code checking for HP Cloud since it is not need.

Fixed Java version check as it was not currently working as expected

Made changes to uninstaller.

Change-Id: I4b6a61f14f0870b0d8c921765532f317243a1147

Don't exclude _SALT_ from requested histograms

The optimizer sets the referenced property for non-user columns

to NOT_REFERENCED, which may override a prior value indicating

that it is referenced for histograms. This prevents Update

Stats automation from creating an empty histogram for it to be

replaced later with an actual histogram when Update Statistics

... On Necessary Columns is executed. The _SALT_ column is part

of the primary key, however, and it is useful to have a histogram

for it. The code was changed to avoid overwriting the referenced

field for the salt column so that its histogram might be used.

It is excluded, however, from the list of warnings for histograms

that are requested but not present, as it is not a column explicitly

created by the user.

Change-Id: If13485a3887484e83fec26e9422050024cad5e62

Closes-Bug: #1411472