RelMisc.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
various lp and other fixes, details below.

-- added support for self referencing constraints

-- limit clause can now be specified as a param

(select * from t limit ?)

-- lp 1448261. alter table add identity col is not allowed and now

returns an error

-- error is returned if a specified constraint in an alter/create statement

exists on any table

-- lp 1447343. cannot have more than one identity columns.

-- embedded compiler is now used to get priv info during invoke/showddl.

-- auth info is is not reread if already initialized

-- sequence value function is now cacheable

-- lp 1448257. inserts in volatile table with identity column now work

-- lp 1447346. inserts with identity col default now work if inserted

in a salted table.

-- only one compiler is now needed to process ddl operations with or

without authorization enabled

-- query cache in embedded compiler is now cleared if user id changes

-- pre-created default schema 'SEABASE' can no longer be dropped

-- default schema 'SCH' is automatically created if running regressions

and it doesn't exist.

-- improvements in regressions run.

-- regressions run no longer call a script from another sqlci session

to init auth, create default schema

and insert into defaults table before every regr script

-- switched the order of regression runs

-- updates from review comments.

Change-Id: Ifb96d9c45b7ef60c67aedbeefd40889fb902a131

  1. … 69 more files in changeset.
SHOWDDL, QUERY Cancel, rework

This delivery addresses security issues with SHOWDDL, adds initial

support for security in query cancel, and implements part of the

proposed GIVE commands.

Bug 1414234: SHOWDDL command now check component privileges.

SHOW is granted to PUBLIC by default, so effectively there are

no new restrictions unless SHOW is revoked from PUBLIC.

SHOWDDL COMPONENT now checks for MANAGE_COMPONENTS or SHOW privilege.

SHOWDDL ROLE now checks for MANAGE_ROLES or SHOW privilege.

SHOWDDL SCHEMA now checks for SHOW privilege.

SHOWDDL USER now checks for MANAGE_USERS or SHOW privilege.

SHOWDDL LIBRARY is implemented. A user must have the USAGE

privilege on the library, or the MANAGE_LIBRARY or SHOW privilege.

New function to determine if the user canceling the query has

the authority: either DB__ROOT, or the user owns the query, or

the user has the QUERY_CANCEL privilege. Note, the code is

delivered in an inactive state pending future integration.

Three new component privileges are added: QUERY_ACTIVATE,

QUERY_CANCEL, and QUERY_SUSPEND. These will be added if

authorization is dropped and reinitialized. A future

delivery will add an INITIALIZE AUTHORIZATION,UPDATE

command that will add these privileges to an existing

instance with authorization enabled.

Support for library objects was added to NATable, but the code

is currently not used. May be integrated into CREATE ROUTINE

and GRANT for libraries in the future.

Also included is minor rework from delivery 1082, and the

GIVE SCHEMA command now updates associated privileges when object

ownership is changed. Note, GIVE commands are still prototype.

A detailed blueprint for GIVE will be released shortly.

This patch merges with changes from 1177 and addresses a couple of

minor comments from the initial submittal.

Change-Id: I60419228f886555ed0e066441bb824c5246ee498

  1. … 28 more files in changeset.
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

  1. … 39 more files in changeset.
Log reading TMUDF, phase 3

blueprint cmp-tmudf-compile-time-interface

- Addressed review comments from phase 2. See

https://review.trafodion.org/#/c/824

- Added a "parse_status" column to the TMUDF, see

updated syntax below

- Added versioning info to new DLL libudr_predef.so

- EVENT_LOG_READER TMUDF now should choose the correct

degree of parallelism without the need for CQDs

- Brought back the REPLICATE PARTITION keyword, which

is used in the TMUDF syntax. This should fix the failure

in regression test udf/TEST108.

- Some remaining issues:

- Newlines in the error message are not handled well,

at best the additional lines are lost, at worst

they will cause parse errors

- log_file_node output column is always 0

- Code is not yet integrated with changes to event

logging

- Not yet tested on clusters

Updated syntax for the log reader TMUDF:

SQL Syntax to invoke this function:

select * from udf(event_log_reader( [options] ));

The optional [options] argument is a character constant. The

following options are supported:

f: add file name output columns (see below)

t: turn on tracing

d: loop in the runtime code, to be able to attach a debugger

(debug build only)

p: force parallel execution on workstation environment with

virtual nodes (debug build only)

Returned columns:

log_ts timestamp(6),

severity char(10 bytes) character set utf8,

component char(24 bytes) character set utf8,

node_number integer,

cpu integer,

pin integer,

process_name char(12 bytes) character set utf8,

sql_code integer,

query_id varchar(200 bytes) character set utf8,

message varchar(4000 bytes) character set utf8

if option "f" was specified, we have four more columns:

log_file_node integer not null,

log_file_name varchar(200 bytes) character set utf8 not null,

log_file_line integer not null,

parse_status char(2 bytes) character set utf8 not null

(log_file_node, log_file_name, log_file_line) form a unique key

in the result table. parse_status indicates whether there were

any errors reading the information:

' ' (two blanks): no errors

'E' (as first or second character): parse error

'T' (as first or second character): truncation or over/underflow

occurred

'C' (as first or second character): character conversion error

Change-Id: Iee3fc8383d4125f0f9b6c6035aa90bb82ceee92e

  1. … 10 more files in changeset.
Phase 2 for log reader TMUDF

blueprint cmp-tmudf-compile-time-interface

Log reader TMUDF is mostly working now.

Still need to set cqd NUM_PARALLEL_ESPS '<num of nodes>' on clusters.

Still needs more work and more testing.

Still seeing some issues with non-ASCII characters.

// SQL Syntax to invoke this function:

//

// select * from udf(event_log_reader( [options] ));

//

// The optional [options] argument is a character constant. The

// following options are supported:

// f: add file name output columns (see below)

// t: turn on tracing

// d: loop in the runtime code, to be able to attach a debugger

// (debug build only)

// p: force parallel execution on workstation environment with

// virtual nodes (debug build only)

More detailed explanation of changes:

- PredefUdrReadfile.cpp: Work on event log reader TMUDF

- sqludr.*: New method to add formal parameters, allows TMUDF to

accept optional parameters.

- OptPhysRelExpr.cpp:

Made some changes for TMUDFs with arity 0 to avoid asserts

and to be able to call okToAttemptESPParallelism in method

RelExpr::synthPhysicalProperty(). This is needed for leaf

operators (arity 0) that want to initiate parallel execution

and TMUDFs seem to be first in that situation.

Changed TableMappingUDF::synthPhysicalProperty to generate

a partitioning function with multiple partitions (and no

partitioning key, so far) if required.

- ExUdr.cpp,

GenUdr.cpp,

sqlparser.y,

DefaultConstants.h,

nadefaults.cpp,

BindItemExpr.cpp,

BindRelExpr.cpp,

ValueDesc.ppp:

Addressed review comments from last phase, got rid of ALLOW_UDF CQD

- Rel*.h

Rel*.cpp

OptPhysRelExpr.cpp (has other changes as well)

Simple but messy change to add one more parameter to

RelExpr::synthPhysicalProperty

Change-Id: I5549e47c0f019beefd4ec1695ae7abf8c3bd43e3

  1. … 39 more files in changeset.
Various LP fixes, bugs and code cleanup.

-- removed obsolete code (label create/alter/delete, get disk/label/buffer stats,

dp2 scan)

-- metadata structs are now created as classes and initialized during

creation. LP 1394649

-- warnings are now being returned from compiler to executor after DDL operations.

-- duplicate constraint names now return error.

-- handle NOT ENFORCED constraints: give warning during creation and not enforce

during use. LP 1361784

-- drop all indexes (enabled and disabled indexes) on a table during drop table

and schema now works. LP 1384380

-- drop constraint on disabled index succeeds. LP 1384479

-- string truncation error is now returned if default value doesn't fit in

column. LP 1394780

-- fixed issue where a failure during multiple constraints creation in a create

stmt was not cleaning up metadata. LP 1389871

-- update where current of is now supported. LP 1324679

Change-Id: Iec1b0b4fc6a8161a33b7f69228c0f1e3f441f330

  1. … 54 more files in changeset.
Identity column and sequence numbers support.

Added support for IDENTITY columns.

Finished sequence numbers functionality.

Bug fixes and perf enhancements in those areas.

This code has been pre-reviewed by Joanie C.

Change-Id: I0445bc9765b60becb9adf8c053c05344395aecaa

  1. … 94 more files in changeset.
Changes to support OSS poc.

This checkin contains multiple changes that were added to support OSS poc.

These changes are enabled through a special cqd mode_special_4 and not

yet externalized for general use.

A separate spec contains details of these changes.

These changes have been contributed and pre-reviewed by Suresh, Jim C,

Ravisha, Mike H, Selva and Khaled.

All dev regressions have been run and passed.

Change-Id: I2281c1b4ce7e7e6a251bbea3bf6dc391168f3ca3

  1. … 143 more files in changeset.
Security changes to support authorization

Added support for authorization commands:

- initialize authorization [, drop]

- create/drop roles

- register/unregister components

- create/drop component operations

- grant/revoke object privileges

- grant/revoke role privileges

- grant/revoke component privileges

- updates to GET and SHOWDDL statements

- checking of privileges for DML requests

- checking of privileges for DDL requests

- regression tests added to catman1 library

Fixed a testware problem in catman1 TEST135 and TEST139

Fixed a parser problem introduced by compGeneral/TESTTOK2 which was recently

introduced.

More details:

This delivery was part of code worked on by many people for several

months on a remote branch. This team held bi-weekly meetings

for several months to design and implement these features. These

meetings also included extensive code reviews.

The security features which include authentication (which was delivered

in June) and authorization is turned off by default. The

traf_authentication_setup script located in $MY_SQROOT/sql/scripts needs

to be run to enable both authentication and authorization. This

procedure is described on the Trafodion Twiki page and will be updated once this

delivery completed to include authorzation.

Delivery updates:

Updated traf_authentication_setup to return consistent error messages

and added a comment to ComSmallDefs.h to address a buf size issue for

metadata tables.

Change-Id: I896f1ee006590284653b2c9882901c05b5f2ba22

  1. … 100 more files in changeset.
Squashed commit of the following:

commit 221d4199001b3f06d5629b82ed1281bdfb95f043

Author: qchen <qifan.chen@hp.com>

Date: Mon Sep 8 19:32:05 2014 +0000

DoP: use a better version of resource estimator

Change-Id: Idf6ea5caa7e4915c65b4d54f58d7e483c26871f4

commit 75c92f255ad3841102757f6188d749db42eaedf3

Author: qchen <qifan.chen@hp.com>

Date: Fri Sep 5 15:55:11 2014 +0000

do not add ESP partition requirement to partial GB leaf1 and leaf2.

commit f7c652adf2953d18f10da4a952f78d3815b5b3d4

Author: qchen <qifan.chen@hp.com>

Date: Fri Sep 5 15:20:40 2014 +0000

finalize

commit a419954b1528ca0d1f84c5298b806b46f561a713

Author: qchen <qifan.chen@hp.com>

Date: Wed Sep 3 23:45:13 2014 +0000

add a check on stats smart ptr before its use.

commit 8a2dcc24f0081d554ec3d69560c050f720da04dd

Author: qchen <qifan.chen@hp.com>

Date: Wed Sep 3 18:31:50 2014 +0000

partial gb root running in Master; use rowcount from stats for Hbase

in AppliedStatMan::getStatsForCANodeId().

commit 55f46ededa9fcb30d38d17be6692e1535b4a274c

Author: qchen <qifan.chen@hp.com>

Date: Fri Aug 29 17:46:00 2014 +0000

improve the logic to disable parallel GB partial root when rows < 5000

commit 743783720d030f38e50a065737c8c218fa47c6b5

Author: Ravisha Neelakanthappa <Ravisha.Neelakanthappa@hp.com>

Date: Wed Aug 20 17:20:09 2014 +0000

Fix for bug 1348317. Enable Adaptive Segmentation logic to

compute DoP based on resource estimation.

Change-Id: I7c2b0756854d5a86d8bde162dad064b4e924233c

Change-Id: Id68e1d8265e478f128ed61d1cabc4849873dae23

  1. … 19 more files in changeset.
Code cleanup (06042014)

Change-Id: Ib7f2f39e42085e026ac1000e43048c9a185b6976

  1. … 14 more files in changeset.
Initial code drop of Trafodion

  1. … 4886 more files in changeset.