CmpSeabaseDDLtable.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
fixes bug 1464364

Update Stats on a table in a schema with a delimited identifier name

failed with a funny syntax error. Turns out in a new sqlci session,

Showddl on the same table fails the same way.

The root cause was the schema name was passed to a ComObjectName

constructor in internal format, when external format was required.

This has been fixed.

Change-Id: Ic81ba3d69e006264340fbdcbff6fb455593e30cf

Fixes for 1450193, 1370754, and performance change

1450193 - GRANT syntax for SHOWDDL output incorrect

1370754 - Objects with delimited names not displayed correctly for SHOWDDL

Made sure names in GRANT statements were qualified correctly and added the

object type, when required. Also changed SHOWDDL commands so they don't

start a tdm_arkcmp to get data details with the except of showddl user and

showddl role.

Performance: changed the default in PrivMgr requests to assume that

authorization is enabled. The PrivMgr code that checks to see if authorization

is enabled causes a new tdm_arkcmp to be started. Since all callers already

check authorization from a cached value before calling PrivMgr, this avoids the

tdm_arkcmp process overhead.

Patch set 2:

Updated copyright

Patch set 3:

Fixed problem in isDelimited method.

Patch set 4:

Fixed another problem in isDelimited method.

Change-Id: I43ff421ddd8e9c508c8fed79a1e2b2c6448456b7

  1. … 8 more files in changeset.
drop column fix.

We currently dont support drop of a column which has an

index (implicit or explicit) defined on it.

But drop column was also not working in cases where a secondary

index was present on a column that was not being dropped.

That has been fixed.

Change-Id: I53264b630794bef5c99ffb7a57b8fd2ba0f13bb5

  1. … 2 more files in changeset.
various fixes and enhancements, details below.

-- improved DDL performance by not invalidating internal create/alter

operations.

-- added an optimization during CREATE INDEX to not go through

'upsert using load' processing if source table is empty.

-- added support for ISO datetime format (2015-06-01T07:35:20Z)

-- added support for RESET option to ALTER SEQUENCE and IDENTITY.

This will reset generated seq num to the START VALUE.

-- added support for cqd TRAF_STRING_AUTO_TRUNCATE.

If set, strings will be automatically truncated during insert/update.

-- fixed sqlci to pass in correct varchar param len indicator (2 or 4 bytes).

-- changed sizeof(short) to correct vcindlen (2 or 4 bytes)

-- removed some NA_SHADOWCALLS defines

Change-Id: Ie6715435d9c210ae6c2db4ff6bc0545c1b196979

  1. … 39 more files in changeset.
Part 2 of ALTER TABLE/INDEX ALTER HBASE_OPTIONS support

blueprint alter-table-hbase-options

This set of changes includes non-transactional HBase semantics

for ALTER TABLE/INDEX ALTER HBASE_OPTIONS. As part of this

change, two sets of logic that manipulate HBase options in

the CREATE TABLE code path were refactored so that logic

could be reused by ALTER TABLE/INDEX. In both these cases,

the logic in question was taken out of a create-side method

and moved into a separate method.

Still to come is the equivalent transactional support, and

also a regression test script that exercises this feature.

With this change, ALTER TABLE/INDEX ALTER HBASE_OPTIONS works

but only in a non-transactional manner. That is, if the

operation is aborted after the request has been issued to HBase,

the change to the Trafodion metadata may be rolled back but

the change to the underlying HBase object may still complete.

Change-Id: Ied11d0b2ac7b7d65110d39a1e5a7fbbac1e1f827

  1. … 9 more files in changeset.
Column-level privileges - part 2

Support for column-level privileges will be in multiple deliveries.

This delivery add the following portions:

1. DML operations (SELECT, INSERT, UPDATE) now recognize granted

column-level privileges.

2. CREATE VIEW now recognizes granted column-level privileges.

3. Revoke of object-level privileges now revokes the corresponding

column-level privilege.

Missing functionality:

1. Privileges can be granted to roles and revoked from roles, but

REVOKE ROLE does not consider column-level privileges when

determining

if an object depends on a role's granted privileges.

2. Column-level revoke does not enforce RESTRICT, i.e., privileges

may be revoked even if there are dependent privileges.

3. ALTER TABLE DROP COLUMN does not remove associated column-level

privileges, nor does it check for dependent objects.

Change-Id: Ieba04c77edb945dfeb1994e9949b54072289465e

  1. … 13 more files in changeset.
Turned on privilege features, reorg'd PrivMgr code

Turned on GRANTED BY, WITH GRANT OPTION, GRANT OPTION FOR clauses

For GRANTED BY -> showddl now displays the GRANTED BY clause when

--> the current user is not the object owner and

--> the current user is not DB ROOT

added object_owner and schema_owner to the SeabaseLibraryDesc

ComTdb.h

Generator.cpp

desc.h

CmpSeabaseDDLtable.cpp ->getSeabaseLibraryDesc

added object owner in calls to PrivMgrPrivileges::getPrivTextForObject

CmpDescribe.cpp

PrivMgrCommands.cpp/.h

PrivMgrPrivileges.cpp/.h

Changed object grant and revoke to store the list of privileges associated

with the object and columns in the PrivMgrPrivileges class.

--> added new methods generateObjectRowList and generateColumnRowList,

changed the destructor to remove these lists, changed code to call

these new methods, and removed extra I/Os

--> removed member trafMetadataLocation_ (it is already stored in parent)

For WITH GRANT OPTION clause at GRANT time:

--> added checks at grant time to:

--> check for potential circular grants (error 1036)

--> added new method getTreeOfGrants to get list of grantors

that have previously granted to the current grantee

For GRANT OPTION FOR clause at REVOKE time:

--> changed error messages returned to be more meaningful

--> moved and activated call checkRevokeRestrict after call to

getAffectedObjects

Added new columns to the COLUMN_PRIVILEGES and SCHEMA_PRIVILEGES tables

to include the object_name, grantor_name, and grantee_name to match

OBJECT_PRIVILEGES, ROLE_USAGES, and COMPONENT_PRIVILEGES;

Reorganized the contents of PrivMgr files:

--> PrivMgr document exists that describes the .h/.cpp structure

--> Added new files PrivMgr.h/PrivMgr.cpp that describes the parent

class for all PrivMgr requests

--> moved existing defines, classes, etc around to match the PrivMgr

document

Fixed a couple of issues:

--> Fixed a bug in initialize authorization where the WGO was not set up

correctly for UDR's

--> Fixed a bug in PrivMgrObject::selectAllWhere where an error condition

was not returned

--> Fixed a bug in seabaseGrantRevoke where the incorrect object type was

sent for views

--> Fixed a bug in update statistics privilege checking that was not

handling HBase tables correctly

Added two regression tests (skipped until catman1 test directory is split up)

--> TEST132 - tests for privilege checking on libraries, populate index,

showddl, invoke, update statistics, and showstats

--> TEST140 - tests for WITH GRANT OPTION and GRANTED by option

Fixed expected result for catman1/TEST137

Change-Id: Iaf523aef763b0bce2101fedae0ee701606c369c7

  1. … 51 more files in changeset.
Stop sending user CQDs at embedded compile switch

Do not send user entered CQDs when switching to embedded META CmpContext

to compile metadata queries.

Also do the holdAndSet needed CQDs only when the embedded (META)

CmpContext is first switched to. These CQDs won't be restored when

switching back so that we won't send the same set of CQDs again next

time the CmpContext is used.

LP bug 1438372, Compiler returns internal error from

../optimizer/opt.cpp line 6907

LP bug 1444739, compGeneral/TEST006 failed with tmp fix for bug 1438372

Change-Id: I92398aeeaf53820d3f8af1fb73e9c1d6f449ce5e

  1. … 7 more files in changeset.
Merge "various lp and other fixes, details below."

  1. … 11 more files in changeset.
Merge "Part 1 of ALTER TABLE/INDEX ALTER HBASE_OPTIONS support"

  1. … 1 more file in changeset.
Part 1 of ALTER TABLE/INDEX ALTER HBASE_OPTIONS support

blueprint alter-table-hbase-options

This set of changes includes the following:

1. Syntax for ALTER TABLE/INDEX ALTER HBASE_OPTIONS

2. Compiler binding and generation logic

3. Run-time logic to update the metadata TEXT table with the

new options.

Still to come is:

1. Logic to actually change the HBase object

2. Transactional logic to actually change the HBase object

The functionality in this set of changes is only marginally

useful. If you manually change hbase options on a Trafodion

object using the hbase shell, you could use this ALTER

TABLE/INDEX command to update the Trafodion metadata. (Of

course some care would have to be taken to use the same

options!).

Change-Id: Id0a5513fe80853c06acdbbf6cc9fd50492fd07b2

  1. … 18 more files in changeset.
various lp and other fixes, details below.

-- added support for self referencing constraints

-- limit clause can now be specified as a param

(select * from t limit ?)

-- lp 1448261. alter table add identity col is not allowed and now

returns an error

-- error is returned if a specified constraint in an alter/create statement

exists on any table

-- lp 1447343. cannot have more than one identity columns.

-- embedded compiler is now used to get priv info during invoke/showddl.

-- auth info is is not reread if already initialized

-- sequence value function is now cacheable

-- lp 1448257. inserts in volatile table with identity column now work

-- lp 1447346. inserts with identity col default now work if inserted

in a salted table.

-- only one compiler is now needed to process ddl operations with or

without authorization enabled

-- query cache in embedded compiler is now cleared if user id changes

-- pre-created default schema 'SEABASE' can no longer be dropped

-- default schema 'SCH' is automatically created if running regressions

and it doesn't exist.

-- improvements in regressions run.

-- regressions run no longer call a script from another sqlci session

to init auth, create default schema

and insert into defaults table before every regr script

-- switched the order of regression runs

-- updates from review comments.

Change-Id: Ifb96d9c45b7ef60c67aedbeefd40889fb902a131

  1. … 69 more files in changeset.
Using the language manager for UDF compiler interface

blueprint cmp-tmudf-compile-time-interface

This change includes new CLI calls, to be used in the compiler to

invoke routines. Right now, only trusted routines are supported,

executed in the same process as the caller, but in the future we may

extend this to isolated routines. Using a CLI call allows us to share

the language manager between compiler and executor, since language

manager resources such as the JVM and loaded DLLs exist only once per

process. This change is in preparation for Java UDFs.

Changes in a bit more detail:

- Added 4 new CLI calls to allocate a routine, invoke it, retrieve

updated invocation and plan infos and deallocate (put) the routine.

The CLI globals now have a C/C++ and a Java language manager that

is allocated on demand.

- The compiler no longer loads a DLL for the UDF compiler interface,

it uses the new CLI calls instead.

- DDL syntax is changed to allow TMUDFs in Java (not officially

supported, so don't use it quite yet).

- TMUDFs in C are no longer supported, only C++ and Java are.

Converted remaining TMUDF tests to C++.

- C++ TMUDFs now do a basic verification at DDL time, so errors

like missing entry points are detected earlier. Validation for

Java TMUDFs is also done through the CLI.

- Make sure we have no memory or resource leaks:

- CmpContext keeps track of UDF-related objects allocated on

system heap and in the CLI, cleaned up at the end of a statement

- CLI keeps a list of allocated trusted routines, cleaned up

when a CLI context is deallocated

- Using ExeCliInterface class to make the new CLI calls (4 new calls

added).

- Removed CmpCli class in the optimizer directory and converted

tracking compiler to use ExeCliInterface as well.

- Compile-time parameter values are no longer baked into the

UDRInvocationInfo. Instead, they are provided as an input row, the

same way as they are provided at runtime.

- Bug fixes in C++ UDR code, mostly related to serialization and

to multiple interactions with the UDF through serialized objects.

- Added more info to UDRInvocationInfo (SQL access type, etc.).

- Since there are multiple plans per invocation, each of which

can have multiple interactions with the UDF, plans need to be

numbered so the UDF side can tell them apart to attach the

right state (owned by the UDF) to it.

- The language manager needs some functions that are provided by

the process it's running in. Added those (empty, for now) functions

as cli/CliImplLmExtFunc.cpp.

- Added a new class for Java TMUDFs, LmRoutineJavaObj. Added methods

to allocate such routines and to load their class as well as to

create Java objects by invoking the default constructor through JNI.

- Java TMUDFs use the new UDR interface (to be provided by Suresh and

Pavani). In the language manager, the container is the class of

the UDF, the external path is the fully qualified jar name. The

Java method name is <init>, the default constructor, with signature

"()V". Some code changes were required to do this.

- Created a new directory trafodion/core/sql/src for Java sources in

the sql engine. Right now, only language manager java

sources are in this directory, but I am planning to move the other

java sources under sql in a future checkin. Suresh and Pavani

will add their UDF-related Java files there as well.

- Renamed the udr jar to trafodion-sql-<version>.jar, in anticipation

of combining all the sql Java sources into this jar.

- Created a maven project file trafodion/core/sql/pom.xml and

changed makefiles to invoke maven to build java sources.

- More work to separate new UDR interface from older SPInfo object,

so that we can get rid of SPInfo if/when we don't support the older

style anymore.

- Small fix to odb makefile, make clean failed when executed twice.

Patch set 2: Adding a custom filter for test regress/udr/TEST108.

Change-Id: Ic827a42ac25505fb1ee451b79636c0f9349d8841

  1. … 98 more files in changeset.
Column-level privileges

Support for column-level privileges will be in multiple deliveries.

This delivery add the following portions:

1. Creation of the metadata table COLUMN_PRIVILEGE.

This table is created when the INITIALIZE AUTHORIZATION command is run.

Existing privileges are preserved, but warnings are issued referring to

existing metadata tables. An UPDATE option will be added later.

2. Granting of column-level privileges

Full support is present for granting column-level privileges.

Privileges can be added and updated for one or more columns on a table or view.

Support for WITH GRANT OPTION is coded, though not enabled until WITH GRANT

OPTION is enabled at the object level.

3. SHOWDDL

The SHOWDDL command displays column-level privileges. Regardless of

the order the privileges were granted, SHOWDDL displays them in column

order, and within each column, in the order they appear in the bitmap

(SELECT, INSERT, UPDATE, REFERENCES).

4. Revoking of column-level privileges

Only partially implemented. The basic operation of revoking granted

column-level privileges and grant option for is implemented. All

relevant security checks are performed. GRANTED BY is not implemented.

RESTRICT and CASCADE options are not supported. Hence, any dependent

objects remain when column-level privileges are revoked.

Missing functionality

In addition to column-level revoke only be partially implemented,

here are other items not present in this delivery:

1. Privileges can be granted to roles and revoked from roles,

but REVOKE ROLE does not consider column-level privileges when

determining if an object depends on a role's granted privileges.

2. Similarly, revoke at the object level does not consider

column-level privileges that may allow an object to remain after

an object-level privilege is revoked.

3. CREATE VIEW does not consider column-level privileges

when determining if the user has authority on the referenced

tables and views.

4. Run-time DML operations do not considered column-level when

determining if the user has authority to perform the query.

Change-Id: Icd3db88708d1e0ae7e9236e10b2a760bba287155

  1. … 17 more files in changeset.
Security fixes for 144553, 1414125, and 1393529

1445583: showstats command performance slow with security enabled

Several changes were made to improve performance:

Performance optimization:

NATable.cpp: NATable::setupPrivs

- If the current user is the object owner, then default the privilege bitmap

to object Owner values - no need to call PrivMgr to get privileges

Caching optimization:

We are now caching privmgr metadata tables in compiler cache when the compiler

context is instantiated. This avoids a metadata lookup for these tables.

- Added new methods that return if the table is part of the PrivMgr schema

- Adjusted CmpSeabaseDDL::createMDdescs to include privmgr metadata in the

cached entries

- Adjusted CmpSeabaseDDL::getMDtableInfo to check for privmgr metadata tables

from the cached entries

- Removed obsolete code CmpSeabaseDDL::alterSeabaseDropColumn

- changed CmpSeabaseDDL::getSeabaseTableDesc to check for both system and

privmgr metadata from compiler cache

- added new method CmpSeabaseDDL::getPKeyInfoForTable that returns the

primary key name and UID for a table. This is needed when dropping privmgr

metadata tables

Removed extraneous recompilations of HISTOGRAM structures:

Today, update statistics and showstats are reloading NATable entries

for HISTOGRAM tables on every access. This is because the parserflag

ALLOW_SPECIALTABLETYPE is turned on. When this flag is turned, the compiler

always reloads the cache entries - see code from CmpMain::sqlcomp:

//if using special tables e.g. using index as base table

//select * from table (index_table T018ibc);

//then refresh metadata cache

if(Get_SqlParser_Flags(ALLOW_SPECIALTABLETYPE) &&

CmpCommon::context()->schemaDB_->getNATableDB()->cachingMetaData())

CmpCommon::context()->schemaDB_->getNATableDB()->refreshCacheInThisStatement();

Changed code to not set ALLOW_SPECIALTABLETYPE and ALLOW_PHONYCHARACTERS

parserflags by default. Individual statements are setting these flags as needed.

1414125: User without priv can view data in metadata tables

The problem is that a user with priv cannot view data in metadata tables.

Even when a user had SELECT privilege on a system or privmgr metadata table,

the request failed.

The problem is that parameter 2 sent to CmpDescribeIsAuthorized in

hs_globals.cpp is NULL so SELECT priv is not checked. If the user has SHOW

component privilege, it works. A call was added to getPrivileges for metadata

tables before calling CmpDescribeIsAuthorized.

1393529: Core dump accessing MD table descriptors

When "UPDATE STATISTICS LOG [ON, OFF, CLEAR]" is specified by a non DB__ROOT

user, a core dump occurred. This happens because the isAuthorized check is

performed expecting a NATable structure. This command does not need any

special security checks.

Updated traf_authentication_setup script to support a new installation option

Change-Id: If7dbf3ec66e5beb7d88bda61ef32611401dd97b9

  1. … 10 more files in changeset.
Merge "Temporary fix for LP bug 1438372"

Temporary fix for LP bug 1438372

Hold and set CQD optimization_level to 3 during DDL query compilation

when hash joins and merge joins are turned off. This is to prevent the

optimization level set by user to other than 3 (maximum) to affect

meta-data queries. In the future, any secondary compiler won't get any

user CQDs, thus this is temporary fix as work-around.

Change-Id: Idc36d3e98f7729937e6c8a16ab95a41f68c8823d

LP Bug 1442949

-- alter add col was not running under an internal Xn.

that caused rollback to not work.

-- change to allow internal cleanup command to proceed if

running under a transaction. This is needed for internal

cleanups after error.

-- better error msg if index population fails.

Change-Id: I0fd85be97a2ab96544cd037f920df0a3f5868542

  1. … 4 more files in changeset.
additional changes to support ALIGNED row format.

This feature is not externalized yet.

Change-Id: Idbf19022916d437bb7bb69019194de5057cbcb65

  1. … 21 more files in changeset.
Merge "Preventing user CQDs from affecting metadata queries"

  1. … 2 more files in changeset.
DROP SCHEMA CASCADE and other fixes

Corrects a problem (Launchpad bug 1435967 and 1437474) with DROP

SCHEMA CASCADE when a view is defined that references a UDF in

the schema.

Corrects a problem where DB__ROOT could not grant privileges on

objects owned by DB__ROOT except for tables and views.

Corrects a problem where histogram tables in shared schemas were

owned by the first user to run update statistics; the tables

are now owned by the schema owner regardless of which user runs

the UPDATE STATISTICS command first.

Change-Id: I3289d456ac0d1fdd3d7923892395ddf9229e47bf

  1. … 3 more files in changeset.
Preventing user CQDs from affecting metadata queries

We now won't pass user CQDs to new CmpContext instance (CI) during

embedded CI switch. This is to prevent metadata query plans

being influenced by user CQDs unintentionally. Also, temporarily added a

flag to retrieve the latest hbase snapshot info for user tables

disrespect to whether we use the snapshots for scan or not. The CQD

TRAF_TABLE_SNAPSHOT_SCAN will only affect the user query plan

generations.

Reworked LP 1420826, Turning off nested_joins fails meta-data query

compilation, by reverting the changes because when compiling the

metadata queries, the default optimization level would be '3' and won't

be affected by user setting anymore.

LP 1426479: Row mismatch between index/table cause init auth to fail.

Fixed LP 1411355: User-level CQDs also seem to apply to metadata

queries.

Change-Id: Idd0388b626ecd3dbf3c3d1e75c1d6dc30b0ce021

  1. … 4 more files in changeset.
Miscellaneous DDL and security bug fixes

Fixed a testware issue with fullstack2/TEST062 that occurred during

release testing

Bug 1415196 - Alter volatile table add column cores at CmpSeabaseDDL::alterSeabaseTableAddColumn()

Added a check to not allow add or drop column for volatile tables:

- sqlcomp/CmpSeabaseDDLtable.cpp

Bug 1415232 - A failed create view causes a volatile table to disappear

The code to bind a view does not correctly reset the volatile schema in use

session parameter in case of an error. Subsequent calls do not check for

volatile objects.

parser/BindStmtDDL.cpp

Bug 1371265 - should not allow grants to DB__ROOT or current user

Added a check at grant to prevent this

- sqlcomp/PrivMgrPrivileges.cpp

Bug 1392491 - Unavailability of privmgr metadata error is incomplete

If not all the privmgr metadata is available, then a new Compile context

flag called IS_AUTHORIZATION_READY is set. This flag is adjusted when

a new compiler context is started, and when authorization is enabled and

disabled.

When isAuthorizationEnabled is called and authorization is incomplete,

error 1234 is now returned by default.

After coding changes were added, a request to not check all privmgr metadata

table at context startup was requseted - a performance concern. Fix was

changed to check all tables for debug builds but check only one table for

release builds. If the performance problem is fixed, then we can go back

and check for all privmgr tables.

- arkcmp/CmpContext.h

- arkcmp/CmpContext.cpp

- sqlcomp/CmpSeabaseDDLcommon.cpp

- sqlcomp/nadefaults.cpp

Bug 1402009 - DB__ROOT is unable to grant privilege on object in private schema

When DB__ROOT executes a grant or revoke on objects it does not own, need to

change the grantor from DB__ROOT to the object owner. This matches the same

behavior for other DDL operations such as CREATE.

As part of this fix, the GRANTED BY clause is now allowed for GRANT

statements but it won't be complete until LP bug 1414225 is done.

- sqlcomp/CmpSeabaseDDLtable.cpp (seabaseGrantRevoke)

- sqlcomp/PrivMgrCommands.h

- sqlcomp/PrivMgrCommands.cpp

- sqlcomp/PrivMgrPrivileges.h

- sqlcomp/PrivMgrPrivileges.cpp

- sqlcomp/PrivMgrMD.h

Bug 1414125 - User without priv can view data in metadata tables

Fixes were in place for all metadata tables except the privmgr metadata

tables. The priv information was always being set to none in setupPrivInfo

(NATable) and revoking a privilege was not correctly removing privilege

information from object_privileges.

- optimizer/NATable.cpp

- sqlcomp/PrivMgrCommands.cpp

Bug - create library checking privileges when authorization is not enabled

- CmpSeabaseDDLroutine.cpp

Enhanced the sqlci env command:

- alphabetize the output

- add the following information

-- authentication status

-- authorization status

-- external (LDAP) user connected

A new session parameter called SESSION_EXTERNAL_USER_NAME was added to return

the external user name connected.

A new cli request called SQL_EXEC_GetAuthState_Internal was written to return

the authentication & authorization status. Code was also added, but not yet

supported, for auditing status.

Renamed member/methods that use ldap to external

Changed sqlci env command to return new format

- cli/sqlcli.h

- cli/SQLCLIdev.h

- cli/CliExtern.cpp

- cli/Cli.h

- cli/Cli.cpp

- cli/Context.h

- cli/Context.cpp

- qmscommon/QRQueries.cpp

- sqlci/SqlciEnv.h

- sqlci/SqlciEnv.cpp

- regress/fullstack2/EXPECTED062.SB

- regress/funnstack2/DIFF062.KNOWN.SB.OS

Change-Id: I04627435a0e644c6b14bbf6bd8aa1162d81224fb

  1. … 27 more files in changeset.
Support cancel for DDL, update stats, and utils

This change propagates parent query ID to queries created

by DDL, update stats and utils. In this way, preexisting

code sets up child query IDs and the preexisting cancel

broker code in RMS cancels the entire query tree below

the indicated parent. Also added is logic to check for a

canceled parent at the time a child query ID is setup in

Statement::execute, to make the scheme more robust and

to support a common scenario where the parent is compiling

a child query at the time of cancel.

This change also makes ex_root_tcb::cancel to wait for

cancel broker message completion. This fixes a problem where

a query (e.g., CREATE INDEX or DROP INDEX) is prepared,

executed once and gets an error, then re-execed. In this

case, sqlci was not closing the statement after the error.

To improve robustness, after an error we will now wait for

message completion, if needed.

Since we do not yet have transaction protection for DDL,

it will usually be required to cleanup canceled DDL

operations by using the CLEANUP command.

Change-Id: I8940f7108906d5d8d1a8aa4574aacf2b9ffcf0b6

  1. … 27 more files in changeset.
review updates and more. See below.

--Code updates from previous review.

--Bug fixes and enablement of ALIGNED FORMAT row storage.

This is for testing only.

--Error handling for explain statement.

Change-Id: Ia01b3813e0b72ce66e2c03cc50777d4f1f8d5a78

  1. … 19 more files in changeset.
repository upgrade and explain enhancements. Details below.

-- repository upgrade infrastructure integrated

with 'initialize trafodion, upgrade'

-- fields have been added/removed/modified in repos tables

-- new repos table has been added

-- upgrade enhanced to only upgrade subset of components that

need to be upgraded. One or more of metadata, repos, views, priv.

-- metadata version now tracks release version and indicates

the traf release where upgrade was needed.

Current metadata version will be 1.1.0.

-- packed explain data is now returned to caller so it could be

stored in repository

-- explain info from repository for a query id could be retrieved

and displayed

explain qid <query-id>;

explain options 'f' qid <query-id>

select * from table(explain(null, 'EXPLAIN_QID=<query-id>'));

-- a sql query could be explained and returned in relational format.

select * from table(explain(null, 'EXPLAIN_STMT=<query-str>));

Users only need to have select permission on tables referenced

in query-str.

-- explain and statistics tables could be invoked

invoke table(explain(null, null));

invoke table(statistics(null, null));

Squashed commit of the following:

commit 09004ff7260b771378bc1a29f0ecb82e5e0c6100

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 16 12:24:03 2015 -0700

repos and explain, #12

Change-Id: Ia08151b5c7087b6b7daa5b662df2a584e5a7b2a1

commit 46f05ada264094626e45445d3875ccf876cad802

Merge: 3393587 6b3a4c4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 21:27:48 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 3393587de319a3ad43d7d360cd74501e8b8103e4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 21:26:30 2015 -0700

repos and explain, #11

Change-Id: I895a4b4766e0a92b7472a6b347f0bb3ee07fa9b6

commit cf6ffee549b37ffbcd6ce750a1fdbf7f0c410d61

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 16:07:22 2015 -0700

repos and explain, #10

Change-Id: I19e182ec6600525cec986f173afaf68a44e04af2

commit e331eec68feea59d9fa11e0154e3093762120eb2

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 18:19:03 2015 -0700

repos and explain, #9

Change-Id: I783c75513a432f718d2cd7c6030d410c419a79c8

commit 9e9da221eb45b760963741657fe9cb910a753c84

Merge: 20b64e5 9eb7f37

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 14:52:09 2015 -0700

Merge remote branch 'gerrit/master' into fix1

Conflicts:

sql/common/ComSmallDefs.h

sql/executor/ExExeUtilCli.h

sql/generator/GenRelExeUtil.cpp

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLupgrade.cpp

sql/sqlcomp/CmpSeabaseDDLupgrade.h

Change-Id: Id21b10ff35ad506f409a83a8d955c2e643a2bcf5

commit 20b64e5552d8f614586ed843aad76d87d3473e2e

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 13:44:15 2015 -0700

repos and explain, #8

Change-Id: I2b49d962f08a22ca6c8b437a583855e2344f513c

commit 500b7a9486f23fff6b1c17bb6e79c2f614a7e3e6

Merge: f2c050e fc1c31d

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 08:07:34 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit f2c050ea048b409b7d6769172e16c4ab26267bd0

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 08:07:15 2015 -0700

repos and explain, #7

Change-Id: I0bc04b6e8f147af1defbfd023c3700e26cc5b6f1

commit fabdb7dadd34238a8a227e880aa868ce484dc580

Merge: 59bb689 74640d7

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Mar 12 09:01:20 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 59bb6893437abe3999a53188503c60cf47633458

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Mar 12 09:00:44 2015 -0700

repos and explain, #7

Change-Id: I6595a04ad59c1f705a04beee18c877cb81db0fac

commit a86353b45436126915fd2ba74c9af53f8bfd4d19

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 9 16:21:16 2015 -0700

repos and explain, #6

Change-Id: Ie4fe2b2bece7de3697eb8ada0d1cb1067b89bb88

commit 61272e0d6107536ab8b691415770e856267b6da5

Merge: 1708cb4 ca5dfb4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 9 15:52:43 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 1708cb44511809bf8e63800d226ac8ea34916726

Merge: 1284f4f 31f25ec

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 6 14:55:13 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 1284f4f78f1f43657f29899cc8b9a96c02f37815

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 6 14:54:54 2015 -0800

commit: repos and explain, #6

Change-Id: I38d796fd4ab58bf29fa62c4bd5492ce44de9c8a0

commit f639911a1ea4247ae4c297bbbd904b84c7587b3a

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 2 06:31:21 2015 -0800

repos and explain, #5

Change-Id: Ie97ba3cac8dc78784d0a7eef57176e51b9e16224

commit da69fe9595d4ad5c9adf3b32ae8f848a0e6be964

Merge: 9b95b8c a5caeef

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 2 06:31:00 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 9b95b8c3240a4186704b8b7f2ff8cefebf211732

Merge: ea2f687 8cd8a92

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Feb 26 08:50:25 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit ea2f6871b7c7803d0f1e3fa664b474e069a0befd

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Feb 26 08:50:03 2015 -0800

commit: repos and explain, #5

Change-Id: I44d8f87b3652e6300e41302d3a3e88b9b18652a3

commit 522a47eb991df2245005c693e4d034235adbfdc5

Merge: d986988 f42694e

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 25 13:13:02 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit d98698856a4be479a7603b0fb79210e80e88e1c1

Merge: 5841e47 2aad3ef

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 18 14:00:24 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 5841e47f1e3f9a018424298a92cd134f4430e6bf

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 18 13:57:46 2015 -0800

repos and explain, #4

Change-Id: I01f53003069598381743c9be80b405ed37cefe6d

commit e7b5966d8ce653a57b8128c0b24c15fc05737668

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 11 11:28:20 2015 -0800

repos and explain, #3

Change-Id: Iff148ef175a7641ea37e27e60435ccf8f613e6db

commit aba98dc8cfda62d4e708cd892ed626f2a3051eb0

Merge: 66f1926 19a0c32

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 10 14:09:58 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 66f1926d279f2b408d436b0aa53a3a6a5388901e

Merge: 9d0ef2a b68f59b

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Feb 9 10:52:35 2015 -0800

Merge remote branch 'gerrit/master' into fix1

Conflicts:

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLtable.cpp

Change-Id: I6348fec3aece2baad9e9570749f43c086416219e

commit 9d0ef2aa01a62ea3e4e0ab35f8ec9687ef3336ef

Merge: aac268d 2cd8fbb

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Feb 6 14:15:49 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit aac268d097afbe506dcf76de4d8f49652a1d4df7

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 16:31:17 2015 -0800

repos and explain, #2

Change-Id: Ia2a50159e577cd6ff8110dca35236fa90ea0d260

commit 2e6160411c8facb448e78d387fa3227e7b759c5d

Merge: c6e2fe5 566706d

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 15:21:03 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit c6e2fe529f9d1e59d3eb26cb145b260d81814443

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 15:16:36 2015 -0800

repository updated and explain changes, #1

Change-Id: I5daf66a2064b46ad65f1200579385965f7f56808

Change-Id: I0211d754524b8c06a05bb7ade20f884ec91340d2

  1. … 58 more files in changeset.
Trafodion Metadata Cleanup command support.

Various changes to support cleanup command has been added.

A separate external spec contains the details.

Summary of syntax:

cleanup [ table t | index i | sequence s | object o] [, uid <value>]

cleanup [private | shared] schema sch

cleanup uid <value>

cleanup metadata, check, return details

In addition, a new command to get names of various hbase objects

has also been added:

get [ all | user | system | external ] hbase objects;

Change-Id: I93f1f45e7fd78091bacd7c9f166420edd7c1abee

  1. … 79 more files in changeset.
Merge "Snapshot Scan changes"

  1. … 5 more files in changeset.
Snapshot Scan changes

The changes in this delivery include:

-decoupling the snapshot scan from the bulk unload feature. Setup of the

temporary space and folders before running the query and cleanup afterwards

used to be done by the bulk unload operator because snapshot scan was specific

to bulk unload. In order the make snapshot scan indepenednt from bulk unload

and use it in any query the setup and cleanup tasks are now done by the query

itself at run time (the scan and root operators).

-caching of the snapshot information in NATable to optimize compilation time

Rework for chaching: when the user sets TRAF_TABLE_SNAPSHOT_SCAN to LATEST

we flush the metadata and then we set the caching back to on so that metadata

get cached again. If newer snapshots are created after setting the cqd they

won't be seen if they are already cached unless the user issue a command/cqd

to invalidate or flush the cache. One way for doing that can be to issue

"cqd TRAF_TABLE_SNAPSHOT_SCAN 'latest';" again

-code cleanup

below is a description of the CQds used with snapshot scan:

TRAF_TABLE_SNAPSHOT_SCAN

this CQD can be set to :

NONE--> (default)Snapshot scan is disabled and regular scan is used ,

SUFFIX --> Snapshot scan is enabled for the bulk unload (bulk unload

behavior is not changed)

LATEST --> Snapshot Scan is enabled independently from bulk unload and

the latest snapshot is used if it exists. If no snapshot exists

the regular scan is used. For this phase of the project the user

needs to create the snapshots using hbase shell or other tools.

And in the next phase of the project new comands to create,

delete and manage snapshots will be add.

TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX

This CQD is used with bulk unload and its value is used to build the

snapshot name as the table name followed by the suffix string

TRAF_TABLE_SNAPSHOT_SCAN_TABLE_SIZE_THRESHOLD

When the estimated table size is below the threshold (in MBs) defined by

this CQD the regular scan is used instead of snapshot scan. This CQD

does not apply to bulk unload which maintains the old behavior

TRAF_TABLE_SNAPSHOT_SCAN_TIMEOUT

The timeout beyond which we give up trying to create the snapshot scanner

TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION

Location for temporary links and files produced by snapshot scan

Change-Id: Ifede88bdf36049bac8452a7522b413fac2205251

  1. … 44 more files in changeset.
Adding more run-time memory allocations from NAHeap

This set of changes moves some of the string vector variables in HBase

access operators from standard string template to our NAList and

NAString (or HbaseStr for row IDs). In the process, allocationis of the

objects will be from our HAHeap instead of the system heap. This would

help us tracking memory usage and detecting leaks easier.

In addition, a change in ExHbaseAccessTcb::setupListOfColNames()

prevents unnecessary allocations to populate the columns list unless it

is empty. The Google profiling tools helped us on identifying this

problem.

also, removed ExHbaseAccessDeleteTcb operator which was not used.

Change-Id: I87ab674ab8e3d291f2fc9563718d88de537ae96b

  1. … 10 more files in changeset.