BindItemExpr.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
OSIM (Optimizer Simulator) redesign 1.

Simulate query plan generation of production cluster on dev workstation,

by collecting information from production cluster, and restore it on dev worksation.

--runnig on production clusters, collect table DDLs, statistics, CQD, to osim-directory,

--the directory path can either full(absolute) or relative.

osim capture location '<osim-directory>'[, force];

--runing queries on cluster

osim capture stop;

--restore DDLs, CQDs, statistics and cluster information.

osim load from '<osim-directory>';

--setup runtime information, like cpu number, node number.

osim simulate start|continue '<osim-directory>';

Change-Id: I30882e87a6ea0f08c9aa64685705eebebcbb3bf0

  1. … 38 more files in changeset.
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

  1. … 39 more files in changeset.
Phase 2 for log reader TMUDF

blueprint cmp-tmudf-compile-time-interface

Log reader TMUDF is mostly working now.

Still need to set cqd NUM_PARALLEL_ESPS '<num of nodes>' on clusters.

Still needs more work and more testing.

Still seeing some issues with non-ASCII characters.

// SQL Syntax to invoke this function:

//

// select * from udf(event_log_reader( [options] ));

//

// The optional [options] argument is a character constant. The

// following options are supported:

// f: add file name output columns (see below)

// t: turn on tracing

// d: loop in the runtime code, to be able to attach a debugger

// (debug build only)

// p: force parallel execution on workstation environment with

// virtual nodes (debug build only)

More detailed explanation of changes:

- PredefUdrReadfile.cpp: Work on event log reader TMUDF

- sqludr.*: New method to add formal parameters, allows TMUDF to

accept optional parameters.

- OptPhysRelExpr.cpp:

Made some changes for TMUDFs with arity 0 to avoid asserts

and to be able to call okToAttemptESPParallelism in method

RelExpr::synthPhysicalProperty(). This is needed for leaf

operators (arity 0) that want to initiate parallel execution

and TMUDFs seem to be first in that situation.

Changed TableMappingUDF::synthPhysicalProperty to generate

a partitioning function with multiple partitions (and no

partitioning key, so far) if required.

- ExUdr.cpp,

GenUdr.cpp,

sqlparser.y,

DefaultConstants.h,

nadefaults.cpp,

BindItemExpr.cpp,

BindRelExpr.cpp,

ValueDesc.ppp:

Addressed review comments from last phase, got rid of ALLOW_UDF CQD

- Rel*.h

Rel*.cpp

OptPhysRelExpr.cpp (has other changes as well)

Simple but messy change to add one more parameter to

RelExpr::synthPhysicalProperty

Change-Id: I5549e47c0f019beefd4ec1695ae7abf8c3bd43e3

  1. … 39 more files in changeset.
Hybrid Query Cache feature implemented.

The Hybrid Query Cache (HQC) is an enhancement of existing Query Cache,

which is trying to find match queries in existing query cache at an earlier point,

i.e. just after parser and before binder so as to avoid binder overhead if there's a hit.

Two virutal table ISPs are added to show stats of Hybrid Query Cache.

Add control of ISP to run locally or remotely.

Changes after reviewers' comments.

Fixup errors in SqlciErrors.txt that cause core/TEST014 failure.

Fix minor issues about (hybrid)query cache ISP.

Add HQC virtual table ISP tests to compGeneral/TEST042.

Change-Id: Ib5be56e04990639153747255834b30fc9c3f3829

  1. … 40 more files in changeset.
Fixes bug 1398896.

To get the debug output in ColReference::bindNode, you must now

export the environment variable COLREFERENCE_DEBUG first.

Change-Id: Ifa7b3ab46fa97dff89c05b3eacd15f0313ae00b3

Merge "removal of obsolete code that is no longer valid and unused in Trafodion"

  1. … 9 more files in changeset.
removal of obsolete code that is no longer valid and unused in Trafodion

Change-Id: Ie9212627d102661fe1a7b02fd6e9ae9a23198f2a

  1. … 35 more files in changeset.
LOB datatype infrastructure support

Technology preview.

More changes expected as part of this work before it

is user ready.

blueprint lob-support

This checkin contains basic support for create blob/clob datatypes.

The feature is disabled by default. Instructions on how to enable are

listed below.

New test executor/TEST130 that turns the feature on and tests out the

functionality.

New mxlobsrvr process will be started as part of sqstart.

Create and drop of tables with LOB columns.

No support for alter.

DML support for LOB datatypes.

Insert, update and deletes. Joins of 2 tables with LOB columns are

allowed but joins on LOB columns temselves are not allowed.

Insert-select from one LOB table another not yet supported.

Link to document from the blueprint will be added shortly.

To enable and try LOBs:

On a developer workstation :

cqd TRAF_BLOB_AS_VARCHAR 'OFF';

On a cluster after installing the code 2 steps are needed:

1. cqd TRAF_BLOB_AS_VARCHAR 'OFF';

2. sudo su hdfs --command "hadoop fs -mkdir /lobs"

sudo su hdfs --command "hadoop fs -chown -R trafodion:trafodion

/lobs"

This checkin includes several merges from the mainline and each of

the lines below represents one commit to the project branch where this

work was done. `

-Turning off LOB code by default. But turning it on in

executor/TEST130 just to ensure testing the code path.

-Support for showddl and some syntax for external files and stream.

-LOB regression test

-Workaround for dtm issue LP 378167

-Changes to make append work. Changes to use lob heap.

-Fix for using system heap for all LOB allocations and handling NULLs.

-Added workaround for cursor delete issue. LP 1376969

-Fixes for update.

-Parser changes for exe_util_lob_extract

-Pull in lob extract code

-Adding mxlobsrvr directory

-Fixed the LOB interface to use 2 new params for cursor fetches. They do

not overload the LOB Handle and LOB handle length anymore.

Added a flag to lobGlobals to indetify it's a hive access.

Cleaned up parser code.

-LOB support for create,drop,insert,delete,select.

Change-Id: I7c8125696e847b71580b746388632e75741bd347

  1. … 52 more files in changeset.
Native external hbase table access (select, IUD) changes.

-- IUD on external hbase tables is now enabled by default

-- predicates on native hbase tables can now be pushed down to

hbase region server

-- traf varchar col maxlength is now 200K by default,

can be changed by cqd max_character_col_size

-- executor handles column values length greater than 32K during

move to/from JNI

-- error is correctly returned if data retrieved from hbase exceeds expected

max row length

-- hbase column_create function now takes an expression/param as its

column name operand

Change-Id: Ieb3fcabfebaa22008eff2a049fc1e2000e68861e

  1. … 46 more files in changeset.
Identity column and sequence numbers support.

Added support for IDENTITY columns.

Finished sequence numbers functionality.

Bug fixes and perf enhancements in those areas.

This code has been pre-reviewed by Joanie C.

Change-Id: I0445bc9765b60becb9adf8c053c05344395aecaa

  1. … 94 more files in changeset.
Various changes done to upgrade metadata from v23 to v30.

-- initialize trafodion, upgrade; will upgrade metadata from v23 to v30.

-- Following columns were added/dropped/modified in metadata tables:

-- COLUMNS: added sql_data_type

-- OBJECTS: added DROPPABLE column to indicated droppable table.

added schema_owner column.

-- TABLES: added num_salt_partns, table_format, comment, key_length,

row_data_length, row_total_length.

Removed hbase_create_options and moved it to TEXT.

-- SEQ_GEN: redef_ts and upd_ts added.

-- TEXT: text_type, sub_id added.

-- TABLE_CONSTRAINTS: added IS_ENFORCED, DROPPABLE, DISABLED,

VALIDATED, LAST_VALIDATED_TIME

-- reserved flags column added to metadata tables, initialize to zero.

-- default_value in COLUMNS is now UTF8 instead of UCS2

-- new columns populated during upgrade for existing objects.

-- values that moved from one metadata table to another populated during upgrade.

-- new columns populated during create operation for new objects.

-- step added to drop/create metadata views as part of upgrade

Change-Id: I173c33cbe7ee9314c66b84d16a6ef8b2d7394335

  1. … 20 more files in changeset.
Set authorization enabled/Sequence generator privs

Code to set authorization enabled at startup

Contains changes to check authorization at process startup time and

code review comments from previous deliveries

Description of changes to check authorization at process startup time:

At process && compiler context startup time a check has been added to

see if authorization is enabled. Based on this check a new flag is set

in the compiler context.

Any operation wishing to see what the authorization status is, just need

to look at this flag.

This code has been reviewed internally by the security team.

There will be a subsequent set of changes in the PrivMgr code to return

better errors.

Details:

Arkcmp/CmpContext.h

Added a new flag containing authorization status and methods that get

and set this flag.

Sqlcomp/nadefaults.cpp

In method: NADefaults::readFromSQLTables added code that checks to see

if authorization is enabled and sets the flag in CmpContext.

It calls CmpSeabaseDDL::isPrivMgrMetadataInitialized to determine

privmgr metadata status

Sqlcomp:CmpSeabaseDDL.h/CmpSeabaseDDLcommon.cpp

Implementation of method isPrivMgrMetadataInitialized

Changed isAuthorizationEnabled to look at the CmpContext flag instead of

the flag (which was removed) in the CmpSeabaseDDL class

Changed initSeabaseAuthorization and dropSeabaseAuthorization to change

the flag in the context and kill compiler processes

Changed all calls to PrivMgrnnnn::isAuthorizationEnabled to use the

CmpSeabaseDDL::isAuthorizationEnabled or directly from CmpContext

Bin/SqlciErrors.txt & sqlcomp/CmpDDLCatErrorCodes.h to create a new

error 1234 (currently unused)

Sqlcomp/PrivMgrMD.cpp changed mapping of PrivMDStatus to match what was

done in nadefaults.cpp

Optimizer/BindRelExpr.cpp && sqlcomp/nadefaults.cpp to look in

CmpContext for authorization enabled flag

Check privileges for Sequence generator

Adds the code in compiler to check for usage privilege

for any sequence generators used in a query.

Additional privilege checks, plus

This delivery includes:

Verifying that user had correct privileges to perform all DDL

operations. This is performed through a call to

isDDDLOperationAuthorized. The signature changed to pass the object

owner instead of the object name. This eliminates an I/O and made the

method simpler. All callers were changed to use the new signature and

all DDL operations now call this method after the NATable structure has

been retrieved. A new regression test was added (TEST138).

As part of DDL privilege checking, the ALTER and DELETE component

privilege is no longer granted during initialize authorization.

Updated files to address code review checkin for change ID:

If7538eee38178c2345fe418172c6196b25a20b33.

Fixed a problem where SHOWDDL was not returning an error when user does

not have appropriate privilege.

Made the PRIVMGR_MD schema a reserved schema.

Added code to switch contexts for several PrivMgr operations. This

required a change to not grant owner privileges when creating the

OBJECT_PRIVILEGES table.

Added a KNOWN diff file for TEST133. There is an issue where rows are

not being loaded into OBJECT_PRIVILEGES during an error test.

Change-Id: I7448e7171e5f1f09feb6d1f688470b72dc1f43d4

  1. … 26 more files in changeset.
Changes to support OSS poc.

This checkin contains multiple changes that were added to support OSS poc.

These changes are enabled through a special cqd mode_special_4 and not

yet externalized for general use.

A separate spec contains details of these changes.

These changes have been contributed and pre-reviewed by Suresh, Jim C,

Ravisha, Mike H, Selva and Khaled.

All dev regressions have been run and passed.

Change-Id: I2281c1b4ce7e7e6a251bbea3bf6dc391168f3ca3

  1. … 143 more files in changeset.
added support for externalized Sequence numbers.

-- sql statements: create/drop/alter/get/showddl sequence for sequence objects

-- function 'seqnum' to retrieve sequence numbers.

An external spec has been created.

Launchpad #1349985

Code reviewd by Joanie C, Suresh S, Selva and Sandhya.

Full dev regressions run and passed.

Change-Id: Ie11dbab4d24ff6a1106697f7e2253ea895e6c873

  1. … 71 more files in changeset.
Various Launchpad and other fixes.

-- metadata and statistics tables will no longer be created with

serialization attribute even if cqd hbase_serialization is set to ON.

also, reenabled HBASE_SERIALIZATION for regressions run

(sqlcomp/CmpSeabaseDDLcommon.cpp, regress/tools/sbdefs)

-- rowwise hbase rows from native hbase tables are now being created correctly

in all cases.

executor/ExHbaseAccess.*

exp/exp_function.*, optimizer/BindItemExpr.cpp, ItemExpr.cpp

-- IUD and SELECT execution state is now being correctly initialized at the

beginning of a run. Multiple executions were failing otherwise.

(executor/ExHbaseIUD.cpp, ExHbaseSelect.cpp)

-- sign is now allowed in an interval literal

(generator/GenItemFunc.cpp, GenRelScan.cpp, ItemFunc.h, ValueDesc.cpp)

-- location value being returned from updates was not being set correctly in

some cases. That has been fixed

(generator/GenRelUpdate.cpp)

-- self referencing updates were not returning the right values due to

halloween issue. It is fixed by transforming it to insert/delete.

(optimizer/BindRelExpr.cpp)

-- purgedata now returns an error if issued on hbase, hive, neoview tables,

or on a view.

(optimizer/RelExeUtil.cpp, sqlcomp/CmpSeabaseDDLcommon.cpp)

-- referencing and referenced columns in a foreign key are now enforced

to have the same datatype attributes

(sqlcomp/CmpSeabaseDDLtable.cpp)

-- drop schema now works with delimited schema names

-- in some cases, a create constraint failure was not dropping the table on

which the constraint was being created.

that has been fixed.

(sqlcomp/CmpSeabaseDDLtable.cpp)

-- some additional infra changes for Traf as a mysql storage engine.

(cli/*, executor/ExExeUtilCli.*)

Change-Id: I94d5eb13c826efdf44ba10c04ac52a671f86553e

  1. … 23 more files in changeset.
Code cleanup (06042014)

Change-Id: Ib7f2f39e42085e026ac1000e43048c9a185b6976

  1. … 14 more files in changeset.
Initial code drop of Trafodion

    • -0
    • +12398
    ./BindItemExpr.cpp
  1. … 4886 more files in changeset.