Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
remove WITH DIVISION clause

Change-Id: I986fe56a8559f88f2781adf700933d5b03c9e697

Changes to sqenvcom.sh Java Warning

Java warning will now only be seen if the Java version is less than

1.7.0_67.

Change-Id: Ibbf5a5f82428ffccce0bade9ac754b41060284e1

Authorization checks for DDL & utilities

Fixed issues from code comments.

LOAD/UNLOAD authorization checks:

Code was added during code generation to make sure user has privileges,

if the user had necessary privileges, then the EXEUTIL parser flag is

turned on to avoid further privilege checks. When load/unload

completes, the parser flag is reset.

Update/showstats Statistics authorization checks:

Added a new error message

Changed hs_globals to support a new isAuthorized method and store

parser flags when class is instantiated and reset them when done

Changed hs_cli.cpp to use new IF NOT EXISTS syntax when creating

histogram tables, make owner of histogram tables DB__ROOT

(will need to adjust when schema privileges happen), and clean up

CreateHistTables method to remove old authorization mechanism

Changed hs_update.cpp which controls the update and showstats operation

to add authorization checks

Purgedata and populate index changes:

Changed CmpSeabaseDDLcommon.cpp to check privileges for purgedata

Changed CmpSeabaseDDLindex.cpp to check privileges for popindex

Additional component privileges and checks:

Added support for new component privileges in PrivMgrMD.h/.cpp

Added support for MANAGE_COMPONENTS

Added support for CREATE_INDEX and DROP_INDEX component privs

Fixes from last delivery that were postponed:

Context.cpp - fix for previous code review

CmpSeabaseDDLtable - added calls to deallocEHI

PrivMgrMD - fixed wording in a comment

Miscellaneous changes:

ComUser - added new convenience method - isRootUserID()

NATable.cpp (setupPrivInfo) to always set up privInfo_ and to call

the embedded compiler while extracting privileges

Privilege adjustments to take advantage of privInfo stored in NATable:

Added code to mark and rewind errors in diags.

Fix for LP bug 1392895

Change-Id: I6f7245ae7e66086769c0e92d901399c99e8f2af3

    • -0
    • +224
    /sql/generator/GenRelExeUtil.cpp
    • -0
    • +174
    /sql/regress/catman1/EXPECTED137
    • -29
    • +128
    /sql/regress/catman1/EXPECTED138
    • -2
    • +39
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
  1. … 19 more files in changeset.
Fix performance regression due to QI for DDL

This check-in omits object UIDs from query plans for the tables

SB_HISTOGRAMS and SB_HISTOGRAMS_INTERVALS. Previously, when the

code generator tried to add the object UIDs for these, it had to

make a special query to the metadata, since the corresponding

internal cached structure omitted object UIDs when they were

created via methods like Generator::createVirtualTableDesc. The

special query to lookup these object UIDs was shown to be

responsible for a large pathlengh regression.

Change-Id: Id5046c5c55a4fc8dd2ba3f891449ea87d35a5534

Closes-Bug: #1398600

Computed column key predicates for MDAM

Moving the generation of computed column predicates out of the

SearchKey logic and making it available as a static method on

class ScanKey. This allows us to compute these predicates before

we create the Disjuncts data structure that is used in a file

scan, where it will go into a SearchKey or an MdamKey.

Also fixing a bug that stopped after the first predicate found

on a computed column, so it failed to produce both a begin and

and end key value when selecting a range of values

(removed a "break" in ScanKey::createComputedColumnPredicates)

Change set 2: Addressed reviewer comments. Moved computation of

computed preds to Scan::addIndexInfo and ValueIdSet

that stores these preds from FileScan to Scan.

Change-Id: I4297d789ded8522eb67d5441ac281657ff90e774

    • -3
    • +1
    /sql/optimizer/SimpleScanOptimizer.cpp
Fixed typo

Change-Id: I0a748ea4403812886908e68889ec472631c71743

Reducing the logging level of several statements

Change-Id: I4030124763e2f3f5e983e67da9574d405ed6141d

Merge "Removed commitRequest INFO tracing statement"

Drop view QI fix & fix for LP 1384485

Removed sqlparser.y change - it was not intended to be delivered.

Drop view QI fix:

At drop view time, we need to send a QI message for the view being

dropped and remove base tables referenced by the view from cache.

Added a structure which describes a base table reference

Added code to create a list of base table references by calling

getListOfReferencedTables

getListOfReferencedTables gets the list for the first layer of

referenced objects (getListOfDirectlyReferencedObjects).

If the referenced object is a view, it recursively calls

getListOfReferencedTables

If the referenced object is a table, it gets added to the base

table list

Set up a QI key during drop view

Removed naTable entries for all tables referenced by the view

Updated TEST122 with drop view tests

Fix for LP 1384485:

Alter table allows adding a constraint to a _MD_ table

Reworded error message 1289

Added checks for reserved schema in missing places, mostly

constraints

Change-Id: I258983f2a28199ae522ba72b831f57fd22730a2f

    • -13
    • +517
    /sql/regress/executor/EXPECTED122
    • -10
    • +115
    /sql/regress/executor/TEST122
    • -2
    • +51
    /sql/sqlcomp/CmpSeabaseDDLtable.cpp
    • -0
    • +144
    /sql/sqlcomp/CmpSeabaseDDLview.cpp
Removed commitRequest INFO tracing statement

Change-Id: If495ff790ec58f02d95fab92656810c6520f645e

Added new HBase settings

Added:

ipc.server.tcpnodelay = true

ipc.client.tcpnodelay = true

Change-Id: I66595df2bb295062a5a5d826fa0f17fe26ab6a3b

Expected file change to address RPAD function

Change-Id: Iae75862f2e348eb620349724e096f94c7c8e6b64

Merge "Fix for fully qualified domain named host."

Merge "SQL syntax to cancel executing query, phase 1"

Fix for fully qualified domain named host.

Change-Id: Id2ce37be414a88f7270ccdcb0ed187e533e78dcf

Merge "Bug fixes"

    • -2
    • +10
    /sql/sqlcomp/CmpSeabaseDDLtable.cpp
Merge "Support for divisioning (multi-temperature data)"

SQL syntax to cancel executing query, phase 1

This check-in implements the command "control query cancel qid <qid>".

The command will send a cancel message to the MXSSMP cancel broker,

but since the master executor is not yet sending a query started

message,

no query will be canceled, and if the message is successful SQLCODE

-8026 will be raised. The change also validates the cancel command.

produces new EXPLAIN and SHOWPLAN output and handles errors, as shown

in a new regression test.

Change-Id: I472fca043aa12d684c13581264baf22a7ea5f49b

Implements: blueprint sql-query-cancel

    • -0
    • +628
    /sql/executor/ExCancel.cpp
    • -0
    • +169
    /sql/regress/executor/EXPECTED106
    • -0
    • +1420
    /sql/regress/executor/TEST106
    • -3
    • +1
    /sql/regress/tools/runregr_executor.ksh
Support for divisioning (multi-temperature data)

This is the initial support for divisioning. See

blueprint cmp-divisioning for more information:

https://blueprints.launchpad.net/trafodion/+spec/cmp-divisioning

Also, this change fixes the following LaunchPad bugs:

Bug 1388458 insert using primary key default value into a salted

table asserts in generator

Bug 1385543 salt clause on a table with large number of primary

key columns returns error

Bug 1392450 Internal error 2005 when querying a Hive table with

an unsupported data type

In addition, it changes the following behavior:

- The _SALT_ column now gets added as the last column in the

CREATE TABLE statement, rather than the first column after

SYSKEY. The position of _SALT_ in the clustering key does

not change. This will cause some differences in INVOKE and

in the column number assigned to columns.

- For CREATE TABLE LIKE, the defaults of the WITH clauses

are changing. CREATE TABLE LIKE now copies constraints,

SALT and DIVISION clauses by default. The WITH CONSTRAINTS

clause is now the default and should no longer be used.

Instead, WITHOUT CONSTRAINTS, WITHOUT SALT and WITHOUT

DIVISIONING clauses are supported.

- For CREATE INDEX ... SALT LIKE TABLE, we now give a

warning instead of an error if the table is not salted.

- Also added an optimization for BETWEEN predicates. If

part or all of them can be converted to an equals predicate,

we do this now. Example:

(a,b,c,d) between (1,2,3,4) and (1,2,5,6)

is transformed into

a=1 and b=2 and (c,d) between (3,4) and (5,6).

More detailed description of changes:

- arkcmp/CmoStoredProc.cpp

sqlcat/desc.h

+ other files

Using the new FLAGS column in the COLUMNS metadata table to store

whether a column is a salt or divisioning column. Note that since

there may be existing salted tables without this flag set, the flag

is so far only reliable for divisioning columns.

- comexe/ComTdb.h

comexe/*.h

generator/Generator.cpp

sqlcomp/CmpSeabaseDDLmd.h:

Changed the column class field in struct

ComTdbVirtTableColumnInfo from a string to the corresponding

enum. Sorry, this caused lots of small changes (deleting "_LIT"

from the initializers). Also added the column flags.

- executor/hiveHook.cpp: Added a check for partitioned tables

(having multiple SDs). This is part of the fix for

bug 1353632.

- GenRelUpdate.cpp: When generating the key encoding expression

for an insert inside a MERGE operation, we assumed the new

record expression was in the order of the key columns. Added

a step to sort by key column, so we can pass the expression

in any order.

- optimizer/ItemExpr.cpp

optimizer/ItemNAType.h:

Added a named NATypeToItem item expression.

This is used to do a primitive "bind" operation of an item expression

when processing a DDL statement. Specifically, to bind the DIVISION BY

clause in a CREATE TABLE statement.

- optimizer/ItemFunc.h

optimizer/SynthType.cpp: The DDL time "binder" gets expressions as

they come out of the parser, e.g. a ZZZBinderFunction. Need to add

type synthesis for some cases of the ZZZBinderFunction.

- optimizer/NATable.cpp

Removing some dead code. Adding an error message when we encounter

a Hive column type we can't handle yet. Bug 1392450.

- optimizer/TableDesc.*

Method TableDesc::validateDivisionByClauseForDDL() got moved

to CmpSeabaseDDL::validateDivisionByExprForDDL().

- optimizer/NormItemExpr.cpp

BETWEEN transformation described above.

- optimizer/ValueDesc.cpp

Avoid hard-codeing the "_SALT_" name and adding a comment about

possibility to use the flag in the future.

- parser

Lots of small changes for salt and divisioning option changes.

Simplifying the syntax for salt options somewhat. I think the older

syntax was so complex because it needed to record the starting and

ending position of the divisioning clause, something we don't need

anymore.

- regress: Adding new test

- sqlcomp/CmpDescribe.cpp: Support for describing DIVISION BY clause

and also supporting the new WITHOUT SALT | DIVISION options

for CREATE TABLE LIKE, which relies on the describe feature.

- sqlcomp/CmpSeabaseDDLcommon.cpp

sqlcomp/CmpSeabaseDDL.h

+ Handling the new column flags and making sure they are not

confused with the HBase column flags (e.g. for serialization).

+ Setting the new COLUMNS.FLAGS when writing metadata.

+ Also, writing the computed column text to the TEXT table.

+ For DROP TABLE, unconditionally deleting TEXT rows, since the

table could contain computed columns.

+ When building ColInfoArray, check system column flags, since

system columns can now appear at any position.

+ Add method to "bind" an item expression during DDL processing

without going through the full binder. This replaces any column

reference with a named NATypeToItem node, since all we really

need is the type and the name for unparsing.

+ Method TableDesc::validateDivisionByClauseForDDL() got moved

to CmpSeabaseDDL::validateDivisionByExprForDDL() with some minor

adjustments, since it used to be called on a bound ItemExpr, now

it gets called on something that came out of the parser and went

through the DDL time "binder".

- sqlcomp/CmpSeabaseDDLindex.cpp:

Support for CREATE INDEX ... DIVISION LIKE TABLE. If this is

set, add the division columns in front of the index key, otherwise

don't.

- sqlcomp/CmpSeabaseDDLtable.cpp:

+ Code to make sure column flags and column class is set and propagated.

+ Fix for bug 1385543: Now that we use the TEXT table for computed

column text, we no longer have a length limit. This is true for both

divisioning and salt expressions.

+ When processing the column list in seabaseCreateTable() we have a

bit of a chicken and egg problem: We need the column list to validate

the DIVISION BY expressions, but the DIVISION BY columns need to be part

of the column list. So, we do this a first time without divisioning

columns, then we add those, and produce the final list in a second

iteration.

+ getTextFromMD method now takes a sub-id as an input parameter. That's

the column number for computed column text.

+ read computed column text from the TEXT table. Note: This also needs

to handle older tables where the computed column text is stored in

the default value.

Change-Id: I7c3ebe39a950c1d01f31855bdc92cbb98e5eb275

    • -24
    • +0
    /sql/SqlCompilerDebugger/TDBTreeView.cpp
  1. … 36 more files in changeset.
Bug fixes

-- fix for LP bug 1396311: failed primary key constraint was

not cleaning up metadata entry

-- removed obsolete code reference in gui debugger

-- disabled display of software version stored in metadata until

install scripts add enhancement to update metadata with

current software version

Change-Id: I3922e689feec1d0e048083b9a36e6b43f2def49d

    • -24
    • +0
    /sql/SqlCompilerDebugger/TDBTreeView.cpp
    • -2
    • +10
    /sql/sqlcomp/CmpSeabaseDDLtable.cpp
Minor tracing updates, addresses TransactionState leak

Change-Id: I75cb9f2486404b3e6cbce39247bbdf2b668d4562

    • -0
    • +4
    /sql/regress/seabase/DIFF025.KNOWN
    • -298
    • +300
    /sql/regress/seabase/EXPECTED025
    • -265
    • +262
    /sql/regress/seabase/TEST025
Merge "fix for hive/test018"

Merge "removal of obsolete code that is no longer valid and unused in Trafodion"

fix for hive/test018

Query caching seems to cause hive/test018 to fail

disabling it for now till the query caching

issue is fixed

bug 1396386

Change-Id: Ife834050ce8f7d7b968488f5a5ff3f189ac2b666

removal of obsolete code that is no longer valid and unused in Trafodion

Change-Id: Ie9212627d102661fe1a7b02fd6e9ae9a23198f2a

    • -136
    • +0
    /sql/comexe/ComTdbInterpretAsRow.cpp
    • -155
    • +0
    /sql/comexe/ComTdbInterpretAsRow.h
  1. … 21 more files in changeset.
LOB datatype infrastructure support

Technology preview.

More changes expected as part of this work before it

is user ready.

blueprint lob-support

This checkin contains basic support for create blob/clob datatypes.

The feature is disabled by default. Instructions on how to enable are

listed below.

New test executor/TEST130 that turns the feature on and tests out the

functionality.

New mxlobsrvr process will be started as part of sqstart.

Create and drop of tables with LOB columns.

No support for alter.

DML support for LOB datatypes.

Insert, update and deletes. Joins of 2 tables with LOB columns are

allowed but joins on LOB columns temselves are not allowed.

Insert-select from one LOB table another not yet supported.

Link to document from the blueprint will be added shortly.

To enable and try LOBs:

On a developer workstation :

cqd TRAF_BLOB_AS_VARCHAR 'OFF';

On a cluster after installing the code 2 steps are needed:

1. cqd TRAF_BLOB_AS_VARCHAR 'OFF';

2. sudo su hdfs --command "hadoop fs -mkdir /lobs"

sudo su hdfs --command "hadoop fs -chown -R trafodion:trafodion

/lobs"

This checkin includes several merges from the mainline and each of

the lines below represents one commit to the project branch where this

work was done. `

-Turning off LOB code by default. But turning it on in

executor/TEST130 just to ensure testing the code path.

-Support for showddl and some syntax for external files and stream.

-LOB regression test

-Workaround for dtm issue LP 378167

-Changes to make append work. Changes to use lob heap.

-Fix for using system heap for all LOB allocations and handling NULLs.

-Added workaround for cursor delete issue. LP 1376969

-Fixes for update.

-Parser changes for exe_util_lob_extract

-Pull in lob extract code

-Adding mxlobsrvr directory

-Fixed the LOB interface to use 2 new params for cursor fetches. They do

not overload the LOB Handle and LOB handle length anymore.

Added a flag to lobGlobals to indetify it's a hive access.

Cleaned up parser code.

-LOB support for create,drop,insert,delete,select.

Change-Id: I7c8125696e847b71580b746388632e75741bd347

    • -0
    • +1138
    /sql/executor/ExExeUtilLoad.cpp
  1. … 38 more files in changeset.
Merge "Change default for USTAT_USE_IS_WHEN_NO_STATS"

Merge "Port change to get stats from memory rather than RMS, which is expensive."

NJ costing changes.

Change-Id: I1bab98a9cbb6745ad64fa8e8fd0e0011f6ef55c6

    • -13
    • +18
    /sql/optimizer/OptPhysRelExpr.cpp
    • -23
    • +262
    /sql/optimizer/ScmCostMethod.cpp
    • -0
    • +43
    /sql/optimizer/SimpleScanOptimizer.cpp
    • -142
    • +142
    /sql/regress/compGeneral/EXPECTED013.SB
Native external hbase table access (select, IUD) changes.

-- IUD on external hbase tables is now enabled by default

-- predicates on native hbase tables can now be pushed down to

hbase region server

-- traf varchar col maxlength is now 200K by default,

can be changed by cqd max_character_col_size

-- executor handles column values length greater than 32K during

move to/from JNI

-- error is correctly returned if data retrieved from hbase exceeds expected

max row length

-- hbase column_create function now takes an expression/param as its

column name operand

Change-Id: Ieb3fcabfebaa22008eff2a049fc1e2000e68861e

    • -811
    • +49
    /sql/generator/GenPreCode.cpp
  1. … 32 more files in changeset.