SqlciErrors.txt

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Using the language manager for UDF compiler interface

blueprint cmp-tmudf-compile-time-interface

This change includes new CLI calls, to be used in the compiler to

invoke routines. Right now, only trusted routines are supported,

executed in the same process as the caller, but in the future we may

extend this to isolated routines. Using a CLI call allows us to share

the language manager between compiler and executor, since language

manager resources such as the JVM and loaded DLLs exist only once per

process. This change is in preparation for Java UDFs.

Changes in a bit more detail:

- Added 4 new CLI calls to allocate a routine, invoke it, retrieve

updated invocation and plan infos and deallocate (put) the routine.

The CLI globals now have a C/C++ and a Java language manager that

is allocated on demand.

- The compiler no longer loads a DLL for the UDF compiler interface,

it uses the new CLI calls instead.

- DDL syntax is changed to allow TMUDFs in Java (not officially

supported, so don't use it quite yet).

- TMUDFs in C are no longer supported, only C++ and Java are.

Converted remaining TMUDF tests to C++.

- C++ TMUDFs now do a basic verification at DDL time, so errors

like missing entry points are detected earlier. Validation for

Java TMUDFs is also done through the CLI.

- Make sure we have no memory or resource leaks:

- CmpContext keeps track of UDF-related objects allocated on

system heap and in the CLI, cleaned up at the end of a statement

- CLI keeps a list of allocated trusted routines, cleaned up

when a CLI context is deallocated

- Using ExeCliInterface class to make the new CLI calls (4 new calls

added).

- Removed CmpCli class in the optimizer directory and converted

tracking compiler to use ExeCliInterface as well.

- Compile-time parameter values are no longer baked into the

UDRInvocationInfo. Instead, they are provided as an input row, the

same way as they are provided at runtime.

- Bug fixes in C++ UDR code, mostly related to serialization and

to multiple interactions with the UDF through serialized objects.

- Added more info to UDRInvocationInfo (SQL access type, etc.).

- Since there are multiple plans per invocation, each of which

can have multiple interactions with the UDF, plans need to be

numbered so the UDF side can tell them apart to attach the

right state (owned by the UDF) to it.

- The language manager needs some functions that are provided by

the process it's running in. Added those (empty, for now) functions

as cli/CliImplLmExtFunc.cpp.

- Added a new class for Java TMUDFs, LmRoutineJavaObj. Added methods

to allocate such routines and to load their class as well as to

create Java objects by invoking the default constructor through JNI.

- Java TMUDFs use the new UDR interface (to be provided by Suresh and

Pavani). In the language manager, the container is the class of

the UDF, the external path is the fully qualified jar name. The

Java method name is <init>, the default constructor, with signature

"()V". Some code changes were required to do this.

- Created a new directory trafodion/core/sql/src for Java sources in

the sql engine. Right now, only language manager java

sources are in this directory, but I am planning to move the other

java sources under sql in a future checkin. Suresh and Pavani

will add their UDF-related Java files there as well.

- Renamed the udr jar to trafodion-sql-<version>.jar, in anticipation

of combining all the sql Java sources into this jar.

- Created a maven project file trafodion/core/sql/pom.xml and

changed makefiles to invoke maven to build java sources.

- More work to separate new UDR interface from older SPInfo object,

so that we can get rid of SPInfo if/when we don't support the older

style anymore.

- Small fix to odb makefile, make clean failed when executed twice.

Patch set 2: Adding a custom filter for test regress/udr/TEST108.

Change-Id: Ic827a42ac25505fb1ee451b79636c0f9349d8841

  1. … 98 more files in changeset.
Column-level privileges

Support for column-level privileges will be in multiple deliveries.

This delivery add the following portions:

1. Creation of the metadata table COLUMN_PRIVILEGE.

This table is created when the INITIALIZE AUTHORIZATION command is run.

Existing privileges are preserved, but warnings are issued referring to

existing metadata tables. An UPDATE option will be added later.

2. Granting of column-level privileges

Full support is present for granting column-level privileges.

Privileges can be added and updated for one or more columns on a table or view.

Support for WITH GRANT OPTION is coded, though not enabled until WITH GRANT

OPTION is enabled at the object level.

3. SHOWDDL

The SHOWDDL command displays column-level privileges. Regardless of

the order the privileges were granted, SHOWDDL displays them in column

order, and within each column, in the order they appear in the bitmap

(SELECT, INSERT, UPDATE, REFERENCES).

4. Revoking of column-level privileges

Only partially implemented. The basic operation of revoking granted

column-level privileges and grant option for is implemented. All

relevant security checks are performed. GRANTED BY is not implemented.

RESTRICT and CASCADE options are not supported. Hence, any dependent

objects remain when column-level privileges are revoked.

Missing functionality

In addition to column-level revoke only be partially implemented,

here are other items not present in this delivery:

1. Privileges can be granted to roles and revoked from roles,

but REVOKE ROLE does not consider column-level privileges when

determining if an object depends on a role's granted privileges.

2. Similarly, revoke at the object level does not consider

column-level privileges that may allow an object to remain after

an object-level privilege is revoked.

3. CREATE VIEW does not consider column-level privileges

when determining if the user has authority on the referenced

tables and views.

4. Run-time DML operations do not considered column-level when

determining if the user has authority to perform the query.

Change-Id: Icd3db88708d1e0ae7e9236e10b2a760bba287155

  1. … 17 more files in changeset.
Eliminate manual steps in load/ustat integration

The fix achieves full integration of the bulk load utility with

Update Statistics. The Hive backing sample table is now creeated

automatically (formerly, we only wrote the HDFS files to be

used by the Hive external table), the correct sampling percentage

for the sample table is calculated, and the ustat command is

launched fro1m the executor as one of the steps in execution of

the bulk load utility.

Change-Id: I9d5600c65f0752cbc7386b1c78cd10a091903015

Closes-Bug: #1436939

  1. … 26 more files in changeset.
TIMESERIES UDF for repository queries and UDF bug fixes

Bug fixes:

bug 1436593 TMUDF: getScale() returns a wrong scale for the TIME column

bug 1400812 Name resolution for predefined table mapping functions may need to be improved

bug 1436963 TMUDF: Unsigned numeric is mapped to TypeInfo::NUMERIC

bug 1436450 TMUDF: copyPassThruData() fails to pad nchar data properly

Added a predefined UDF to do timeseries queries. "Predefined" means that

like a built-in function it is not registered in the metadata. It is still

a UDF, though, using the SDK for UDFs.

Here is how to invoke the UDF:

select ...

from udf(timeseries(table(select ...

from ...

[partition by ...]

order by <tscol>),

<name of time slice column>,

<time slice width>

[ { , <col name>, <instr> } ... ]

)

)

<tscol> is a date, time or timestamp column from the input table

that describes the time dimension of the data. The data

can optionally be partitioned into multiple time series

that are independent of each other, using a PARTITION BY.

<name of time slice column> is the name of the generated output

column that contains the starting time of each time slice.

<time slice width> is an interval literal that determines how wide

each time slice is.

An optional list of pairs of <col name> and <instr> indicates

column values to be interpolated, according to the instructions.

Instructions:

Instruction Value at Interpolation Ignore nulls

----------- --------- ------------- ------------

FC beginning constant no

LC end constant no

FCI beginning constant yes

LCI end constant yes

FL beginning linear no

LL end linear no

FLI beginning linear yes

LLI end linear yes

Example:

select *

from udf(timeseries(table(select cust_id,

tstamp,

kwh

from e_meters

partition by cust_id

order by tstamp),

'HOURLY_READING', -- name of time slice column

interval '1' hour, -- time slice width

'KWH', 'FL', -- value of KWH column at beginning

-- of time slice, use linear

-- interpolation

'KWH', 'LCI')); -- end value, constant interpolation,

-- ignore NULL values

This will chop the time range of each customer into time slices,

1 hour wide, and will use linear interpolation for the meter readings

(assume we have readings for cust1 at 8:00 for 1000 and 10:30 for 1002

and readings for cust2 at 8:00 for 400 and at 9:30 with a NULL value).

CUST_ID HOURLY_READING KWH_FL KWH_LCI

------- ------------------- -------- -------

cust1 2015-03-20 08:00:00 1000.00 1000

cust1 2015-03-20 09:00:00 1000.80 1000

cust1 2015-03-20 10:00:00 1001.60 1002

cust2 2015-03-20 08:00:00 400.00 400

cust2 2015-03-20 09:00:00 ? 400

Other changes:

- Added DCS gui support to install_local_hadoop. If installed with

non-standard ports, see file $MY_SQROOT/sql/scripts/swurls.html

for the port numbers to use. I would recommend that you bookmark

this file in the browser you are using locally on your workstation.

- Addressed comments made by Dave B. in earlier checkins:

- Make error message 3286 more easy to understand.

- Change name resolution rules for predefined UDRs such that

real (user-defined) UDRs take precedence.

- Add comments to Trafodion engine files where some logic

is duplicated in the UDR SDK (file sqludr/sqludr.cpp and

in the future the equivalent Java file).

- Fix for "orphan entries in up queue" assert when canceling

a TMUDF while it is still reading data from its table-valued

inputs.

Patch set 2: The jenkins build flagged some warnings as errors

that were not flagged on my workstation.

Change-Id: I1b806e35e2b2e91a42318fbbfd788e92d8cba070

  1. … 22 more files in changeset.
cli api to store explain in repository and few more changes.

-- new api to stored packed explain in repository

SQL_EXEC_StoreExplainData.

-- storage of packed explain in encoded form to

mask out any null characters. This helps with

treating of explain data as a string.

-- bug fix in cleanup code

-- fix to handle input param values greater than short max.

-- test code in sqlci frontend to test explain get/store apis.

-- fixed udr/TEST001

Change-Id: I1d501cb93eb7d32808405863afdbdb5c1cec79ee

  1. … 23 more files in changeset.
Fixes for a few scalar UDF bugs

LP 1426605: change in NormRelExpr.cpp. When left linearizing a join backbone

sufficients inputs were not being provided. The change ensures that inputs

from the old tree and still marked as required inputs for a node in the new

tree.

LP 1420530: Error handling added to BiArith::bindNode.

LP 1420938: Error handling to CREATE FUNCTION statement to flag more than 32

parameters.

LP 1421438: showddl [function | procedure | table_mapping function] <name>;

now works. If one of the optional tokens is not specified then we will look

for a table called <name>.

Patch Set 1

Changes to address comments by Dave.

One more fix in ExUdr.cpp. There is no LP for this bug. If a dll is missing

at runtime or other LOAD errors during UDF fixup could lead to an assertion,

since we try to place an error in UDF's up queue, before there are entries

in the corresponding down queue. Fix is to remove this line and let existing

error handling report this error. Thanks for your help Hans.

Couple of items that I forgot to mention before

1) Changes in Analyzer.cpp related to printing predecessorJBBC are due to Hans.

2) Showddl code is mostly refactored from previous versions.

Change-Id: Idfde89d73c47735c4405befa6b9cdd4ae0d2e641

  1. … 14 more files in changeset.
Fixes for TMUDF bugs

Bugs fixed in this change:

bug 1430034 TMUDF: processData() fails to handle several data type

bug 1430438 TMUDF: A TMUDF with nonexisting external name crashes

sqlci with a core

bug 1430453 TMUDF: A TMUDF missing 'language cpp' crashes sqlci with a core

bug 1430484 TMUDF: User defined error number for UDRException() not

returned at run time

bug 1404053 Core with SIGSEGV during CREATE PROCEDURE statement when

procedure validation fails

Summary of changes:

- Support for more data types, including char types with character set UCS2

and interval types. Unlike in other types of UDFs, intervals are

represented as integers, but that representation is not directly

exposed to the UDF writer. CLOBs and BLOBs are only supported when

they are mapped to VARCHARS.

- Support to get and set values by specifying a C++ data type of time_t

for datetime and certain interval values.

- The default language for creating UDFs is changing. It used to be

C, the new rule depends on the type of the library. If we can determine

the library to be a jar file, the default language is Java. For

other libraries, the default is C for scalar UDFs, C++ for table-valued

UDFs. Also the PARAMETER STYLE clause is no longer used for TMUDFs, the

parameter style is computed automatically for now. PARAMETER STYLE is

now optional for Java stored procedures.

- Error handling is somewhat improved, fewer internal errors occur when

a UDR raises an exception. Still needs work.

- Making interfaces of OrderInfo and PartitionInfo more similar

by giving them similar methods.

Change-Id: I0bd0cebcf32f2185907c7d3573d4a511b17ead3e

  1. … 36 more files in changeset.
repository upgrade and explain enhancements. Details below.

-- repository upgrade infrastructure integrated

with 'initialize trafodion, upgrade'

-- fields have been added/removed/modified in repos tables

-- new repos table has been added

-- upgrade enhanced to only upgrade subset of components that

need to be upgraded. One or more of metadata, repos, views, priv.

-- metadata version now tracks release version and indicates

the traf release where upgrade was needed.

Current metadata version will be 1.1.0.

-- packed explain data is now returned to caller so it could be

stored in repository

-- explain info from repository for a query id could be retrieved

and displayed

explain qid <query-id>;

explain options 'f' qid <query-id>

select * from table(explain(null, 'EXPLAIN_QID=<query-id>'));

-- a sql query could be explained and returned in relational format.

select * from table(explain(null, 'EXPLAIN_STMT=<query-str>));

Users only need to have select permission on tables referenced

in query-str.

-- explain and statistics tables could be invoked

invoke table(explain(null, null));

invoke table(statistics(null, null));

Squashed commit of the following:

commit 09004ff7260b771378bc1a29f0ecb82e5e0c6100

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 16 12:24:03 2015 -0700

repos and explain, #12

Change-Id: Ia08151b5c7087b6b7daa5b662df2a584e5a7b2a1

commit 46f05ada264094626e45445d3875ccf876cad802

Merge: 3393587 6b3a4c4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 21:27:48 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 3393587de319a3ad43d7d360cd74501e8b8103e4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 21:26:30 2015 -0700

repos and explain, #11

Change-Id: I895a4b4766e0a92b7472a6b347f0bb3ee07fa9b6

commit cf6ffee549b37ffbcd6ce750a1fdbf7f0c410d61

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 16:07:22 2015 -0700

repos and explain, #10

Change-Id: I19e182ec6600525cec986f173afaf68a44e04af2

commit e331eec68feea59d9fa11e0154e3093762120eb2

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 18:19:03 2015 -0700

repos and explain, #9

Change-Id: I783c75513a432f718d2cd7c6030d410c419a79c8

commit 9e9da221eb45b760963741657fe9cb910a753c84

Merge: 20b64e5 9eb7f37

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 14:52:09 2015 -0700

Merge remote branch 'gerrit/master' into fix1

Conflicts:

sql/common/ComSmallDefs.h

sql/executor/ExExeUtilCli.h

sql/generator/GenRelExeUtil.cpp

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLupgrade.cpp

sql/sqlcomp/CmpSeabaseDDLupgrade.h

Change-Id: Id21b10ff35ad506f409a83a8d955c2e643a2bcf5

commit 20b64e5552d8f614586ed843aad76d87d3473e2e

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 13:44:15 2015 -0700

repos and explain, #8

Change-Id: I2b49d962f08a22ca6c8b437a583855e2344f513c

commit 500b7a9486f23fff6b1c17bb6e79c2f614a7e3e6

Merge: f2c050e fc1c31d

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 08:07:34 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit f2c050ea048b409b7d6769172e16c4ab26267bd0

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 08:07:15 2015 -0700

repos and explain, #7

Change-Id: I0bc04b6e8f147af1defbfd023c3700e26cc5b6f1

commit fabdb7dadd34238a8a227e880aa868ce484dc580

Merge: 59bb689 74640d7

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Mar 12 09:01:20 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 59bb6893437abe3999a53188503c60cf47633458

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Mar 12 09:00:44 2015 -0700

repos and explain, #7

Change-Id: I6595a04ad59c1f705a04beee18c877cb81db0fac

commit a86353b45436126915fd2ba74c9af53f8bfd4d19

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 9 16:21:16 2015 -0700

repos and explain, #6

Change-Id: Ie4fe2b2bece7de3697eb8ada0d1cb1067b89bb88

commit 61272e0d6107536ab8b691415770e856267b6da5

Merge: 1708cb4 ca5dfb4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 9 15:52:43 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 1708cb44511809bf8e63800d226ac8ea34916726

Merge: 1284f4f 31f25ec

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 6 14:55:13 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 1284f4f78f1f43657f29899cc8b9a96c02f37815

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 6 14:54:54 2015 -0800

commit: repos and explain, #6

Change-Id: I38d796fd4ab58bf29fa62c4bd5492ce44de9c8a0

commit f639911a1ea4247ae4c297bbbd904b84c7587b3a

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 2 06:31:21 2015 -0800

repos and explain, #5

Change-Id: Ie97ba3cac8dc78784d0a7eef57176e51b9e16224

commit da69fe9595d4ad5c9adf3b32ae8f848a0e6be964

Merge: 9b95b8c a5caeef

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 2 06:31:00 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 9b95b8c3240a4186704b8b7f2ff8cefebf211732

Merge: ea2f687 8cd8a92

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Feb 26 08:50:25 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit ea2f6871b7c7803d0f1e3fa664b474e069a0befd

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Feb 26 08:50:03 2015 -0800

commit: repos and explain, #5

Change-Id: I44d8f87b3652e6300e41302d3a3e88b9b18652a3

commit 522a47eb991df2245005c693e4d034235adbfdc5

Merge: d986988 f42694e

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 25 13:13:02 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit d98698856a4be479a7603b0fb79210e80e88e1c1

Merge: 5841e47 2aad3ef

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 18 14:00:24 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 5841e47f1e3f9a018424298a92cd134f4430e6bf

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 18 13:57:46 2015 -0800

repos and explain, #4

Change-Id: I01f53003069598381743c9be80b405ed37cefe6d

commit e7b5966d8ce653a57b8128c0b24c15fc05737668

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 11 11:28:20 2015 -0800

repos and explain, #3

Change-Id: Iff148ef175a7641ea37e27e60435ccf8f613e6db

commit aba98dc8cfda62d4e708cd892ed626f2a3051eb0

Merge: 66f1926 19a0c32

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 10 14:09:58 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 66f1926d279f2b408d436b0aa53a3a6a5388901e

Merge: 9d0ef2a b68f59b

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Feb 9 10:52:35 2015 -0800

Merge remote branch 'gerrit/master' into fix1

Conflicts:

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLtable.cpp

Change-Id: I6348fec3aece2baad9e9570749f43c086416219e

commit 9d0ef2aa01a62ea3e4e0ab35f8ec9687ef3336ef

Merge: aac268d 2cd8fbb

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Feb 6 14:15:49 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit aac268d097afbe506dcf76de4d8f49652a1d4df7

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 16:31:17 2015 -0800

repos and explain, #2

Change-Id: Ia2a50159e577cd6ff8110dca35236fa90ea0d260

commit 2e6160411c8facb448e78d387fa3227e7b759c5d

Merge: c6e2fe5 566706d

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 15:21:03 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit c6e2fe529f9d1e59d3eb26cb145b260d81814443

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 15:16:36 2015 -0800

repository updated and explain changes, #1

Change-Id: I5daf66a2064b46ad65f1200579385965f7f56808

Change-Id: I0211d754524b8c06a05bb7ade20f884ec91340d2

  1. … 58 more files in changeset.
Trafodion Metadata Cleanup command support.

Various changes to support cleanup command has been added.

A separate external spec contains the details.

Summary of syntax:

cleanup [ table t | index i | sequence s | object o] [, uid <value>]

cleanup [private | shared] schema sch

cleanup uid <value>

cleanup metadata, check, return details

In addition, a new command to get names of various hbase objects

has also been added:

get [ all | user | system | external ] hbase objects;

Change-Id: I93f1f45e7fd78091bacd7c9f166420edd7c1abee

  1. … 79 more files in changeset.
Normalizer interface for TMUDFs, blueprint cmp-tmudf-compile-time-interface

Added new compiler interfaces:

describeParamsAndColumns has been extended to allow updating

PARTITION BY and ORDER BY clauses specified for input tables.

describeDataFlowAndPredicates() allows the TMUDF to eliminate

columns not needed by the query and to push predicates into

the TMUDF or to the children.

describeConstraints() allows the TMUDF to see cardinality and

uniqueness constraints of the table-valued inputs (children) and

to synthesize cardinality and uniqueness constraints on the

TMUDF result.

TMUDFs now have 3 function types:

GENERIC - makes most conservative assumptions in the compiler

MAPPER - assumes TMUDF carries no state between input rows

REDUCER - assumes TMUDF carries no state between input partitions

defined by PARTITION BY clause

Query id and user id are now available to the UDR.

Added doxygen documentation for the C++ UDR interface. The resulting

web page will be published on the wiki. To generate the documentation

yourself, do the following:

cd $MY_SQROOT/../sql/sqludr

doxygen doxygen_tmudr.1.6.config

# now open tmudr_1.0/html/index.html in a web browser

Patch set 2: Updated copyrights in 2 files.

Change-Id: I3735eb3dd7e5292ba308ac00332fdebdf66a7472

  1. … 25 more files in changeset.
Snapshot Scan changes

The changes in this delivery include:

-decoupling the snapshot scan from the bulk unload feature. Setup of the

temporary space and folders before running the query and cleanup afterwards

used to be done by the bulk unload operator because snapshot scan was specific

to bulk unload. In order the make snapshot scan indepenednt from bulk unload

and use it in any query the setup and cleanup tasks are now done by the query

itself at run time (the scan and root operators).

-caching of the snapshot information in NATable to optimize compilation time

Rework for chaching: when the user sets TRAF_TABLE_SNAPSHOT_SCAN to LATEST

we flush the metadata and then we set the caching back to on so that metadata

get cached again. If newer snapshots are created after setting the cqd they

won't be seen if they are already cached unless the user issue a command/cqd

to invalidate or flush the cache. One way for doing that can be to issue

"cqd TRAF_TABLE_SNAPSHOT_SCAN 'latest';" again

-code cleanup

below is a description of the CQds used with snapshot scan:

TRAF_TABLE_SNAPSHOT_SCAN

this CQD can be set to :

NONE--> (default)Snapshot scan is disabled and regular scan is used ,

SUFFIX --> Snapshot scan is enabled for the bulk unload (bulk unload

behavior is not changed)

LATEST --> Snapshot Scan is enabled independently from bulk unload and

the latest snapshot is used if it exists. If no snapshot exists

the regular scan is used. For this phase of the project the user

needs to create the snapshots using hbase shell or other tools.

And in the next phase of the project new comands to create,

delete and manage snapshots will be add.

TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX

This CQD is used with bulk unload and its value is used to build the

snapshot name as the table name followed by the suffix string

TRAF_TABLE_SNAPSHOT_SCAN_TABLE_SIZE_THRESHOLD

When the estimated table size is below the threshold (in MBs) defined by

this CQD the regular scan is used instead of snapshot scan. This CQD

does not apply to bulk unload which maintains the old behavior

TRAF_TABLE_SNAPSHOT_SCAN_TIMEOUT

The timeout beyond which we give up trying to create the snapshot scanner

TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION

Location for temporary links and files produced by snapshot scan

Change-Id: Ifede88bdf36049bac8452a7522b413fac2205251

  1. … 44 more files in changeset.
Misleading error when Hivestats can't be created

When Update Statistics is performed on a Hive table, the

trafodion.hivestats schema is created if it doesn't already

exist. If the user issuing the ustat statement does not

have the create_schema component privilege, this will fail, but

the code as it existed ignored the error, assuming it was

a "schema already exists" error, which is expected in most

cases. If the schema does not exist, an error is then given

when an attempt is made to create the histograms table in

the non-existing schema.

This has been changed to check the error when the schema

creation attempt fails, so the correct error is generated.

This is a partial fix, because a user with manage_statistics

privilege should not be prevented from executing Update

Statistics just because the trafodion.hivestats schema did

not already exist. The likely full solution would be to

create the schema automatically at Trafodion initialization.

Change-Id: I26561100bab9f6a59f3f5d0d56ce9d2aab380fa3

Partial-Bug: #1425748

  1. … 4 more files in changeset.
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

  1. … 39 more files in changeset.
C++ run-time interface for TMUDFs

blueprint cmp-tmudf-compile-time-interface

- Support for C++ run-time interface:

- A new language, C++ is added to langman, the existing

LanguageManagerC handles both C and C++

- Two new parameter styles got added, C++ and Java

object-oriented parameter styles. Routines written in C++

use the new object-oriented C++ parameter style. The compiler

interface is only supported for that style (and in the future

for the Java object-oriented style).

- Also added one more compile time interface, the "completeDescription()"

call in the generator. Added logic to extract the UDRPlanInfo of

the optimal plan.

- Changes to UDRInvocationInfo and UDRPlanInfo classes:

- UDRInvocationInfo and UDRPlanInfo objects can now be serialized

and they are added to generated plans, as part of the UDR TDB.

- Split TableInfo into TupleInfo and TableInfo classes. TupleInfo

is now the common base class for describing both parameters and

input/output tables.

- TypeInfo now has offsets for data, null indicator and varchar

indicator.

- New get<type> and set<type> methods on class TupleInfo, to be

used at compile time for parameters and at runtime for parameters,

input and output tables.

- Added a "call phase" member, to be able to throw exceptions when

certain methods are called at the wrong time (e.g. trying to modify

compile time members at runtime).

- Routine class in langman now has a new subclass, LmRoutineCppObj

and a new method, invokeRoutineMethod, that is used to invoke

the object-oriented methods, requiring UDRInvocationInfo and

UDRPlanInfo as parameters.

- Fixed some executor issues with error handling for UDFs, this is

still not very well supported

- Emitting the EOD row in the UDF is no longer required, and no longer

supported or even possible.

- UDRPlanInfo is now part of the physical properties, so that we

can extract it from the optimal plan.

- Disabling TMUDF as the inner of a nested join - for now.

We might support this "routine join" at a later time.

- regress/udr/TEST001:

- SESSIONIZE_STATIC remains in C, but other TMUDFs are now

rewritten in C++ (the runtime part that was not yet in C++)

- SESSIONIZE_DYNAMIC is now the same as the example on the wiki

- regress/udr/TEST002: Added some tests for event log reader UDF,

but can't add the part that copies a sample log file, since

in Jenkins, we don't have $MY_SQROOT set. Tried the test on my

workstation, though. Steve tells me $MY_SQROOT should be available,

so in a future checkin I'll enable this code again.

- For patch set 2: Removed fix for LP bug 1420539 and addressed

other review comments.

Change-Id: I008ad68a8f25f1aaee94e1c45bbf097a267129bb

  1. … 73 more files in changeset.
Bulk unload optimization using snapshot scan

resubmitting after facing git issues

The changes consist of:

*implementing the snapshot scan optimization in the Trafodion scan operator

*changes to the bulk unload changes to use the new snapshot scan.

*Changes to scripts and permissions (using ACLS)

*Rework based on review

Details:

*Snapshot Scan:

----------------------

**Added support for snapshot scan to Trafodion scan

**The scan expects the hbase snapshots themselves to be created before running

the query. When used with bulk unload the snapshots can created by bulk unload

**The snapshot scan implementation can be used without the bulk-unload. To use

the snapshot scan outside bulk-unload we need to use the below cqds

cqd TRAF_TABLE_SNAPSHOT_SCAN 'on'; --

-- the snapshot name will the table name concatenated with the suffix-string

cqd TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX 'suffix-string';

-- temp dir needed for the hbase snapshotsca

cqd TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION '/bulkload/temp_scan_dir/'; n

**snapshot scan can be used with table scan, index scans etc…

*Bulk unload utility :

-------------------------------

**The bulk unload optimization is due the newly added support for snapshot scan.

By default bulk unload uses the regular scan. But when snapshot scan is

specified it will use snapshot scan instead of regular scan

**To use snapshot scan with Bulk unload we need to specify the new options in

the bulk unload syntax : NEW|EXISTING SNAPHOT HAVING SUFFIX QUOTED_STRING

***using NEW in the above syntax means the bulk unload tool will create new

snapshots while using EXISTING means bulk unload expect the snapshot to

exist already.

***The snapshot names are based on the table names in the select statement. The

snapshot name needs to start with table name and have a suffix QUOTED-STRING

***For example for “unload with NEW SNAPSHOT HAVING SUFFIX ‘SNAP111’ into ‘tmp’

select from cat.sch.table1; “ the unload utiliy will create a snapshot

CAT.SCH.TABLE1_SNAP111; and for “unload with EXISTING SNAPSHOT HAVING SUFFIX

‘SNAP111’ into ‘tmp’ select from cat.sch.table1; “ the unload utility will

expect a snapshot CAT.SCH.TABLE1_SNAP111; to be existing already. Otherwise

an error is produced.

***If this newly added options is not used in the syntax bulk unload will use

the regular scan instead of snapshot scan

**The bulk unload queries the explain plan virtual table to get the list of

Trafodion tables that will be scanned and based on the case it either creates

the snapshots for those tables or verifies if they already exists or not

*Configuration changes

--------------------------------

**Enable ACLs in hdfs

**

*Testing

--------

**All developper regression tests were run and all passed

**bulk unload and snapshot scan were tested on the cluster

*Examples:

**Example of using snapshot scan without bulk unload:

(we need to create the snapshot first )

>>cqd TRAF_TABLE_SNAPSHOT_SCAN 'on';

--- SQL operation complete.

>>cqd TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX 'SNAP777';

--- SQL operation complete.

>>cqd TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION '/bulkload/temp_scan_dir/';

--- SQL operation complete.

>>select [first 5] c1,c2 from tt10;

C1 C2

--------------------- --------------------

.00 0

.01 1

.02 2

.03 3

.04 4

--- 5 row(s) selected.

**Example of using snapshot scan with unload:

UNLOAD

WITH PURGEDATA FROM TARGET

NEW SNAPSHOT HAVING SUFFIX 'SNAP778'

INTO '/bulkload/unload_TT14_3' select * from seabase.TT20 ;

Change-Id: Idb1d1807850787c6717ab0aa604dfc9a37f43dce

  1. … 35 more files in changeset.
The following Launchpad bugs are fixed in this change:

Bug 1370749: Now using MAX_USERNAME_LEN instead of hardcoded value

Bug 1413760: CREATE TABLE LIKE was failing in some circumstances because

SHOWDDL was including the BY clause. Ownership rules changes in

CREATE TABLE changed when ANSI schemas was implemented, so the BY clause

is no longer needed.

Bug 1392107: Privileges granted on a view are no longer lost if the

view is replaced via CREATE OR REPLACE VIEW.

Bug 1370740: A potential memory corruption problem is now avoided

by reworking the authorization name lookup functions.

Bug 1413767: Previously DROP SCHEMA CASCADE would fail to drop a

table with an IDENTITY column.

Bug 1413758: Previously DROP TABLE CASCADE did not drop nested views.

Bug 1412891: Previously DROP TABLE CASCADE failed if a dependent object

contained a delimited name.

Changes are present for 1392086, but the work is not yet completed.

This problem is related to roles and security keys.

Code changes are also present for giving ownership of an object to

another authorization ID, but these changes are not complete. A

description of

the changes is included.

The GIVE command transfers ownership of a SQL item from one

authorization ID to another. Implemented in this delivery is

GIVE SCHEMA and GIVE ALL.

GIVE ALL transfers all SQL items owned by an authorization ID to another

authorization ID. Current or new owner can be a user or a role. The

GIVE ALL command requires the ALTER privilege.

GIVE ALL FROM authID TO authID

GIVE SCHEMA behavior depends on the type of schema and whether RESTRICT

or CASCADE is specified. For private schemas, all the objects in the

schema are given, as well as the schema itself. For shared schemas,

only the

schema is given, unless the CASCADE option is specified. In that case,

ownership

of all the objects in the shared schema is given to the new owner. Use

of

the CASCADE option requires the ALTER_SCHEMA privilege. Otherwise, GIVE

SCHEMA only requires the user to be the owner of the schema.

GIVE SCHEMA schema-name TO authID [RESTRICT|CASCADE]

NOTE: RESTRICT and CASCADE are not applicable to private schemas and are

ignored.

GIVE OBJECT is added to the syntax but is not implemented and may not

be implemented.

A more detailed blueprint will be provided prior to the final delivery

of GIVE.

Change-Id: I7449da599dc80de1c0659164e684841cda4647c8

  1. … 34 more files in changeset.
Fix for bug 1409939 create table like does not handle salt option

showddl does not display salt clause for table that was created via

'create table...like...salt using n partitions'.

There are really two issues here. First, the condition to salt the

target table in CREATE TABLE LIKE is too restrictive (requires with

PARTITIONS clause). Fixed that. Second, we accept additional table

attributes in CREATE TABLE LIKE but we don't actually process them -

yet. Added a new error to indicate that.

Change-Id: Ia82322b4f587674257e888a080420e3d41031e31

  1. … 2 more files in changeset.
Merge " Hybrid Query Cache feature implemented."

  1. … 4 more files in changeset.
Manageability changes - event mgmt and stats publication

Implements changes to support event management using log4cpp.

Configuration files are located in $MY_SQROOT/conf folder and all logs

files are located in $MY_SQROOT/logs folder

For more information see the blueprint at:

https://blueprints.launchpad.net/trafodion/+spec/eventmanagement

Implements changes for publication of statistics to repository. For more

information see the blueprint at:

https://blueprints.launchpad.net/trafodion/+spec/repositorymetrics

Note:

In this initial delivery publication of statistics is disabled by

default and it can be enabled via DCS property. This code has been

reviewed internally prior to merging with mainline

Documentation:

https://wiki.trafodion.org/wiki/index.php/Trafodion_Manageability

Included timestamp to be part of the primarykey for metric aggregation

table

Addressed some of the comments and incorporated Anoop's change for

repository

Changed the queryBuf size in sql/sqlcomp/CmpSeabaseDDLrepos.cpp to 20000

Modified the sql/regress/seabase/EXPECTED024

Change-Id: I517575233c10b2a8683cdd1d53a2eec96d7c2a6f

  1. … 781 more files in changeset.
ANSI Schema changes

ANSI Schema

Implements the changes to support ANSI schemas. For more information

see the blueprint at:

https://blueprints.launchpad.net/trafodion/+spec/security-ansi-schemas

The syntax changes for REGISTER USER and CREATE ROLE were not

implemented in this delivery.

NOTE: This code was reviewed internally prior to merging with the

main branch.

Change-Id: I1c7937dbcd067e792dcacb65f12c43e4f84a25ad

Change-Id: I98395eeef1e8bde424d9e83f96928358f0b1991b

  1. … 75 more files in changeset.
Various changes, details listed below.

-- fixed error msg 1429 text

-- added code to set objectUID & owner for metadata, histogram and

sequence tables during creation of metadata structs for these objects.

-- removed previously added code in binder that computed objectUID

for sequence.

-- Updated method lookupObjectUid to call an existing method

to get objectuid.

-- removed obsolete code for reorg, replicate and load

Change-Id: I60d161cfa72bcc674dc6c64e3a07237c7522ee6c

  1. … 28 more files in changeset.
Fixes and removal of obsolete code.

-- LP 1400556 'get tables in schema' is not supported on external

hbase tables. An error is now returned.

-- LP 1400553 Insert into external hbase tables in _ROW_ format must use

column_create function and VALUES clause to create rows.

An error is returned otherwise.

-- a bug with that prevented a boundary case when sequence increment value

was one less than largeint max has been fixed.

-- error message to indicate what options can be used during alter sequence

has been updated

-- create table as select stmt now returns an error if running within a user

transaction. This is the same behavior as other DDL operations.

This will be

removed once we have transaction support for DDL stmts.

-- create table as select now uses non-transactional 'upsert using load' to

populate target table instead of transactional 'insert...select' stmt.

-- hive/test020 has been enabled. This tests for access to ORC files.

-- obsolete sidetree insert and NVT user load code has been removed.

Change-Id: I14d321deaa52321777acd1d8ca55420f1e973367

  1. … 31 more files in changeset.
TMUDF C++ compiler interface, part of log-reading TMUDF

This is the infrastructure for a new C++ interface for TMUDFs

(table-mapping UDFs). It is used by a new log-reading TMUDF that

is not yet complete, but should be finished in the next few days.

See blueprint cmp-tmudf-compile-time-interface for more info.

Change-Id: I5a74e461462313b6d9722ac0deb21cd16c4b02ce

  1. … 55 more files in changeset.
Hybrid Query Cache feature implemented.

The Hybrid Query Cache (HQC) is an enhancement of existing Query Cache,

which is trying to find match queries in existing query cache at an earlier point,

i.e. just after parser and before binder so as to avoid binder overhead if there's a hit.

Two virutal table ISPs are added to show stats of Hybrid Query Cache.

Add control of ISP to run locally or remotely.

Changes after reviewers' comments.

Fixup errors in SqlciErrors.txt that cause core/TEST014 failure.

Fix minor issues about (hybrid)query cache ISP.

Add HQC virtual table ISP tests to compGeneral/TEST042.

Change-Id: Ib5be56e04990639153747255834b30fc9c3f3829

  1. … 40 more files in changeset.
Authorization checks for DDL & utilities

Fixed issues from code comments.

LOAD/UNLOAD authorization checks:

Code was added during code generation to make sure user has privileges,

if the user had necessary privileges, then the EXEUTIL parser flag is

turned on to avoid further privilege checks. When load/unload

completes, the parser flag is reset.

Update/showstats Statistics authorization checks:

Added a new error message

Changed hs_globals to support a new isAuthorized method and store

parser flags when class is instantiated and reset them when done

Changed hs_cli.cpp to use new IF NOT EXISTS syntax when creating

histogram tables, make owner of histogram tables DB__ROOT

(will need to adjust when schema privileges happen), and clean up

CreateHistTables method to remove old authorization mechanism

Changed hs_update.cpp which controls the update and showstats operation

to add authorization checks

Purgedata and populate index changes:

Changed CmpSeabaseDDLcommon.cpp to check privileges for purgedata

Changed CmpSeabaseDDLindex.cpp to check privileges for popindex

Additional component privileges and checks:

Added support for new component privileges in PrivMgrMD.h/.cpp

Added support for MANAGE_COMPONENTS

Added support for CREATE_INDEX and DROP_INDEX component privs

Fixes from last delivery that were postponed:

Context.cpp - fix for previous code review

CmpSeabaseDDLtable - added calls to deallocEHI

PrivMgrMD - fixed wording in a comment

Miscellaneous changes:

ComUser - added new convenience method - isRootUserID()

NATable.cpp (setupPrivInfo) to always set up privInfo_ and to call

the embedded compiler while extracting privileges

Privilege adjustments to take advantage of privInfo stored in NATable:

Added code to mark and rewind errors in diags.

Fix for LP bug 1392895

Change-Id: I6f7245ae7e66086769c0e92d901399c99e8f2af3

  1. … 33 more files in changeset.
Drop view QI fix & fix for LP 1384485

Removed sqlparser.y change - it was not intended to be delivered.

Drop view QI fix:

At drop view time, we need to send a QI message for the view being

dropped and remove base tables referenced by the view from cache.

Added a structure which describes a base table reference

Added code to create a list of base table references by calling

getListOfReferencedTables

getListOfReferencedTables gets the list for the first layer of

referenced objects (getListOfDirectlyReferencedObjects).

If the referenced object is a view, it recursively calls

getListOfReferencedTables

If the referenced object is a table, it gets added to the base

table list

Set up a QI key during drop view

Removed naTable entries for all tables referenced by the view

Updated TEST122 with drop view tests

Fix for LP 1384485:

Alter table allows adding a constraint to a _MD_ table

Reworded error message 1289

Added checks for reserved schema in missing places, mostly

constraints

Change-Id: I258983f2a28199ae522ba72b831f57fd22730a2f

  1. … 5 more files in changeset.
Support for divisioning (multi-temperature data)

This is the initial support for divisioning. See

blueprint cmp-divisioning for more information:

https://blueprints.launchpad.net/trafodion/+spec/cmp-divisioning

Also, this change fixes the following LaunchPad bugs:

Bug 1388458 insert using primary key default value into a salted

table asserts in generator

Bug 1385543 salt clause on a table with large number of primary

key columns returns error

Bug 1392450 Internal error 2005 when querying a Hive table with

an unsupported data type

In addition, it changes the following behavior:

- The _SALT_ column now gets added as the last column in the

CREATE TABLE statement, rather than the first column after

SYSKEY. The position of _SALT_ in the clustering key does

not change. This will cause some differences in INVOKE and

in the column number assigned to columns.

- For CREATE TABLE LIKE, the defaults of the WITH clauses

are changing. CREATE TABLE LIKE now copies constraints,

SALT and DIVISION clauses by default. The WITH CONSTRAINTS

clause is now the default and should no longer be used.

Instead, WITHOUT CONSTRAINTS, WITHOUT SALT and WITHOUT

DIVISIONING clauses are supported.

- For CREATE INDEX ... SALT LIKE TABLE, we now give a

warning instead of an error if the table is not salted.

- Also added an optimization for BETWEEN predicates. If

part or all of them can be converted to an equals predicate,

we do this now. Example:

(a,b,c,d) between (1,2,3,4) and (1,2,5,6)

is transformed into

a=1 and b=2 and (c,d) between (3,4) and (5,6).

More detailed description of changes:

- arkcmp/CmoStoredProc.cpp

sqlcat/desc.h

+ other files

Using the new FLAGS column in the COLUMNS metadata table to store

whether a column is a salt or divisioning column. Note that since

there may be existing salted tables without this flag set, the flag

is so far only reliable for divisioning columns.

- comexe/ComTdb.h

comexe/*.h

generator/Generator.cpp

sqlcomp/CmpSeabaseDDLmd.h:

Changed the column class field in struct

ComTdbVirtTableColumnInfo from a string to the corresponding

enum. Sorry, this caused lots of small changes (deleting "_LIT"

from the initializers). Also added the column flags.

- executor/hiveHook.cpp: Added a check for partitioned tables

(having multiple SDs). This is part of the fix for

bug 1353632.

- GenRelUpdate.cpp: When generating the key encoding expression

for an insert inside a MERGE operation, we assumed the new

record expression was in the order of the key columns. Added

a step to sort by key column, so we can pass the expression

in any order.

- optimizer/ItemExpr.cpp

optimizer/ItemNAType.h:

Added a named NATypeToItem item expression.

This is used to do a primitive "bind" operation of an item expression

when processing a DDL statement. Specifically, to bind the DIVISION BY

clause in a CREATE TABLE statement.

- optimizer/ItemFunc.h

optimizer/SynthType.cpp: The DDL time "binder" gets expressions as

they come out of the parser, e.g. a ZZZBinderFunction. Need to add

type synthesis for some cases of the ZZZBinderFunction.

- optimizer/NATable.cpp

Removing some dead code. Adding an error message when we encounter

a Hive column type we can't handle yet. Bug 1392450.

- optimizer/TableDesc.*

Method TableDesc::validateDivisionByClauseForDDL() got moved

to CmpSeabaseDDL::validateDivisionByExprForDDL().

- optimizer/NormItemExpr.cpp

BETWEEN transformation described above.

- optimizer/ValueDesc.cpp

Avoid hard-codeing the "_SALT_" name and adding a comment about

possibility to use the flag in the future.

- parser

Lots of small changes for salt and divisioning option changes.

Simplifying the syntax for salt options somewhat. I think the older

syntax was so complex because it needed to record the starting and

ending position of the divisioning clause, something we don't need

anymore.

- regress: Adding new test

- sqlcomp/CmpDescribe.cpp: Support for describing DIVISION BY clause

and also supporting the new WITHOUT SALT | DIVISION options

for CREATE TABLE LIKE, which relies on the describe feature.

- sqlcomp/CmpSeabaseDDLcommon.cpp

sqlcomp/CmpSeabaseDDL.h

+ Handling the new column flags and making sure they are not

confused with the HBase column flags (e.g. for serialization).

+ Setting the new COLUMNS.FLAGS when writing metadata.

+ Also, writing the computed column text to the TEXT table.

+ For DROP TABLE, unconditionally deleting TEXT rows, since the

table could contain computed columns.

+ When building ColInfoArray, check system column flags, since

system columns can now appear at any position.

+ Add method to "bind" an item expression during DDL processing

without going through the full binder. This replaces any column

reference with a named NATypeToItem node, since all we really

need is the type and the name for unparsing.

+ Method TableDesc::validateDivisionByClauseForDDL() got moved

to CmpSeabaseDDL::validateDivisionByExprForDDL() with some minor

adjustments, since it used to be called on a bound ItemExpr, now

it gets called on something that came out of the parser and went

through the DDL time "binder".

- sqlcomp/CmpSeabaseDDLindex.cpp:

Support for CREATE INDEX ... DIVISION LIKE TABLE. If this is

set, add the division columns in front of the index key, otherwise

don't.

- sqlcomp/CmpSeabaseDDLtable.cpp:

+ Code to make sure column flags and column class is set and propagated.

+ Fix for bug 1385543: Now that we use the TEXT table for computed

column text, we no longer have a length limit. This is true for both

divisioning and salt expressions.

+ When processing the column list in seabaseCreateTable() we have a

bit of a chicken and egg problem: We need the column list to validate

the DIVISION BY expressions, but the DIVISION BY columns need to be part

of the column list. So, we do this a first time without divisioning

columns, then we add those, and produce the final list in a second

iteration.

+ getTextFromMD method now takes a sub-id as an input parameter. That's

the column number for computed column text.

+ read computed column text from the TEXT table. Note: This also needs

to handle older tables where the computed column text is stored in

the default value.

Change-Id: I7c3ebe39a950c1d01f31855bdc92cbb98e5eb275

  1. … 50 more files in changeset.
Native external hbase table access (select, IUD) changes.

-- IUD on external hbase tables is now enabled by default

-- predicates on native hbase tables can now be pushed down to

hbase region server

-- traf varchar col maxlength is now 200K by default,

can be changed by cqd max_character_col_size

-- executor handles column values length greater than 32K during

move to/from JNI

-- error is correctly returned if data retrieved from hbase exceeds expected

max row length

-- hbase column_create function now takes an expression/param as its

column name operand

Change-Id: Ieb3fcabfebaa22008eff2a049fc1e2000e68861e

  1. … 46 more files in changeset.
Merge "fix for bug 1391643"

fix for bug 1391643

-fix for bug 1391643

-rework based on preliminary review by Mike

Change-Id: Icbc4dd6a3ee71c228c2006017030b71508fa0b6f

  1. … 4 more files in changeset.