regress

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Final move into core sub-tree for restructured repo

  1. … 2 more files in changeset.
Added a hive test to core

Hive tests fail at times since there is no test that is part of the core

suite and sometimes people miss running them. So added one test so it's

part of the gate tests. It's a very short test that runs in a few

seconds. The Hive suite should be runfor more comprehensive coverage.

Change-Id: I75862841e52a71583632408fd06f6cc5fd6d83ac

    • -0
    • +1602
    ./core/EXPECTED162
Merge remote branch 'core/master'

  1. … 107 more files in changeset.
Added privilege manager upgrade support

Recently, column level privileges has been delivered which requires new

metadata tables. This delivery changes Trafodion upgrade to create the

new metadata tables in preparation for a future upgrade in release 2.0.

The version number has not changed.

Also removed two obsolete scripts.

Change-Id: I6d604640d76c2be72bab81eabb968874c01fec7a

  1. … 13 more files in changeset.
initial support for returning multiple versions and column timestamps

This feature is not yet externalized.

Support added to:

-- return multiple versions of rows

-- select * from table {versions N | MAX | ALL}

-- get hbase timestamp of a column

-- select hbase_timestamp(col) from t;

-- get version number of a column.

-- select hbase_version(col) from t

Change-Id: I37921681fc606a22c19d2c0cb87a35dee5491e1e

  1. … 45 more files in changeset.
Fix for LP 1460771

LP 1460771 descrbes a data corruption problem that occurs when upserting into

a table with an index. Fix suggested by Anoop is to transform the upsert

statement into a merge statement. The transformation shown below in now done

in the binder, where the upsert statement provided is changed internally

to the corresponding merge statement.

prepare s1 from upsert into test1 values (1,1) ;

prepare s2 from merge into test1 on a = 1

when matched then update set b = 1

when not matched then insert values (1,1) ;

prepare s1 from upsert into test1 select * from test2 ;

prepare s2 from merge into test1 using (select * from test2) z(a,b) on

test1.a = z.a

when matched then update set b = z.b

when not matched then insert values (z.a,z.b) ;

prepare s1 from upsert into test1 values (1,1),(2,2) ;

prepare s2 from merge into test1 using (values (1,1),(2,2)) z(a,b) on

test1.a = z.a

when matched then update set b = z.b

when not matched then insert values (z.a,z.b) ;

Existing merge statement had an issue data corruption on index table when a

veg is not formed between source value and key column on target side. This

occurs when the source is a tuplelist or something like a valueidUnion. This

has been addressed by breaking VEG formation between old and new values when

a merge is done into a table with an index.

---Patchset 2

Address review comments. This entire fix was possible due to help from Anoop,

Hans and Prashanth. Thank you.

--Patchset 3

I am sorry I forgot to address the comment about system columns. Indeed with

patchset 2 I was not able to upsert into a salted table with index. This is now

addressed as suggested.

Change-Id: I778a7c431e993c8b54dbef6b40b44b09d6cc9f8e

  1. … 2 more files in changeset.
Fixes from review to sqvers

commit 04f3812f112a5629a563f02d7e72c5fa503c6a8d

Author: Sandhya Sundaresan <sandhya.sundaresan@hp.com>

Date: Sun Jun 14 04:23:21 2015 +0000

Preliminary checkin of lob support for external files. Inserts from http

files, hdfs files and lob local files are supported. Added support for

new extract synttax. Extract from lob

columns to hdfs files has been added . More work needed to support

binary files and very large files . Current limit is 1G.

Also fixed some error handling issues

Fixed some substring warning issues in the lobtostring/stringtolob

functions.

Added references and interfaces to curl library that is needed to read external http

files.

More work needed before this support can be used

Change-Id: Ieacaa3e4b7fa2a040764888c90ef0b029f107a8b

Change-Id: Ife3caf13041f8106a999d06808b69e5b6a348a6b

  1. … 29 more files in changeset.
Merge remote branch 'core/master'

    • -41
    • +0
    ./newregr/card/DTESTDML15.known.LINUX
    • -42213
    • +0
    ./newregr/card/ETESTDDL04.LINUX
  1. … 2803 more files in changeset.
Migrate from log4cpp to log4cxx

This change is a wholesale removal of log4cpp from source tree.

log4cxx is an external library installed via RPM, or user build

to default /usr/lib64 and /usr/include directories. Some of the

QRLogger and CommonLogger code was changed to use the new log4cxx

APIs.

Change-Id: I248bac0a8ffbfea6cbc1ba847867b30638892eae

    • -10
    • +10
    ./newregr/mvqr/log4cpp.mvqr.config
    • -15
    • +15
    ./newregr/mvqr/log4cpp.qmm.config
    • -15
    • +15
    ./newregr/mvqr/log4cpp.qmp.config
    • -15
    • +15
    ./newregr/mvqr/log4cpp.qms.config
  1. … 205 more files in changeset.
Fix for intermittent failure in core/TESTRTS

FILTERRTS can now tolerate the variance in the statistics for

EX_TRAF_KEY_SELECT operator

Change-Id: Iccd2f15bbc320fbfc8f4b2fbbf6e7642efb3ac5d

Fixes for 1450193, 1370754, and performance change

1450193 - GRANT syntax for SHOWDDL output incorrect

1370754 - Objects with delimited names not displayed correctly for SHOWDDL

Made sure names in GRANT statements were qualified correctly and added the

object type, when required. Also changed SHOWDDL commands so they don't

start a tdm_arkcmp to get data details with the except of showddl user and

showddl role.

Performance: changed the default in PrivMgr requests to assume that

authorization is enabled. The PrivMgr code that checks to see if authorization

is enabled causes a new tdm_arkcmp to be started. Since all callers already

check authorization from a cached value before calling PrivMgr, this avoids the

tdm_arkcmp process overhead.

Patch set 2:

Updated copyright

Patch set 3:

Fixed problem in isDelimited method.

Patch set 4:

Fixed another problem in isDelimited method.

Change-Id: I43ff421ddd8e9c508c8fed79a1e2b2c6448456b7

  1. … 8 more files in changeset.
Copyright corrections

Put correct form of Apache License into many source files.

The tdm_odbcSrvrMsg.mc file is used on Windows, and is

treated as binary on Linux. However, it is visible in

Linux with "cat", and editable with "notepad" on Windows.

In patch set 2, updated sql/regress/newregr by deleting some

unused files, updating copyrights in scripts, and removing

copyrights from front of expected test results. This is not

a final list of the changes needed, since the test suites in

the "newregr" directory need more work before they run correctly.

Change-Id: If0ea5331b822764191c839c38dfd84218612fe7a

    • -17
    • +18
    ./newregr/exeperf/setup_exeperf.ksh
    • -29
    • +13
    ./newregr/mvqr/CropDescriptor.ksh
    • -4139
    • +0
    ./newregr/mvqr/EXPECTED001.NT
    • -13385
    • +0
    ./newregr/mvqr/EXPECTED004.NT
  1. … 319 more files in changeset.
Security bug fixes

Corrects a number of bugs in security and SQL DDL.

1. 1439316: Internal error when creating view with sequence.

2. 1441825: User in role-owned schema is not given grant option.

3. 1447328: DB__ROOT required to revoke component privileges

by other users.

4. 1447330: DB__ROOT required to revoke object and

column privileges granted by other users.

In addition, Launchpad bugs 1350627 and 1438886 were tested and

found to no longer be a defect.

Two new component-level privileges were added, MANAGE and

MANAGE_PRIVILEGES.

Users who have the MANAGE_PRIVILEGE component-level privilege can

grant and revoke privileges at the schema, object, and column

level on behalf of other users and roles. Users who have the

MANAGE_PRIVILEGE component-level privilege can also grant

"WITH GRANT OPTION" any privilege they have.

A user granted the MANAGE privilege has all the MANAGE_* privileges,

which includes MANAGE_COMPONENTS, MANAGE_LIBRARY, MANAGE_LOAD,

MANAGE_PRIVILEGES, MANAGE_ROLES, MANAGE_STATISTICS, and MANAGE_USERS.

Change-Id: I10873de95f75114b937e1144f3af9aba76884eae

  1. … 6 more files in changeset.
drop column fix.

We currently dont support drop of a column which has an

index (implicit or explicit) defined on it.

But drop column was also not working in cases where a secondary

index was present on a column that was not being dropped.

That has been fixed.

Change-Id: I53264b630794bef5c99ffb7a57b8fd2ba0f13bb5

  1. … 1 more file in changeset.
Removed about 50 obsolete make files from sql

Security is not built inside sql, and sqlutils is not in Trafodion, so

their makefiles are not needed and have been deleted. Also deleted a few

other unused makefiles. In modified makefiles, removed unused parts.

The sql/nskgmake/Makerules* files still have references to sqlutils, those

lines will be considered in a separate change.

In second patch set, deleted unused directory sql/nskgmake/exeindp2

In third patch, added missing copyrights in

sql/executor/OrcFileReader.java

sql/regress/udr/TEST101.java

and deleted empty file

sql/parser/StmtDDLAlterTableAlterColumnRecalibrateSG.h

Change-Id: Id9ab492d4a23579cd0ab6db7279e0da56c55ea4c

  1. … 65 more files in changeset.
Costing and statistics compiler interfaces for UDFs

blueprint cmp-tmudf-compile-time-interface

bug 1433192

This change adds compiler interfaces for UDFs that give information

about statistics of the result table and also a cost estimate. It also

has more code for the upcoming Java UDF feature, retrieving updated

invocation infos and returning them back to the executor/compiler C++

code.

Description of the changes in more detail:

- Addressed remaining review comments from my last checkin,

https://review.trafodion.org/1655

- Make sure that user-generated exceptions during deallocation of

a routine are reported. These happens in the destructor of the

object derived from tmudr::UDR. For Java, we may need a deallocate

method.

- Java and JNI code to serialize the updated UDRInvocationInfo and

UDRPlanInfo object after calling the user code and return them back

through the JNI interface to the calling C++ code.

- The cost method source files had some inline methods defined in

the .cpp file and used an include file that included other .cpp

files. Make didn't pick up changes made in these files. Removed

this code and changed it to regular methods and inlines.

- Replaced some Context * parameters in costing with PlanWorkSpace *,

to be able to get to UDF-related info that's stored in a special

PlanWorkSpace.

- Changed the behavior or isBigMemoryOperator() for TMUDFs. If the

UDF writer specifies the DoP for the UDF invocation, then consider

it a BMO.

- If possible, synthesize the HASH2 partitioning function of a TMUDF's

child as the partitioning function of the UDF. This can be done if

the partitioning key gets passed through the UDF.

- Statistics interface for TMUDFs:

- TMUDF now populates statistics field in the UDRInvocationInfo

object and calls the describeStatistics() method.

- Added an estimated # of partitions for partitioned input tables

of TMUDFs. Also changed row count methods to "estimated" row count.

- Added code to incorporate the information on row count and UEC

provided by the UDF writer into statistics of the TMUDF. This code

is not that suitable for coding it as the default implementation

of describeStatistics(). Therefore, the default implementation of

describeStatistics() does nothing, but the compiler applies some

heuristics in case the UDF writer provides no statistics.

- Changed cost method for TMUDFs to incorporate an estimated cost

per row from the UDF writer. There is no special compiler interface

call to ask for the cost, it can be set from the

describeDesiredDegreeOfParallelism() call and, once supported, from

the describePlanProperties() call. Note that we don't have immediate

plans to support describePlanProperties(), that might come after 2.0.

Patch Set 3: Addressed Dave's review comments.

Patch Set 4: Fixed misplaced copyright in expected file.

Change-Id: Ia9ae076b7ae1fc2968c3d253d6d2d0e1d9a2ea40

    • -18
    • +0
    ./fullstack2/EXPECTED062.SB.RELEASE
  1. … 43 more files in changeset.
various fixes and enhancements, details below.

-- improved DDL performance by not invalidating internal create/alter

operations.

-- added an optimization during CREATE INDEX to not go through

'upsert using load' processing if source table is empty.

-- added support for ISO datetime format (2015-06-01T07:35:20Z)

-- added support for RESET option to ALTER SEQUENCE and IDENTITY.

This will reset generated seq num to the START VALUE.

-- added support for cqd TRAF_STRING_AUTO_TRUNCATE.

If set, strings will be automatically truncated during insert/update.

-- fixed sqlci to pass in correct varchar param len indicator (2 or 4 bytes).

-- changed sizeof(short) to correct vcindlen (2 or 4 bytes)

-- removed some NA_SHADOWCALLS defines

Change-Id: Ie6715435d9c210ae6c2db4ff6bc0545c1b196979

    • -0
    • +21
    ./fullstack2/EXPECTED062.SB.RELEASE
  1. … 25 more files in changeset.
added regressions tests, not yet enabled.

Change-Id: I197db96c2e6942b3428b3ca779a0bbbdc1c22525

    • -0
    • +23
    ./newregr/card/DTESTDML04.KNOWN
    • -0
    • +51
    ./newregr/card/DTESTDML06.known
    • -0
    • +79
    ./newregr/card/DTESTDML07.known
    • -0
    • +45
    ./newregr/card/DTESTDML08.known
    • -0
    • +41
    ./newregr/card/DTESTDML09.known
    • -0
    • +21
    ./newregr/card/DTESTDML10.known
    • -0
    • +21
    ./newregr/card/DTESTDML13.known
    • -0
    • +41
    ./newregr/card/DTESTDML15.known
    • -0
    • +41
    ./newregr/card/DTESTDML15.known.LINUX
    • -0
    • +316
    ./newregr/card/ETESTDDL01
    • -0
    • +307
    ./newregr/card/ETESTDDL02
    • -0
    • +41
    ./newregr/card/ETESTDDL03
    • -0
    • +43
    ./newregr/card/ETESTDDL03.LINUX
    • -0
    • +42213
    ./newregr/card/ETESTDDL04
    • -0
    • +42213
    ./newregr/card/ETESTDDL04.LINUX
  1. … 1023 more files in changeset.
Move core into subdir to combine repos

  1. … 10608 more files in changeset.
Column-level privileges - part 2

Support for column-level privileges will be in multiple deliveries.

This delivery add the following portions:

1. DML operations (SELECT, INSERT, UPDATE) now recognize granted

column-level privileges.

2. CREATE VIEW now recognizes granted column-level privileges.

3. Revoke of object-level privileges now revokes the corresponding

column-level privilege.

Missing functionality:

1. Privileges can be granted to roles and revoked from roles, but

REVOKE ROLE does not consider column-level privileges when

determining

if an object depends on a role's granted privileges.

2. Column-level revoke does not enforce RESTRICT, i.e., privileges

may be revoked even if there are dependent privileges.

3. ALTER TABLE DROP COLUMN does not remove associated column-level

privileges, nor does it check for dependent objects.

Change-Id: Ieba04c77edb945dfeb1994e9949b54072289465e

  1. … 12 more files in changeset.
Turned on privilege features, reorg'd PrivMgr code

Turned on GRANTED BY, WITH GRANT OPTION, GRANT OPTION FOR clauses

For GRANTED BY -> showddl now displays the GRANTED BY clause when

--> the current user is not the object owner and

--> the current user is not DB ROOT

added object_owner and schema_owner to the SeabaseLibraryDesc

ComTdb.h

Generator.cpp

desc.h

CmpSeabaseDDLtable.cpp ->getSeabaseLibraryDesc

added object owner in calls to PrivMgrPrivileges::getPrivTextForObject

CmpDescribe.cpp

PrivMgrCommands.cpp/.h

PrivMgrPrivileges.cpp/.h

Changed object grant and revoke to store the list of privileges associated

with the object and columns in the PrivMgrPrivileges class.

--> added new methods generateObjectRowList and generateColumnRowList,

changed the destructor to remove these lists, changed code to call

these new methods, and removed extra I/Os

--> removed member trafMetadataLocation_ (it is already stored in parent)

For WITH GRANT OPTION clause at GRANT time:

--> added checks at grant time to:

--> check for potential circular grants (error 1036)

--> added new method getTreeOfGrants to get list of grantors

that have previously granted to the current grantee

For GRANT OPTION FOR clause at REVOKE time:

--> changed error messages returned to be more meaningful

--> moved and activated call checkRevokeRestrict after call to

getAffectedObjects

Added new columns to the COLUMN_PRIVILEGES and SCHEMA_PRIVILEGES tables

to include the object_name, grantor_name, and grantee_name to match

OBJECT_PRIVILEGES, ROLE_USAGES, and COMPONENT_PRIVILEGES;

Reorganized the contents of PrivMgr files:

--> PrivMgr document exists that describes the .h/.cpp structure

--> Added new files PrivMgr.h/PrivMgr.cpp that describes the parent

class for all PrivMgr requests

--> moved existing defines, classes, etc around to match the PrivMgr

document

Fixed a couple of issues:

--> Fixed a bug in initialize authorization where the WGO was not set up

correctly for UDR's

--> Fixed a bug in PrivMgrObject::selectAllWhere where an error condition

was not returned

--> Fixed a bug in seabaseGrantRevoke where the incorrect object type was

sent for views

--> Fixed a bug in update statistics privilege checking that was not

handling HBase tables correctly

Added two regression tests (skipped until catman1 test directory is split up)

--> TEST132 - tests for privilege checking on libraries, populate index,

showddl, invoke, update statistics, and showstats

--> TEST140 - tests for WITH GRANT OPTION and GRANTED by option

Fixed expected result for catman1/TEST137

Change-Id: Iaf523aef763b0bce2101fedae0ee701606c369c7

    • -0
    • +2214
    ./catman1/EXPECTED132
    • -0
    • +926
    ./catman1/EXPECTED140
    • -0
    • +444
    ./catman1/TEST132
  1. … 37 more files in changeset.
Changes in Patchset2

Fixed issues found during review.

Most of the changes are related to disbling this change for unique indexes.

When a unique index is found, they alone are disabled during the load.

Other indexes are online and are handled as described below. Once the base

table and regular indexes have been loaded, unique indexes are loaded from

scratch using a new command "populate all unique indexes on <tab-name>".

A simlilar command "alter table <tab-name> disable all unique indexes"

is used to disable all unique indexes on a table at the start of load.

Cqd change setting allow_incompatible_assignment is unrelated and fixes an

issue related to loading timestamp types from hive.

Odb change gets rid of minor warnings.

Thanks to all three reviewers for their helpful comments.

-----------------------------------

Adding support for incremental index maintenance during bulk load.

Previously when bulk loading into a table with indexes, the indexes are first

disabled, base table is loaded and then the indexes are populated from

scratch one by one. This could take a long time when the table has significant

data prior to the load.

Using a design by Hans this change allows indexes to be loaded in the same

query tree as the base table. The query tree looks like this

Root

|

NestedJoin

/ \

Sort Traf_load_prep (into index1)

|

Exchange

|

NestedJoin

/ \

Sort Traf_load_prep (i.e. bulk insert) (into base table)

|

Exchange

|

Hive scan

This design and change set allows multiple indexes to be on the same tree.

Only one index is shown here for simplicity. LOAD CLEANUP and LOAD COMPLETE

statements also now perform these tasks for the base table along with all

enabled indexes

This change is enabled by default. If a table has indexes it will be

incrementally maintained during bulk load.

The WITH NO POPULATE INDEX option has been removed

A new option WITH REBUILD INDEXES has been added. With this option we get

the old behaviour of disabling all indexes before load into table and

then populate all of them from scratch.

Change-Id: Ib5491649e753b81e573d96dfe438c2cf8481ceca

  1. … 32 more files in changeset.
RMS and other related changes

The process level statistics at ESP level can now be obtained

using RMS while the query is running. The SQL command is

select * from table(statistics(null, 'QID=<qid>,DETAIL=1'))

To get a list of ESPs processes participating in the query execution, issue

select distinct tdb_id, text process_name from

table(statistics(null, 'QID=<qid>,DETAIL=1')) order by tdb_id

Explain information can now be obtained using RMS. To do this

CQD EXPLAIN_IN_RMS 'ON' should be issued in the session before the

query is prepared.

EXPLAIN OPTIONS 'F' FOR QID <qid> from a different session can give

the explain info.

TESTRTS is now incorporated into core test suite.

AQR-ed queries were not getting garbage collected from RMS shared segment.

This has been fixed.

Missing code in IpcGuardinServer::serverDied method is now added.

Change-Id: Ib591ee5c9251ab6778e3dec9450f5b0466041e9b

    • -0
    • +3669
    ./core/EXPECTEDRTS
  1. … 8 more files in changeset.
Stop sending user CQDs at embedded compile switch

Do not send user entered CQDs when switching to embedded META CmpContext

to compile metadata queries.

Also do the holdAndSet needed CQDs only when the embedded (META)

CmpContext is first switched to. These CQDs won't be restored when

switching back so that we won't send the same set of CQDs again next

time the CmpContext is used.

LP bug 1438372, Compiler returns internal error from

../optimizer/opt.cpp line 6907

LP bug 1444739, compGeneral/TEST006 failed with tmp fix for bug 1438372

Change-Id: I92398aeeaf53820d3f8af1fb73e9c1d6f449ce5e

  1. … 6 more files in changeset.
Fix for 1452424 vsbb scan/delete cause query to return wrong result

VSBB update/delete were not tracking the number of rows in the

buffer.This has been corrected.

Change-Id: I2a89ccd9a84832c4771481de2ee8503e912ce0d8

  1. … 1 more file in changeset.
Fix 1455585: Error 1234 returned trying to run SQL

If code is refreshed with https://review.trafodion.org/1635 (which adds the

first phase of column level privileges) authorization is enabled, and running

in debug mode; error 1234 is returned and you are not able to drop or

initialize authorization. SQL requests fail.

This delivery fixes the issue. First it allow operations to proceed even if

there are missing privilege manager tables. Second, we have turned off checks

for column level privileges (can be turned on by setting cqd CAT_TEST_BOOL).

Lastly, we do not run the column privileges regression test.

Change-Id: Ic427c4e9413b6c7208313c0e0755ca6aabd8a2cd

  1. … 2 more files in changeset.
added fullstack2/FILTER062

Change-Id: I2ed813d5674897109fb973124c245cdd1aa789f9

    • -0
    • +43
    ./fullstack2/FILTER062
various lp and other fixes, details below.

-- added support for self referencing constraints

-- limit clause can now be specified as a param

(select * from t limit ?)

-- lp 1448261. alter table add identity col is not allowed and now

returns an error

-- error is returned if a specified constraint in an alter/create statement

exists on any table

-- lp 1447343. cannot have more than one identity columns.

-- embedded compiler is now used to get priv info during invoke/showddl.

-- auth info is is not reread if already initialized

-- sequence value function is now cacheable

-- lp 1448257. inserts in volatile table with identity column now work

-- lp 1447346. inserts with identity col default now work if inserted

in a salted table.

-- only one compiler is now needed to process ddl operations with or

without authorization enabled

-- query cache in embedded compiler is now cleared if user id changes

-- pre-created default schema 'SEABASE' can no longer be dropped

-- default schema 'SCH' is automatically created if running regressions

and it doesn't exist.

-- improvements in regressions run.

-- regressions run no longer call a script from another sqlci session

to init auth, create default schema

and insert into defaults table before every regr script

-- switched the order of regression runs

-- updates from review comments.

Change-Id: Ifb96d9c45b7ef60c67aedbeefd40889fb902a131

  1. … 55 more files in changeset.
Enabling Bulk load and Hive Scan error logging/skip feature

Also Fixed the hanging issue with Hive scan (ExHdfsScan operator) when there

is an error in data conversion.

ExHbaseAccessBulkLoadPrepSQTcb was not releasing all the resources when there

is an error or when the last buffer had some rows.

Error logging/skip feature can be enabled in

hive scan using CQDs and in bulk load using the command line options.

For Hive Scan

CQD TRAF_LOAD_CONTINUE_ON_ERROR ‘ON’ to skip errors

CQD TRAF_LOAD_LOG_ERROR_ROWS ‘ON’ to log the error rows in Hdfs files.

For Bulk load

LOAD WITH CONTINUE ON ERROR [TO <location>] – to skip error rows

LOAD WITH LOG ERROR ROWS – to log the error rows in hdfs files.

The default parent error logging directory in hdfs is /bulkload/logs. The error

rows are logged in subdirectory ERR_<date>_<time>. A separate hdfs file is

created for every process/operator involved in the bulk load in this directory.

Error rows in hive scan are logged in

<sourceHiveTableName>_hive_scan_err_<inst_id>

Error rows in bulk upsert are logged in

<destTrafTableName>_traf_upsert_err_<inst_id>

Bulk load can also aborted after a certain number of error rows are seen using

LOAD WITH LOG ERROR ROWS, STOP AFTER <n> ERROR ROWS option

Change-Id: Ief44ebb9ff74b0cef2587705158094165fca07d3

  1. … 32 more files in changeset.
Using the language manager for UDF compiler interface

blueprint cmp-tmudf-compile-time-interface

This change includes new CLI calls, to be used in the compiler to

invoke routines. Right now, only trusted routines are supported,

executed in the same process as the caller, but in the future we may

extend this to isolated routines. Using a CLI call allows us to share

the language manager between compiler and executor, since language

manager resources such as the JVM and loaded DLLs exist only once per

process. This change is in preparation for Java UDFs.

Changes in a bit more detail:

- Added 4 new CLI calls to allocate a routine, invoke it, retrieve

updated invocation and plan infos and deallocate (put) the routine.

The CLI globals now have a C/C++ and a Java language manager that

is allocated on demand.

- The compiler no longer loads a DLL for the UDF compiler interface,

it uses the new CLI calls instead.

- DDL syntax is changed to allow TMUDFs in Java (not officially

supported, so don't use it quite yet).

- TMUDFs in C are no longer supported, only C++ and Java are.

Converted remaining TMUDF tests to C++.

- C++ TMUDFs now do a basic verification at DDL time, so errors

like missing entry points are detected earlier. Validation for

Java TMUDFs is also done through the CLI.

- Make sure we have no memory or resource leaks:

- CmpContext keeps track of UDF-related objects allocated on

system heap and in the CLI, cleaned up at the end of a statement

- CLI keeps a list of allocated trusted routines, cleaned up

when a CLI context is deallocated

- Using ExeCliInterface class to make the new CLI calls (4 new calls

added).

- Removed CmpCli class in the optimizer directory and converted

tracking compiler to use ExeCliInterface as well.

- Compile-time parameter values are no longer baked into the

UDRInvocationInfo. Instead, they are provided as an input row, the

same way as they are provided at runtime.

- Bug fixes in C++ UDR code, mostly related to serialization and

to multiple interactions with the UDF through serialized objects.

- Added more info to UDRInvocationInfo (SQL access type, etc.).

- Since there are multiple plans per invocation, each of which

can have multiple interactions with the UDF, plans need to be

numbered so the UDF side can tell them apart to attach the

right state (owned by the UDF) to it.

- The language manager needs some functions that are provided by

the process it's running in. Added those (empty, for now) functions

as cli/CliImplLmExtFunc.cpp.

- Added a new class for Java TMUDFs, LmRoutineJavaObj. Added methods

to allocate such routines and to load their class as well as to

create Java objects by invoking the default constructor through JNI.

- Java TMUDFs use the new UDR interface (to be provided by Suresh and

Pavani). In the language manager, the container is the class of

the UDF, the external path is the fully qualified jar name. The

Java method name is <init>, the default constructor, with signature

"()V". Some code changes were required to do this.

- Created a new directory trafodion/core/sql/src for Java sources in

the sql engine. Right now, only language manager java

sources are in this directory, but I am planning to move the other

java sources under sql in a future checkin. Suresh and Pavani

will add their UDF-related Java files there as well.

- Renamed the udr jar to trafodion-sql-<version>.jar, in anticipation

of combining all the sql Java sources into this jar.

- Created a maven project file trafodion/core/sql/pom.xml and

changed makefiles to invoke maven to build java sources.

- More work to separate new UDR interface from older SPInfo object,

so that we can get rid of SPInfo if/when we don't support the older

style anymore.

- Small fix to odb makefile, make clean failed when executed twice.

Patch set 2: Adding a custom filter for test regress/udr/TEST108.

Change-Id: Ic827a42ac25505fb1ee451b79636c0f9349d8841

    • -0
    • +52
    ./catman1/udrtest135.cpp
  1. … 84 more files in changeset.