NARoutine.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
Using the language manager for UDF compiler interface

blueprint cmp-tmudf-compile-time-interface

This change includes new CLI calls, to be used in the compiler to

invoke routines. Right now, only trusted routines are supported,

executed in the same process as the caller, but in the future we may

extend this to isolated routines. Using a CLI call allows us to share

the language manager between compiler and executor, since language

manager resources such as the JVM and loaded DLLs exist only once per

process. This change is in preparation for Java UDFs.

Changes in a bit more detail:

- Added 4 new CLI calls to allocate a routine, invoke it, retrieve

updated invocation and plan infos and deallocate (put) the routine.

The CLI globals now have a C/C++ and a Java language manager that

is allocated on demand.

- The compiler no longer loads a DLL for the UDF compiler interface,

it uses the new CLI calls instead.

- DDL syntax is changed to allow TMUDFs in Java (not officially

supported, so don't use it quite yet).

- TMUDFs in C are no longer supported, only C++ and Java are.

Converted remaining TMUDF tests to C++.

- C++ TMUDFs now do a basic verification at DDL time, so errors

like missing entry points are detected earlier. Validation for

Java TMUDFs is also done through the CLI.

- Make sure we have no memory or resource leaks:

- CmpContext keeps track of UDF-related objects allocated on

system heap and in the CLI, cleaned up at the end of a statement

- CLI keeps a list of allocated trusted routines, cleaned up

when a CLI context is deallocated

- Using ExeCliInterface class to make the new CLI calls (4 new calls

added).

- Removed CmpCli class in the optimizer directory and converted

tracking compiler to use ExeCliInterface as well.

- Compile-time parameter values are no longer baked into the

UDRInvocationInfo. Instead, they are provided as an input row, the

same way as they are provided at runtime.

- Bug fixes in C++ UDR code, mostly related to serialization and

to multiple interactions with the UDF through serialized objects.

- Added more info to UDRInvocationInfo (SQL access type, etc.).

- Since there are multiple plans per invocation, each of which

can have multiple interactions with the UDF, plans need to be

numbered so the UDF side can tell them apart to attach the

right state (owned by the UDF) to it.

- The language manager needs some functions that are provided by

the process it's running in. Added those (empty, for now) functions

as cli/CliImplLmExtFunc.cpp.

- Added a new class for Java TMUDFs, LmRoutineJavaObj. Added methods

to allocate such routines and to load their class as well as to

create Java objects by invoking the default constructor through JNI.

- Java TMUDFs use the new UDR interface (to be provided by Suresh and

Pavani). In the language manager, the container is the class of

the UDF, the external path is the fully qualified jar name. The

Java method name is <init>, the default constructor, with signature

"()V". Some code changes were required to do this.

- Created a new directory trafodion/core/sql/src for Java sources in

the sql engine. Right now, only language manager java

sources are in this directory, but I am planning to move the other

java sources under sql in a future checkin. Suresh and Pavani

will add their UDF-related Java files there as well.

- Renamed the udr jar to trafodion-sql-<version>.jar, in anticipation

of combining all the sql Java sources into this jar.

- Created a maven project file trafodion/core/sql/pom.xml and

changed makefiles to invoke maven to build java sources.

- More work to separate new UDR interface from older SPInfo object,

so that we can get rid of SPInfo if/when we don't support the older

style anymore.

- Small fix to odb makefile, make clean failed when executed twice.

Patch set 2: Adding a custom filter for test regress/udr/TEST108.

Change-Id: Ic827a42ac25505fb1ee451b79636c0f9349d8841

  1. … 98 more files in changeset.
metadata fixes and 'sqlmp' code cleanup

-- NATable struct for metadata was being created multiple

times whenever information for a new table was read

from metadata. That has been fixed.

-- an 'initialize trafodion, drop' followed by 'initialize traf'

from the same session was failing due to priv info not getting

reset. This would show up if 'initialize authorization' was

done earlier. That has been fixed.

-- code cleanup mostly related to sqlmp legacy code and reference.

Change-Id: I346e3f3bbc6c7784b38e7e2e1f11d487854c281c

  1. … 54 more files in changeset.
Fixed catman1 TEST137 failure

And some minor changes from a previous review.

Change-Id: I01319b40e3ddf1bdba88c70d5a4b62057698491e

  1. … 6 more files in changeset.
SHOWDDL, QUERY Cancel, rework

This delivery addresses security issues with SHOWDDL, adds initial

support for security in query cancel, and implements part of the

proposed GIVE commands.

Bug 1414234: SHOWDDL command now check component privileges.

SHOW is granted to PUBLIC by default, so effectively there are

no new restrictions unless SHOW is revoked from PUBLIC.

SHOWDDL COMPONENT now checks for MANAGE_COMPONENTS or SHOW privilege.

SHOWDDL ROLE now checks for MANAGE_ROLES or SHOW privilege.

SHOWDDL SCHEMA now checks for SHOW privilege.

SHOWDDL USER now checks for MANAGE_USERS or SHOW privilege.

SHOWDDL LIBRARY is implemented. A user must have the USAGE

privilege on the library, or the MANAGE_LIBRARY or SHOW privilege.

New function to determine if the user canceling the query has

the authority: either DB__ROOT, or the user owns the query, or

the user has the QUERY_CANCEL privilege. Note, the code is

delivered in an inactive state pending future integration.

Three new component privileges are added: QUERY_ACTIVATE,

QUERY_CANCEL, and QUERY_SUSPEND. These will be added if

authorization is dropped and reinitialized. A future

delivery will add an INITIALIZE AUTHORIZATION,UPDATE

command that will add these privileges to an existing

instance with authorization enabled.

Support for library objects was added to NATable, but the code

is currently not used. May be integrated into CREATE ROUTINE

and GRANT for libraries in the future.

Also included is minor rework from delivery 1082, and the

GIVE SCHEMA command now updates associated privileges when object

ownership is changed. Note, GIVE commands are still prototype.

A detailed blueprint for GIVE will be released shortly.

This patch merges with changes from 1177 and addresses a couple of

minor comments from the initial submittal.

Change-Id: I60419228f886555ed0e066441bb824c5246ee498

  1. … 28 more files in changeset.
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

  1. … 39 more files in changeset.
Specify compiler context when querying natable virtual table interface

select * from table(natablecache('ALL','local')) ;

select * from table(natablecacheentries('meta','local')) ;

select * from table(naroutinecache('user','local')) ;

Following the querycache virtual table interface, now the natable and

naroutine cache virtual table interfaces also support specifying the

name of the context we want to query. The first parameter can be 'all' or

name of context (e.g. 'meta', 'user', 'ustat'). The second parameter

can be local or remote. Parameters are case insensitive.

For remote compiler we only query the context pointed to by activeschemadb.

The column num_entries has been added to natablecache virtual table.

This change can be used to monitor memory growth in these caches.

Patch Set 2:

Address issues found by Dave. Changes in 4 files

sql/arkcmp/NATableSt.cpp

sql/arkcmp/NATableSt.h

sql/arkcmp/QueryCacheSt.cpp

sql/optimizer/NARoutine.cpp

Changes cover

A minor leak in in HQCIterator

returning FALSE in NATable/RoutineCacheStats iterator getNext methods

Change-Id: Icf15c93b9ae3c3f523d0abe1580ce7280c5b0d84

  1. … 20 more files in changeset.
TMUDF C++ compiler interface, part of log-reading TMUDF

This is the infrastructure for a new C++ interface for TMUDFs

(table-mapping UDFs). It is used by a new log-reading TMUDF that

is not yet complete, but should be finished in the next few days.

See blueprint cmp-tmudf-compile-time-interface for more info.

Change-Id: I5a74e461462313b6d9722ac0deb21cd16c4b02ce

  1. … 55 more files in changeset.
Security changes to support authorization

Added support for authorization commands:

- initialize authorization [, drop]

- create/drop roles

- register/unregister components

- create/drop component operations

- grant/revoke object privileges

- grant/revoke role privileges

- grant/revoke component privileges

- updates to GET and SHOWDDL statements

- checking of privileges for DML requests

- checking of privileges for DDL requests

- regression tests added to catman1 library

Fixed a testware problem in catman1 TEST135 and TEST139

Fixed a parser problem introduced by compGeneral/TESTTOK2 which was recently

introduced.

More details:

This delivery was part of code worked on by many people for several

months on a remote branch. This team held bi-weekly meetings

for several months to design and implement these features. These

meetings also included extensive code reviews.

The security features which include authentication (which was delivered

in June) and authorization is turned off by default. The

traf_authentication_setup script located in $MY_SQROOT/sql/scripts needs

to be run to enable both authentication and authorization. This

procedure is described on the Trafodion Twiki page and will be updated once this

delivery completed to include authorzation.

Delivery updates:

Updated traf_authentication_setup to return consistent error messages

and added a comment to ComSmallDefs.h to address a buf size issue for

metadata tables.

Change-Id: I896f1ee006590284653b2c9882901c05b5f2ba22

  1. … 100 more files in changeset.
metadata struct updates and fixes.

-- second arkcmp is no longer started to read hist tables.

-- metadata index change got lost in a previous change. That index has been added.

Will require a metadata upgrade from v2.2 to v2.3

-- metadata structs cleanup (no functionality change).

-- object_owner correctly set for indexes.

Change-Id: I78e06c70b67661b2881724b10ede8f6c188c65db

  1. … 16 more files in changeset.
Initial code drop of Trafodion

    • -0
    • +1181
    ./NARoutine.cpp
  1. … 4886 more files in changeset.