fullstack2

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Costing and statistics compiler interfaces for UDFs

blueprint cmp-tmudf-compile-time-interface

bug 1433192

This change adds compiler interfaces for UDFs that give information

about statistics of the result table and also a cost estimate. It also

has more code for the upcoming Java UDF feature, retrieving updated

invocation infos and returning them back to the executor/compiler C++

code.

Description of the changes in more detail:

- Addressed remaining review comments from my last checkin,

https://review.trafodion.org/1655

- Make sure that user-generated exceptions during deallocation of

a routine are reported. These happens in the destructor of the

object derived from tmudr::UDR. For Java, we may need a deallocate

method.

- Java and JNI code to serialize the updated UDRInvocationInfo and

UDRPlanInfo object after calling the user code and return them back

through the JNI interface to the calling C++ code.

- The cost method source files had some inline methods defined in

the .cpp file and used an include file that included other .cpp

files. Make didn't pick up changes made in these files. Removed

this code and changed it to regular methods and inlines.

- Replaced some Context * parameters in costing with PlanWorkSpace *,

to be able to get to UDF-related info that's stored in a special

PlanWorkSpace.

- Changed the behavior or isBigMemoryOperator() for TMUDFs. If the

UDF writer specifies the DoP for the UDF invocation, then consider

it a BMO.

- If possible, synthesize the HASH2 partitioning function of a TMUDF's

child as the partitioning function of the UDF. This can be done if

the partitioning key gets passed through the UDF.

- Statistics interface for TMUDFs:

- TMUDF now populates statistics field in the UDRInvocationInfo

object and calls the describeStatistics() method.

- Added an estimated # of partitions for partitioned input tables

of TMUDFs. Also changed row count methods to "estimated" row count.

- Added code to incorporate the information on row count and UEC

provided by the UDF writer into statistics of the TMUDF. This code

is not that suitable for coding it as the default implementation

of describeStatistics(). Therefore, the default implementation of

describeStatistics() does nothing, but the compiler applies some

heuristics in case the UDF writer provides no statistics.

- Changed cost method for TMUDFs to incorporate an estimated cost

per row from the UDF writer. There is no special compiler interface

call to ask for the cost, it can be set from the

describeDesiredDegreeOfParallelism() call and, once supported, from

the describePlanProperties() call. Note that we don't have immediate

plans to support describePlanProperties(), that might come after 2.0.

Patch Set 3: Addressed Dave's review comments.

Patch Set 4: Fixed misplaced copyright in expected file.

Change-Id: Ia9ae076b7ae1fc2968c3d253d6d2d0e1d9a2ea40

  1. … 45 more files in changeset.
various fixes and enhancements, details below.

-- improved DDL performance by not invalidating internal create/alter

operations.

-- added an optimization during CREATE INDEX to not go through

'upsert using load' processing if source table is empty.

-- added support for ISO datetime format (2015-06-01T07:35:20Z)

-- added support for RESET option to ALTER SEQUENCE and IDENTITY.

This will reset generated seq num to the START VALUE.

-- added support for cqd TRAF_STRING_AUTO_TRUNCATE.

If set, strings will be automatically truncated during insert/update.

-- fixed sqlci to pass in correct varchar param len indicator (2 or 4 bytes).

-- changed sizeof(short) to correct vcindlen (2 or 4 bytes)

-- removed some NA_SHADOWCALLS defines

Change-Id: Ie6715435d9c210ae6c2db4ff6bc0545c1b196979

  1. … 37 more files in changeset.
added fullstack2/FILTER062

Change-Id: I2ed813d5674897109fb973124c245cdd1aa789f9

various lp and other fixes, details below.

-- added support for self referencing constraints

-- limit clause can now be specified as a param

(select * from t limit ?)

-- lp 1448261. alter table add identity col is not allowed and now

returns an error

-- error is returned if a specified constraint in an alter/create statement

exists on any table

-- lp 1447343. cannot have more than one identity columns.

-- embedded compiler is now used to get priv info during invoke/showddl.

-- auth info is is not reread if already initialized

-- sequence value function is now cacheable

-- lp 1448257. inserts in volatile table with identity column now work

-- lp 1447346. inserts with identity col default now work if inserted

in a salted table.

-- only one compiler is now needed to process ddl operations with or

without authorization enabled

-- query cache in embedded compiler is now cleared if user id changes

-- pre-created default schema 'SEABASE' can no longer be dropped

-- default schema 'SCH' is automatically created if running regressions

and it doesn't exist.

-- improvements in regressions run.

-- regressions run no longer call a script from another sqlci session

to init auth, create default schema

and insert into defaults table before every regr script

-- switched the order of regression runs

-- updates from review comments.

Change-Id: Ifb96d9c45b7ef60c67aedbeefd40889fb902a131

  1. … 69 more files in changeset.
Updated expected results for fullstack2/TEST062

The release version for fullstack2 test TEST062 was incorrect. Updated

expected results.

Change-Id: I82d46babf621cc20329aa0dc5f34e9c2ec38a560

Miscellaneous DDL and security bug fixes

Fixed a testware issue with fullstack2/TEST062 that occurred during

release testing

Bug 1415196 - Alter volatile table add column cores at CmpSeabaseDDL::alterSeabaseTableAddColumn()

Added a check to not allow add or drop column for volatile tables:

- sqlcomp/CmpSeabaseDDLtable.cpp

Bug 1415232 - A failed create view causes a volatile table to disappear

The code to bind a view does not correctly reset the volatile schema in use

session parameter in case of an error. Subsequent calls do not check for

volatile objects.

parser/BindStmtDDL.cpp

Bug 1371265 - should not allow grants to DB__ROOT or current user

Added a check at grant to prevent this

- sqlcomp/PrivMgrPrivileges.cpp

Bug 1392491 - Unavailability of privmgr metadata error is incomplete

If not all the privmgr metadata is available, then a new Compile context

flag called IS_AUTHORIZATION_READY is set. This flag is adjusted when

a new compiler context is started, and when authorization is enabled and

disabled.

When isAuthorizationEnabled is called and authorization is incomplete,

error 1234 is now returned by default.

After coding changes were added, a request to not check all privmgr metadata

table at context startup was requseted - a performance concern. Fix was

changed to check all tables for debug builds but check only one table for

release builds. If the performance problem is fixed, then we can go back

and check for all privmgr tables.

- arkcmp/CmpContext.h

- arkcmp/CmpContext.cpp

- sqlcomp/CmpSeabaseDDLcommon.cpp

- sqlcomp/nadefaults.cpp

Bug 1402009 - DB__ROOT is unable to grant privilege on object in private schema

When DB__ROOT executes a grant or revoke on objects it does not own, need to

change the grantor from DB__ROOT to the object owner. This matches the same

behavior for other DDL operations such as CREATE.

As part of this fix, the GRANTED BY clause is now allowed for GRANT

statements but it won't be complete until LP bug 1414225 is done.

- sqlcomp/CmpSeabaseDDLtable.cpp (seabaseGrantRevoke)

- sqlcomp/PrivMgrCommands.h

- sqlcomp/PrivMgrCommands.cpp

- sqlcomp/PrivMgrPrivileges.h

- sqlcomp/PrivMgrPrivileges.cpp

- sqlcomp/PrivMgrMD.h

Bug 1414125 - User without priv can view data in metadata tables

Fixes were in place for all metadata tables except the privmgr metadata

tables. The priv information was always being set to none in setupPrivInfo

(NATable) and revoking a privilege was not correctly removing privilege

information from object_privileges.

- optimizer/NATable.cpp

- sqlcomp/PrivMgrCommands.cpp

Bug - create library checking privileges when authorization is not enabled

- CmpSeabaseDDLroutine.cpp

Enhanced the sqlci env command:

- alphabetize the output

- add the following information

-- authentication status

-- authorization status

-- external (LDAP) user connected

A new session parameter called SESSION_EXTERNAL_USER_NAME was added to return

the external user name connected.

A new cli request called SQL_EXEC_GetAuthState_Internal was written to return

the authentication & authorization status. Code was also added, but not yet

supported, for auditing status.

Renamed member/methods that use ldap to external

Changed sqlci env command to return new format

- cli/sqlcli.h

- cli/SQLCLIdev.h

- cli/CliExtern.cpp

- cli/Cli.h

- cli/Cli.cpp

- cli/Context.h

- cli/Context.cpp

- qmscommon/QRQueries.cpp

- sqlci/SqlciEnv.h

- sqlci/SqlciEnv.cpp

- regress/fullstack2/EXPECTED062.SB

- regress/funnstack2/DIFF062.KNOWN.SB.OS

Change-Id: I04627435a0e644c6b14bbf6bd8aa1162d81224fb

  1. … 25 more files in changeset.
Updated expected files for fullstack2/TEST062

The new Jenkins job core-regress-fullstack2-cdh4.4 was failing

because the release flavor expected file for TEST062 was out of date.

Both flavors have been updated, that is, EXPECTED062.SB for debug

and EXPECTED062.SB.RELEASE for release.

Change-Id: Iaabae7ce89f0b1c7a2ac909808d3e1f9090709ea

setting a cqd in sqlci does not have effect in embedded compiler

The changed set of CQDs in a cmp context is not propagated to another

cmp context when a switch takes place. The change ensures that the list

of CQDs that are changed in a cmp context (list kep in ControlDB) are

pushed to the cmp context we switch to. The change also ensures that the

list of reset CQDs are also kep in the ControlDB. The reason we need

this list now is that in case a CQD value was changed and was pushed its

value to a CI in a previous switch, we need to make sure that if the CQD

is reset, the default value is pushed during a switch.

Change-Id: I6519c4c99c086929f482d09759358b9ebf7f2171

Closes-Bug: #1347810

  1. … 10 more files in changeset.
Initial code drop of Trafodion

    • -0
    • +1117
    ./EXPECTED062.SB
    • -0
    • +1119
    ./EXPECTED062.SB.RELEASE
  1. … 4873 more files in changeset.