GenExpGenerator.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
initial support for returning multiple versions and column timestamps

This feature is not yet externalized.

Support added to:

-- return multiple versions of rows

-- select * from table {versions N | MAX | ALL}

-- get hbase timestamp of a column

-- select hbase_timestamp(col) from t;

-- get version number of a column.

-- select hbase_version(col) from t

Change-Id: I37921681fc606a22c19d2c0cb87a35dee5491e1e

  1. … 48 more files in changeset.
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
metadata fixes and 'sqlmp' code cleanup

-- NATable struct for metadata was being created multiple

times whenever information for a new table was read

from metadata. That has been fixed.

-- an 'initialize trafodion, drop' followed by 'initialize traf'

from the same session was failing due to priv info not getting

reset. This would show up if 'initialize authorization' was

done earlier. That has been fixed.

-- code cleanup mostly related to sqlmp legacy code and reference.

Change-Id: I346e3f3bbc6c7784b38e7e2e1f11d487854c281c

  1. … 54 more files in changeset.
Add new PCode Expression Cache feature.

This new cache is maintained by the SQL Compiler. The purpose of this

cache is to avoid the fairly expensive logic involved in transforming

unoptimized PCode to optimized PCode and, where applicable, to also

avoid

the logic involved in transforming optimized PCode to a Native

Expression. This cache is accessed ONLY by the SQL Compiler code.

NOTES:

* This is second attempt to check in this code. First attempt had to

be abandoned as other developers made changes which prevented

automatic merging.

* This code has been pre-reviewed by Justin, Qifan, Selva, Mike,

Ravisha, Suresh, and Dave B. Many thanks to them for various

suggestions. Most of those suggestions have been incorporated into

this delivery. A few are left for future improvements.

* There is one instance of this new cache per CmpContext.

* There are 5 new CQDs used to control this cache. To be effective for

all instances of the cache, these need to be set in the system

defaults table. The CQD command given to sqlci will affect only the

instance of the cache for the current CmpContext.

The 5 CQDs are:

PCODE_EXPR_CACHE_ENABLED - set to 0 to disable the cache. Default is 1

PCODE_EXPR_CACHE_SIZE - max size in bytes. Default is 2,000,000.

PCODE_EXPR_CACHE_CMP_ONLY - Compare Only mode - useful to QA and

Development only.

PCODE_EXPR_CACHE_DEBUG - set to 1 to enable debug mode. Default is 0

PCODE_DEBUG_LOGDIR - pathname of existing directory where debug log

files will be placed -- one log file per cache

instance. Log files are designed to be easily

imported into an Excel Spreadsheet. No default.

* Also included are a small number of changes to the Native Expressions

feature to (a) Use the new PCODE_DEBUG_LOGDIR cqd to specify where to

put the Native Expressions debug log files, (b) measure cpu-time

rather than wall-clock time for measuring how long it took to produce

a Native Expression, and (c) add a CQD named PCODE_NE_ENABLED so we

can easily disable the Native Expressions feature [though there is

currently no known reason for doing so.]

Change-Id: I58f833f63099743ff6c1107acdff94fe8aef4b70

  1. … 14 more files in changeset.
C++ run-time interface for TMUDFs

blueprint cmp-tmudf-compile-time-interface

- Support for C++ run-time interface:

- A new language, C++ is added to langman, the existing

LanguageManagerC handles both C and C++

- Two new parameter styles got added, C++ and Java

object-oriented parameter styles. Routines written in C++

use the new object-oriented C++ parameter style. The compiler

interface is only supported for that style (and in the future

for the Java object-oriented style).

- Also added one more compile time interface, the "completeDescription()"

call in the generator. Added logic to extract the UDRPlanInfo of

the optimal plan.

- Changes to UDRInvocationInfo and UDRPlanInfo classes:

- UDRInvocationInfo and UDRPlanInfo objects can now be serialized

and they are added to generated plans, as part of the UDR TDB.

- Split TableInfo into TupleInfo and TableInfo classes. TupleInfo

is now the common base class for describing both parameters and

input/output tables.

- TypeInfo now has offsets for data, null indicator and varchar

indicator.

- New get<type> and set<type> methods on class TupleInfo, to be

used at compile time for parameters and at runtime for parameters,

input and output tables.

- Added a "call phase" member, to be able to throw exceptions when

certain methods are called at the wrong time (e.g. trying to modify

compile time members at runtime).

- Routine class in langman now has a new subclass, LmRoutineCppObj

and a new method, invokeRoutineMethod, that is used to invoke

the object-oriented methods, requiring UDRInvocationInfo and

UDRPlanInfo as parameters.

- Fixed some executor issues with error handling for UDFs, this is

still not very well supported

- Emitting the EOD row in the UDF is no longer required, and no longer

supported or even possible.

- UDRPlanInfo is now part of the physical properties, so that we

can extract it from the optimal plan.

- Disabling TMUDF as the inner of a nested join - for now.

We might support this "routine join" at a later time.

- regress/udr/TEST001:

- SESSIONIZE_STATIC remains in C, but other TMUDFs are now

rewritten in C++ (the runtime part that was not yet in C++)

- SESSIONIZE_DYNAMIC is now the same as the example on the wiki

- regress/udr/TEST002: Added some tests for event log reader UDF,

but can't add the part that copies a sample log file, since

in Jenkins, we don't have $MY_SQROOT set. Tried the test on my

workstation, though. Steve tells me $MY_SQROOT should be available,

so in a future checkin I'll enable this code again.

- For patch set 2: Removed fix for LP bug 1420539 and addressed

other review comments.

Change-Id: I008ad68a8f25f1aaee94e1c45bbf097a267129bb

  1. … 73 more files in changeset.
SHOWPLAN related changes

Reworked the fix for bug 1392522 - mxosrvr core dumped doing showplan

(with N.E. enabled), plus other related code changes.

1) The native code (also known as native expression), if generated, is

stored in the expression's constant area. SHOWPLAN will dump the native

code in the assembly format. The display can be disabled by CQD

PCODE_NE_IN_SHOWPLAN to "OFF". It is "ON" by default. This part had been

reviewed by Jim Capps and Mike Hanlon.

2) Add several SHOWPLAN statements in core/TEST019 without logging the

output. This is to ensure no core generated when getting executor

operator (TDBs) info via SHOWPLAN.

3) A temporary fix to ComTdbHbaseAccess::displayRowId(). The current way

of retrieving begin or end row IDs from the HbaseScanRows for SHOWPLAN

does not match with the way those row IDs are generated (see

HbaseAccess::genListsOfRows), causing core dumps in some cases.

Change-Id: Id33e8c854452c03e15c4bd21f668f123180d9471

  1. … 9 more files in changeset.
LOB datatype infrastructure support

Technology preview.

More changes expected as part of this work before it

is user ready.

blueprint lob-support

This checkin contains basic support for create blob/clob datatypes.

The feature is disabled by default. Instructions on how to enable are

listed below.

New test executor/TEST130 that turns the feature on and tests out the

functionality.

New mxlobsrvr process will be started as part of sqstart.

Create and drop of tables with LOB columns.

No support for alter.

DML support for LOB datatypes.

Insert, update and deletes. Joins of 2 tables with LOB columns are

allowed but joins on LOB columns temselves are not allowed.

Insert-select from one LOB table another not yet supported.

Link to document from the blueprint will be added shortly.

To enable and try LOBs:

On a developer workstation :

cqd TRAF_BLOB_AS_VARCHAR 'OFF';

On a cluster after installing the code 2 steps are needed:

1. cqd TRAF_BLOB_AS_VARCHAR 'OFF';

2. sudo su hdfs --command "hadoop fs -mkdir /lobs"

sudo su hdfs --command "hadoop fs -chown -R trafodion:trafodion

/lobs"

This checkin includes several merges from the mainline and each of

the lines below represents one commit to the project branch where this

work was done. `

-Turning off LOB code by default. But turning it on in

executor/TEST130 just to ensure testing the code path.

-Support for showddl and some syntax for external files and stream.

-LOB regression test

-Workaround for dtm issue LP 378167

-Changes to make append work. Changes to use lob heap.

-Fix for using system heap for all LOB allocations and handling NULLs.

-Added workaround for cursor delete issue. LP 1376969

-Fixes for update.

-Parser changes for exe_util_lob_extract

-Pull in lob extract code

-Adding mxlobsrvr directory

-Fixed the LOB interface to use 2 new params for cursor fetches. They do

not overload the LOB Handle and LOB handle length anymore.

Added a flag to lobGlobals to indetify it's a hive access.

Cleaned up parser code.

-LOB support for create,drop,insert,delete,select.

Change-Id: I7c8125696e847b71580b746388632e75741bd347

  1. … 52 more files in changeset.
Various changes done to upgrade metadata from v23 to v30.

-- initialize trafodion, upgrade; will upgrade metadata from v23 to v30.

-- Following columns were added/dropped/modified in metadata tables:

-- COLUMNS: added sql_data_type

-- OBJECTS: added DROPPABLE column to indicated droppable table.

added schema_owner column.

-- TABLES: added num_salt_partns, table_format, comment, key_length,

row_data_length, row_total_length.

Removed hbase_create_options and moved it to TEXT.

-- SEQ_GEN: redef_ts and upd_ts added.

-- TEXT: text_type, sub_id added.

-- TABLE_CONSTRAINTS: added IS_ENFORCED, DROPPABLE, DISABLED,

VALIDATED, LAST_VALIDATED_TIME

-- reserved flags column added to metadata tables, initialize to zero.

-- default_value in COLUMNS is now UTF8 instead of UCS2

-- new columns populated during upgrade for existing objects.

-- values that moved from one metadata table to another populated during upgrade.

-- new columns populated during create operation for new objects.

-- step added to drop/create metadata views as part of upgrade

Change-Id: I173c33cbe7ee9314c66b84d16a6ef8b2d7394335

  1. … 20 more files in changeset.
Merge "Changes to support OSS poc."

  1. … 6 more files in changeset.
Changes to support OSS poc.

This checkin contains multiple changes that were added to support OSS poc.

These changes are enabled through a special cqd mode_special_4 and not

yet externalized for general use.

A separate spec contains details of these changes.

These changes have been contributed and pre-reviewed by Suresh, Jim C,

Ravisha, Mike H, Selva and Khaled.

All dev regressions have been run and passed.

Change-Id: I2281c1b4ce7e7e6a251bbea3bf6dc391168f3ca3

  1. … 143 more files in changeset.
DBSecurity: REVOKE ROLE, credential propagation, +

Overview

1) Corrects a CLI/Executor overwrite problem and removes workaround

code in PrivMgr. Launchpad bug #1371176.

2) REVOKE ROLE now lists referencing and referenced objects when a

revoke request is refused due to dependencies.

3) REVOKE ROLE now reports that the specified grant cannot be found

when grantor has not granted the role to the user. Previously the

misleading error "Not Authorized" was issued, which as confusing when

the user was DB__ROOT. The same change was made for REVOKE COMPONENT

PRIVILEGE. A similar change will be made in the future for revoking

object privileges.

4) REVOKE ROLE now considers grants to PUBLIC before concluding a

revoke would require a dependent object to be dropped.

5) User credential are now propagated to the the compiler process.

Launchpad bug 1373112.

Externals

If the priv/role, grantee, grantor tuple does not exist, REVOKE

ROLE/REVOKE COMPONENT PRIVILEGE now reports error 1018: Grant of role or

privilege <name> from <grantor> to <grantee> not found, revoke request

ignored.

When REVOKE ROLE detects a dependent object, error message 1364 now

reports the referencing and the referenced object.

Cannot revoke role <role-name>. Object <referencing-object> depends on

privileges on object <referenced-object>.

Details for user credential propagation:

The propagate user credentials code has only been partially implemented.

The existing code sends the user ID to the first compiler process.

Other compiler processes started would not get the connected user ID

instead the DB__ROOT user ID became the user by default. Therefore,

privilege checks are succeeding when they should fail.

User credentials consist of an integer user ID and a username. The

existing code only passed the user ID. The compiler process would

then do a metadata look-up to get the username. If we kept this

model, then we would get into an infinite loop:

When the compiler process received the user ID, it did a

metadata read to get the associated username. After reading the

metadata, both the username and user ID was set in context globals.

The metadata lookup code will start another arkcmp process for the

compilation request. The compilation would then start a compiler

process. That compiler process would start another compiler process,

etc.

The solution is to send both the username and user ID to the compiler

process. Both values are known at the time the compiler process is

started. This alleviates the need for a database look-up when the

compiler process starts. To do this a new session attribute was

created - SESSION_DATABASE_USER. This session attribute sends both the

user ID and username to the compiler process during startup processing.

Once we were able to start a compiler process and store a user ID other

than DB__ROOT in the Context globals, another similar infinite loop

occurred during privilege checking. For example, a showddl command

starts a compiler process when extracting privilege information. The

compiler calls checkPrivileges to make sure the current user has

privileges. The checkPrivileges statement makes a metadata request

that requires a compilation. This starts up another compiler process.

This compiler process is sent the metadata request. When compiling the

metadata request in the new compiler process, checkPrivileges is called

which starts a compiler process, …

This worked previously because the user passed was DB__ROOT, and the code

in checkPrivileges is short circuited and the metadata call is avoided.

A fix to set the parserflag (INTERNAL_QUERY_FROM_EXEUTIL) before the

metadata request was performed. This fix requires that the file

"SqlParserGlobalsCmn.h" be included in additional files. Including this

file needs to be done with care. In order to get everything to compile,

we changed where this file was included in several places.

Once all these changes were made, the envvar: DBUSER_DEBUG now works.

If set, then good details about how users are sent to different

processes is displayed.

Change-Id: If7538eee38178c2345fe418172c6196b25a20b33

  1. … 30 more files in changeset.
Various launchpad fixes.

-- upshift datatype in default clause is now handled correctly

-- ctas creation was returning error if constraints were used. Fixed by not starting a transaction for the root query.

-- compiler crash fixed in parser due to a null ptr reference for a ctas stmt

-- hist stats tables are correctly dropped from schema during a drop schema stmt

-- unique constraint is now created for a table with column level primary key clause

-- sometimes tables with constraints were not dropped correctly during drop schema. That has been fixed.

-- get indexes in schema is now supported

Change-Id: Ic05649a9f279dc4660d1dd33e8960142bb7b3430

  1. … 10 more files in changeset.
Initial code drop of Trafodion

    • -0
    • +5788
    ./GenExpGenerator.cpp
  1. … 4886 more files in changeset.