NATable.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Code change for ESP colocation strategy described in LP 1464306

Change-Id: I838bc57541bf753e3bad90ae7776e870958c930a

  1. … 14 more files in changeset.
various fixes and enhancements, details below.

-- improved DDL performance by not invalidating internal create/alter

operations.

-- added an optimization during CREATE INDEX to not go through

'upsert using load' processing if source table is empty.

-- added support for ISO datetime format (2015-06-01T07:35:20Z)

-- added support for RESET option to ALTER SEQUENCE and IDENTITY.

This will reset generated seq num to the START VALUE.

-- added support for cqd TRAF_STRING_AUTO_TRUNCATE.

If set, strings will be automatically truncated during insert/update.

-- fixed sqlci to pass in correct varchar param len indicator (2 or 4 bytes).

-- changed sizeof(short) to correct vcindlen (2 or 4 bytes)

-- removed some NA_SHADOWCALLS defines

Change-Id: Ie6715435d9c210ae6c2db4ff6bc0545c1b196979

  1. … 39 more files in changeset.
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
Part 1 of ALTER TABLE/INDEX ALTER HBASE_OPTIONS support

blueprint alter-table-hbase-options

This set of changes includes the following:

1. Syntax for ALTER TABLE/INDEX ALTER HBASE_OPTIONS

2. Compiler binding and generation logic

3. Run-time logic to update the metadata TEXT table with the

new options.

Still to come is:

1. Logic to actually change the HBase object

2. Transactional logic to actually change the HBase object

The functionality in this set of changes is only marginally

useful. If you manually change hbase options on a Trafodion

object using the hbase shell, you could use this ALTER

TABLE/INDEX command to update the Trafodion metadata. (Of

course some care would have to be taken to use the same

options!).

Change-Id: Id0a5513fe80853c06acdbbf6cc9fd50492fd07b2

  1. … 18 more files in changeset.
get indexLevel and blockSize from Hbase metadata to use in costing code.

Change-Id: I7b30364ec83a763d3391ddc39e12adec2ca1bd00

  1. … 9 more files in changeset.
Merge "Remove some dead code"

Remove some dead code

Remove dead code concerned with constraint and schema labels.

This is an anachronism from pre-open-source versions of the code.

Most of the code removed is in the compiler, with a small amount

of cli and executor code removed.

Change-Id: Ic8a833bb15d1ca9a0e2e2683f2d4644b44c4f96b

  1. … 13 more files in changeset.
Security fixes for 144553, 1414125, and 1393529

1445583: showstats command performance slow with security enabled

Several changes were made to improve performance:

Performance optimization:

NATable.cpp: NATable::setupPrivs

- If the current user is the object owner, then default the privilege bitmap

to object Owner values - no need to call PrivMgr to get privileges

Caching optimization:

We are now caching privmgr metadata tables in compiler cache when the compiler

context is instantiated. This avoids a metadata lookup for these tables.

- Added new methods that return if the table is part of the PrivMgr schema

- Adjusted CmpSeabaseDDL::createMDdescs to include privmgr metadata in the

cached entries

- Adjusted CmpSeabaseDDL::getMDtableInfo to check for privmgr metadata tables

from the cached entries

- Removed obsolete code CmpSeabaseDDL::alterSeabaseDropColumn

- changed CmpSeabaseDDL::getSeabaseTableDesc to check for both system and

privmgr metadata from compiler cache

- added new method CmpSeabaseDDL::getPKeyInfoForTable that returns the

primary key name and UID for a table. This is needed when dropping privmgr

metadata tables

Removed extraneous recompilations of HISTOGRAM structures:

Today, update statistics and showstats are reloading NATable entries

for HISTOGRAM tables on every access. This is because the parserflag

ALLOW_SPECIALTABLETYPE is turned on. When this flag is turned, the compiler

always reloads the cache entries - see code from CmpMain::sqlcomp:

//if using special tables e.g. using index as base table

//select * from table (index_table T018ibc);

//then refresh metadata cache

if(Get_SqlParser_Flags(ALLOW_SPECIALTABLETYPE) &&

CmpCommon::context()->schemaDB_->getNATableDB()->cachingMetaData())

CmpCommon::context()->schemaDB_->getNATableDB()->refreshCacheInThisStatement();

Changed code to not set ALLOW_SPECIALTABLETYPE and ALLOW_PHONYCHARACTERS

parserflags by default. Individual statements are setting these flags as needed.

1414125: User without priv can view data in metadata tables

The problem is that a user with priv cannot view data in metadata tables.

Even when a user had SELECT privilege on a system or privmgr metadata table,

the request failed.

The problem is that parameter 2 sent to CmpDescribeIsAuthorized in

hs_globals.cpp is NULL so SELECT priv is not checked. If the user has SHOW

component privilege, it works. A call was added to getPrivileges for metadata

tables before calling CmpDescribeIsAuthorized.

1393529: Core dump accessing MD table descriptors

When "UPDATE STATISTICS LOG [ON, OFF, CLEAR]" is specified by a non DB__ROOT

user, a core dump occurred. This happens because the isAuthorized check is

performed expecting a NATable structure. This command does not need any

special security checks.

Updated traf_authentication_setup script to support a new installation option

Change-Id: If7dbf3ec66e5beb7d88bda61ef32611401dd97b9

  1. … 10 more files in changeset.
Fix for bug 1442932 and bug 1442966, encoding for varchar

Submitting this before finishing regressions on workstation, in the

interest of time.

Key encodings for VARCHAR values used to put a varchar length indicator

in front of the encoded value. The value was the max. length of the

varchar and the indicator was 2 or 4 bytes long, depending on the

length of the indicator in the source field. That length used to

depend only on the max number of bytes in the field, for >32767

bytes we would use a 4 byte VC length indicator.

Now, with the introduction of long rows, the varchar indicator length

for varchars in aligned rows is always 4 bytes, regardless of the

character length. This causes a problem for the key encoding.

We could have computed the encoded VC indicator length from the field

length. Anoop suggested a better solution, not to include the VC

indicator at all, since that is unnecessary. Note that for HBase row

keys stored on disk, we already remove the VC indicator by converting

such keys from varchar to fixed char. Therefore, the issue happens

only for encoding needed in a query, for example when sorting or in a

merge join or union.

Description of the fix:

1. Change CompEncode::synthType not to include the VC length

indicator in the encoded buffer. This change also includes

some minor code clean-up.

2. Change the assert in CompEncode::codeGen not to include the

VC indicator length anymore.

3. Changes in ex_function_encode::encodeKeyValue():

a) Read 2 and 4 byte VC length indicators for VARCHAR/NVARCHAR.

b) Small code cleanup, don't copy buffer for case-insensitive

encode, since that is not necessary.

c) Don't write max length as VC length indicator into target

and adjust target offsets accordingly (for VARCHAR/NVARCHAR).

4. Other changes in sql/exp/exp_function.cpp:

d) Handle 2 and 4 byte VC len indicators in hash function

and Hive hash function (problems unrelated to LP bugs fixed).

e) Add some asserts for cases where we assume VC length indicator

is a 2 byte integer.

CompDecode is not yet changed. Filed bug 1444134 to do that for

the next release, since that change is less urgent.

Patch set 2: Copyright notice changes only.

Patch set 3: Updated expected regression test file that

prints out encoded key in hex.

Change-Id: Idab3ed488f8c1b9aabedba4689bfb8d7286b9538

  1. … 5 more files in changeset.
metadata fixes and 'sqlmp' code cleanup

-- NATable struct for metadata was being created multiple

times whenever information for a new table was read

from metadata. That has been fixed.

-- an 'initialize trafodion, drop' followed by 'initialize traf'

from the same session was failing due to priv info not getting

reset. This would show up if 'initialize authorization' was

done earlier. That has been fixed.

-- code cleanup mostly related to sqlmp legacy code and reference.

Change-Id: I346e3f3bbc6c7784b38e7e2e1f11d487854c281c

  1. … 54 more files in changeset.
Miscellaneous DDL and security bug fixes

Fixed a testware issue with fullstack2/TEST062 that occurred during

release testing

Bug 1415196 - Alter volatile table add column cores at CmpSeabaseDDL::alterSeabaseTableAddColumn()

Added a check to not allow add or drop column for volatile tables:

- sqlcomp/CmpSeabaseDDLtable.cpp

Bug 1415232 - A failed create view causes a volatile table to disappear

The code to bind a view does not correctly reset the volatile schema in use

session parameter in case of an error. Subsequent calls do not check for

volatile objects.

parser/BindStmtDDL.cpp

Bug 1371265 - should not allow grants to DB__ROOT or current user

Added a check at grant to prevent this

- sqlcomp/PrivMgrPrivileges.cpp

Bug 1392491 - Unavailability of privmgr metadata error is incomplete

If not all the privmgr metadata is available, then a new Compile context

flag called IS_AUTHORIZATION_READY is set. This flag is adjusted when

a new compiler context is started, and when authorization is enabled and

disabled.

When isAuthorizationEnabled is called and authorization is incomplete,

error 1234 is now returned by default.

After coding changes were added, a request to not check all privmgr metadata

table at context startup was requseted - a performance concern. Fix was

changed to check all tables for debug builds but check only one table for

release builds. If the performance problem is fixed, then we can go back

and check for all privmgr tables.

- arkcmp/CmpContext.h

- arkcmp/CmpContext.cpp

- sqlcomp/CmpSeabaseDDLcommon.cpp

- sqlcomp/nadefaults.cpp

Bug 1402009 - DB__ROOT is unable to grant privilege on object in private schema

When DB__ROOT executes a grant or revoke on objects it does not own, need to

change the grantor from DB__ROOT to the object owner. This matches the same

behavior for other DDL operations such as CREATE.

As part of this fix, the GRANTED BY clause is now allowed for GRANT

statements but it won't be complete until LP bug 1414225 is done.

- sqlcomp/CmpSeabaseDDLtable.cpp (seabaseGrantRevoke)

- sqlcomp/PrivMgrCommands.h

- sqlcomp/PrivMgrCommands.cpp

- sqlcomp/PrivMgrPrivileges.h

- sqlcomp/PrivMgrPrivileges.cpp

- sqlcomp/PrivMgrMD.h

Bug 1414125 - User without priv can view data in metadata tables

Fixes were in place for all metadata tables except the privmgr metadata

tables. The priv information was always being set to none in setupPrivInfo

(NATable) and revoking a privilege was not correctly removing privilege

information from object_privileges.

- optimizer/NATable.cpp

- sqlcomp/PrivMgrCommands.cpp

Bug - create library checking privileges when authorization is not enabled

- CmpSeabaseDDLroutine.cpp

Enhanced the sqlci env command:

- alphabetize the output

- add the following information

-- authentication status

-- authorization status

-- external (LDAP) user connected

A new session parameter called SESSION_EXTERNAL_USER_NAME was added to return

the external user name connected.

A new cli request called SQL_EXEC_GetAuthState_Internal was written to return

the authentication & authorization status. Code was also added, but not yet

supported, for auditing status.

Renamed member/methods that use ldap to external

Changed sqlci env command to return new format

- cli/sqlcli.h

- cli/SQLCLIdev.h

- cli/CliExtern.cpp

- cli/Cli.h

- cli/Cli.cpp

- cli/Context.h

- cli/Context.cpp

- qmscommon/QRQueries.cpp

- sqlci/SqlciEnv.h

- sqlci/SqlciEnv.cpp

- regress/fullstack2/EXPECTED062.SB

- regress/funnstack2/DIFF062.KNOWN.SB.OS

Change-Id: I04627435a0e644c6b14bbf6bd8aa1162d81224fb

  1. … 27 more files in changeset.
repository upgrade and explain enhancements. Details below.

-- repository upgrade infrastructure integrated

with 'initialize trafodion, upgrade'

-- fields have been added/removed/modified in repos tables

-- new repos table has been added

-- upgrade enhanced to only upgrade subset of components that

need to be upgraded. One or more of metadata, repos, views, priv.

-- metadata version now tracks release version and indicates

the traf release where upgrade was needed.

Current metadata version will be 1.1.0.

-- packed explain data is now returned to caller so it could be

stored in repository

-- explain info from repository for a query id could be retrieved

and displayed

explain qid <query-id>;

explain options 'f' qid <query-id>

select * from table(explain(null, 'EXPLAIN_QID=<query-id>'));

-- a sql query could be explained and returned in relational format.

select * from table(explain(null, 'EXPLAIN_STMT=<query-str>));

Users only need to have select permission on tables referenced

in query-str.

-- explain and statistics tables could be invoked

invoke table(explain(null, null));

invoke table(statistics(null, null));

Squashed commit of the following:

commit 09004ff7260b771378bc1a29f0ecb82e5e0c6100

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 16 12:24:03 2015 -0700

repos and explain, #12

Change-Id: Ia08151b5c7087b6b7daa5b662df2a584e5a7b2a1

commit 46f05ada264094626e45445d3875ccf876cad802

Merge: 3393587 6b3a4c4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 21:27:48 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 3393587de319a3ad43d7d360cd74501e8b8103e4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 21:26:30 2015 -0700

repos and explain, #11

Change-Id: I895a4b4766e0a92b7472a6b347f0bb3ee07fa9b6

commit cf6ffee549b37ffbcd6ce750a1fdbf7f0c410d61

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Sun Mar 15 16:07:22 2015 -0700

repos and explain, #10

Change-Id: I19e182ec6600525cec986f173afaf68a44e04af2

commit e331eec68feea59d9fa11e0154e3093762120eb2

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 18:19:03 2015 -0700

repos and explain, #9

Change-Id: I783c75513a432f718d2cd7c6030d410c419a79c8

commit 9e9da221eb45b760963741657fe9cb910a753c84

Merge: 20b64e5 9eb7f37

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 14:52:09 2015 -0700

Merge remote branch 'gerrit/master' into fix1

Conflicts:

sql/common/ComSmallDefs.h

sql/executor/ExExeUtilCli.h

sql/generator/GenRelExeUtil.cpp

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLupgrade.cpp

sql/sqlcomp/CmpSeabaseDDLupgrade.h

Change-Id: Id21b10ff35ad506f409a83a8d955c2e643a2bcf5

commit 20b64e5552d8f614586ed843aad76d87d3473e2e

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 13:44:15 2015 -0700

repos and explain, #8

Change-Id: I2b49d962f08a22ca6c8b437a583855e2344f513c

commit 500b7a9486f23fff6b1c17bb6e79c2f614a7e3e6

Merge: f2c050e fc1c31d

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 08:07:34 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit f2c050ea048b409b7d6769172e16c4ab26267bd0

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 13 08:07:15 2015 -0700

repos and explain, #7

Change-Id: I0bc04b6e8f147af1defbfd023c3700e26cc5b6f1

commit fabdb7dadd34238a8a227e880aa868ce484dc580

Merge: 59bb689 74640d7

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Mar 12 09:01:20 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 59bb6893437abe3999a53188503c60cf47633458

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Mar 12 09:00:44 2015 -0700

repos and explain, #7

Change-Id: I6595a04ad59c1f705a04beee18c877cb81db0fac

commit a86353b45436126915fd2ba74c9af53f8bfd4d19

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 9 16:21:16 2015 -0700

repos and explain, #6

Change-Id: Ie4fe2b2bece7de3697eb8ada0d1cb1067b89bb88

commit 61272e0d6107536ab8b691415770e856267b6da5

Merge: 1708cb4 ca5dfb4

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 9 15:52:43 2015 -0700

Merge remote branch 'gerrit/master' into fix1

commit 1708cb44511809bf8e63800d226ac8ea34916726

Merge: 1284f4f 31f25ec

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 6 14:55:13 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 1284f4f78f1f43657f29899cc8b9a96c02f37815

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Mar 6 14:54:54 2015 -0800

commit: repos and explain, #6

Change-Id: I38d796fd4ab58bf29fa62c4bd5492ce44de9c8a0

commit f639911a1ea4247ae4c297bbbd904b84c7587b3a

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 2 06:31:21 2015 -0800

repos and explain, #5

Change-Id: Ie97ba3cac8dc78784d0a7eef57176e51b9e16224

commit da69fe9595d4ad5c9adf3b32ae8f848a0e6be964

Merge: 9b95b8c a5caeef

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Mar 2 06:31:00 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 9b95b8c3240a4186704b8b7f2ff8cefebf211732

Merge: ea2f687 8cd8a92

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Feb 26 08:50:25 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit ea2f6871b7c7803d0f1e3fa664b474e069a0befd

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Thu Feb 26 08:50:03 2015 -0800

commit: repos and explain, #5

Change-Id: I44d8f87b3652e6300e41302d3a3e88b9b18652a3

commit 522a47eb991df2245005c693e4d034235adbfdc5

Merge: d986988 f42694e

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 25 13:13:02 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit d98698856a4be479a7603b0fb79210e80e88e1c1

Merge: 5841e47 2aad3ef

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 18 14:00:24 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 5841e47f1e3f9a018424298a92cd134f4430e6bf

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 18 13:57:46 2015 -0800

repos and explain, #4

Change-Id: I01f53003069598381743c9be80b405ed37cefe6d

commit e7b5966d8ce653a57b8128c0b24c15fc05737668

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Wed Feb 11 11:28:20 2015 -0800

repos and explain, #3

Change-Id: Iff148ef175a7641ea37e27e60435ccf8f613e6db

commit aba98dc8cfda62d4e708cd892ed626f2a3051eb0

Merge: 66f1926 19a0c32

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 10 14:09:58 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit 66f1926d279f2b408d436b0aa53a3a6a5388901e

Merge: 9d0ef2a b68f59b

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Mon Feb 9 10:52:35 2015 -0800

Merge remote branch 'gerrit/master' into fix1

Conflicts:

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLtable.cpp

Change-Id: I6348fec3aece2baad9e9570749f43c086416219e

commit 9d0ef2aa01a62ea3e4e0ab35f8ec9687ef3336ef

Merge: aac268d 2cd8fbb

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Fri Feb 6 14:15:49 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit aac268d097afbe506dcf76de4d8f49652a1d4df7

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 16:31:17 2015 -0800

repos and explain, #2

Change-Id: Ia2a50159e577cd6ff8110dca35236fa90ea0d260

commit 2e6160411c8facb448e78d387fa3227e7b759c5d

Merge: c6e2fe5 566706d

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 15:21:03 2015 -0800

Merge remote branch 'gerrit/master' into fix1

commit c6e2fe529f9d1e59d3eb26cb145b260d81814443

Author: Anoop Sharma <anoop.sharma@hp.com>

Date: Tue Feb 3 15:16:36 2015 -0800

repository updated and explain changes, #1

Change-Id: I5daf66a2064b46ad65f1200579385965f7f56808

Change-Id: I0211d754524b8c06a05bb7ade20f884ec91340d2

  1. … 58 more files in changeset.
Trafodion Metadata Cleanup command support.

Various changes to support cleanup command has been added.

A separate external spec contains the details.

Summary of syntax:

cleanup [ table t | index i | sequence s | object o] [, uid <value>]

cleanup [private | shared] schema sch

cleanup uid <value>

cleanup metadata, check, return details

In addition, a new command to get names of various hbase objects

has also been added:

get [ all | user | system | external ] hbase objects;

Change-Id: I93f1f45e7fd78091bacd7c9f166420edd7c1abee

  1. … 79 more files in changeset.
Snapshot Scan changes

The changes in this delivery include:

-decoupling the snapshot scan from the bulk unload feature. Setup of the

temporary space and folders before running the query and cleanup afterwards

used to be done by the bulk unload operator because snapshot scan was specific

to bulk unload. In order the make snapshot scan indepenednt from bulk unload

and use it in any query the setup and cleanup tasks are now done by the query

itself at run time (the scan and root operators).

-caching of the snapshot information in NATable to optimize compilation time

Rework for chaching: when the user sets TRAF_TABLE_SNAPSHOT_SCAN to LATEST

we flush the metadata and then we set the caching back to on so that metadata

get cached again. If newer snapshots are created after setting the cqd they

won't be seen if they are already cached unless the user issue a command/cqd

to invalidate or flush the cache. One way for doing that can be to issue

"cqd TRAF_TABLE_SNAPSHOT_SCAN 'latest';" again

-code cleanup

below is a description of the CQds used with snapshot scan:

TRAF_TABLE_SNAPSHOT_SCAN

this CQD can be set to :

NONE--> (default)Snapshot scan is disabled and regular scan is used ,

SUFFIX --> Snapshot scan is enabled for the bulk unload (bulk unload

behavior is not changed)

LATEST --> Snapshot Scan is enabled independently from bulk unload and

the latest snapshot is used if it exists. If no snapshot exists

the regular scan is used. For this phase of the project the user

needs to create the snapshots using hbase shell or other tools.

And in the next phase of the project new comands to create,

delete and manage snapshots will be add.

TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX

This CQD is used with bulk unload and its value is used to build the

snapshot name as the table name followed by the suffix string

TRAF_TABLE_SNAPSHOT_SCAN_TABLE_SIZE_THRESHOLD

When the estimated table size is below the threshold (in MBs) defined by

this CQD the regular scan is used instead of snapshot scan. This CQD

does not apply to bulk unload which maintains the old behavior

TRAF_TABLE_SNAPSHOT_SCAN_TIMEOUT

The timeout beyond which we give up trying to create the snapshot scanner

TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION

Location for temporary links and files produced by snapshot scan

Change-Id: Ifede88bdf36049bac8452a7522b413fac2205251

  1. … 44 more files in changeset.
OSIM (Optimizer Simulator) redesign 1.

Simulate query plan generation of production cluster on dev workstation,

by collecting information from production cluster, and restore it on dev worksation.

--runnig on production clusters, collect table DDLs, statistics, CQD, to osim-directory,

--the directory path can either full(absolute) or relative.

osim capture location '<osim-directory>'[, force];

--runing queries on cluster

osim capture stop;

--restore DDLs, CQDs, statistics and cluster information.

osim load from '<osim-directory>';

--setup runtime information, like cpu number, node number.

osim simulate start|continue '<osim-directory>';

Change-Id: I30882e87a6ea0f08c9aa64685705eebebcbb3bf0

  1. … 38 more files in changeset.
Fixed catman1 TEST137 failure

And some minor changes from a previous review.

Change-Id: I01319b40e3ddf1bdba88c70d5a4b62057698491e

  1. … 6 more files in changeset.
SHOWDDL, QUERY Cancel, rework

This delivery addresses security issues with SHOWDDL, adds initial

support for security in query cancel, and implements part of the

proposed GIVE commands.

Bug 1414234: SHOWDDL command now check component privileges.

SHOW is granted to PUBLIC by default, so effectively there are

no new restrictions unless SHOW is revoked from PUBLIC.

SHOWDDL COMPONENT now checks for MANAGE_COMPONENTS or SHOW privilege.

SHOWDDL ROLE now checks for MANAGE_ROLES or SHOW privilege.

SHOWDDL SCHEMA now checks for SHOW privilege.

SHOWDDL USER now checks for MANAGE_USERS or SHOW privilege.

SHOWDDL LIBRARY is implemented. A user must have the USAGE

privilege on the library, or the MANAGE_LIBRARY or SHOW privilege.

New function to determine if the user canceling the query has

the authority: either DB__ROOT, or the user owns the query, or

the user has the QUERY_CANCEL privilege. Note, the code is

delivered in an inactive state pending future integration.

Three new component privileges are added: QUERY_ACTIVATE,

QUERY_CANCEL, and QUERY_SUSPEND. These will be added if

authorization is dropped and reinitialized. A future

delivery will add an INITIALIZE AUTHORIZATION,UPDATE

command that will add these privileges to an existing

instance with authorization enabled.

Support for library objects was added to NATable, but the code

is currently not used. May be integrated into CREATE ROUTINE

and GRANT for libraries in the future.

Also included is minor rework from delivery 1082, and the

GIVE SCHEMA command now updates associated privileges when object

ownership is changed. Note, GIVE commands are still prototype.

A detailed blueprint for GIVE will be released shortly.

This patch merges with changes from 1177 and addresses a couple of

minor comments from the initial submittal.

Change-Id: I60419228f886555ed0e066441bb824c5246ee498

  1. … 28 more files in changeset.
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

  1. … 39 more files in changeset.
Changes to destroy more elements in NATable destructor

Changed to use CmpStatement Heap instead of NATable Heap

while creating the table descrptor.

With these changes, the NATable Heap allocated size

is around 22K at the end of seabase/TEST020 with no table

entries in the cache. Earlier, the allocated size was

around 1.5MB.

Patch Set #5:Addressed all the review comments

Change-Id: I61777dcb02f67f87092340d69086ba0ec0de2493

  1. … 4 more files in changeset.
Merge "More memory leak Changes"

Specify compiler context when querying natable virtual table interface

select * from table(natablecache('ALL','local')) ;

select * from table(natablecacheentries('meta','local')) ;

select * from table(naroutinecache('user','local')) ;

Following the querycache virtual table interface, now the natable and

naroutine cache virtual table interfaces also support specifying the

name of the context we want to query. The first parameter can be 'all' or

name of context (e.g. 'meta', 'user', 'ustat'). The second parameter

can be local or remote. Parameters are case insensitive.

For remote compiler we only query the context pointed to by activeschemadb.

The column num_entries has been added to natablecache virtual table.

This change can be used to monitor memory growth in these caches.

Patch Set 2:

Address issues found by Dave. Changes in 4 files

sql/arkcmp/NATableSt.cpp

sql/arkcmp/NATableSt.h

sql/arkcmp/QueryCacheSt.cpp

sql/optimizer/NARoutine.cpp

Changes cover

A minor leak in in HQCIterator

returning FALSE in NATable/RoutineCacheStats iterator getNext methods

Change-Id: Icf15c93b9ae3c3f523d0abe1580ce7280c5b0d84

  1. … 20 more files in changeset.
More memory leak Changes

Changes to delete more elements in NATable. LIST or collection type

elements in NATable that collect references are now deleted.

Also removed some defensive code in RMS because it is not valid.

Added code to delete the internal statements when the row

is not found

PatchSet 3 contains the fix for T4 Phoenix test failures

Change-Id: Ib43da75582dd56d2562cb753738abed0ea5842d3

  1. … 2 more files in changeset.
Merge "Fix for more leaks in NATable"

Merge "Change to avoid placing large scan results in RegionServer cache"

  1. … 9 more files in changeset.
Remove code and cqds related to Thrift interface

ExpHbaseInterface_Thrift class was removed a few months ago. Completing

that cleanup work. exp/Hbase_types.{cpp,h} still remain. These are Thrift

generated files but we use the structs/classes generated for JNI access.

Change-Id: I7bc2ead6cc8d6025fb38f86fbdf7ed452807c445

  1. … 19 more files in changeset.
Change to avoid placing large scan results in RegionServer cache

By default the result of every Scan and Get request to HBase is placed in

the RegionServer cache. When a scan returns a lot of rows this can lead to

cache thrashing, causing results which are being shared by other queries

to be flushed out. This change uses cardinality estimates and hbase row size

estimates along with the configured size of region server cache size to

determine when such thrashing may occur. Heap size for region server is

specified through a cqd HBASE_REGION_SERVER_MAX_HEAP_SIZE. The units are in MB.

The fraction of this heap allocated to block cache is read from the config

file once per session through a lightweight (no disk access) JNI call. The

hueristic used is approximate as it does not consider total number of region

servers or that sometimes a scan may be concentrated in one or a few region

servers. We simply do not place rows in RS cache, if the memory used by all

rows in a scan will exceed the cache in a single RS. Change can be overridden

with cqd HBASE_CACHE_BLOCKS 'ON'. The default is now SYSTEM. Change applies

to both count(*) coproc plans and regular scans.

Change-Id: I0afc8da44df981c1dffa3a77fb3efe2f806c3af1

  1. … 20 more files in changeset.
Fix for more leaks in NATable

Added code to delete more elements in NATable destructor. This

fixes leaks when NATable is destroyed.

Changed the NATable Cache Management to use allocated size rather

than the total size.

This also enables seabase/TEST020 to pass in debug mode.

Change-Id: Ifebc20e602c0149f73f9cefcd11b37c69d9eec74

port skew buster to Trafodion

1. add simplified TEST062

2. reuse cached partitioning expression when only doVarCharCast

is the same as the when the expression is created in

TableHashPartitioningFunction::createPartitioningExpressionImp().

3. rework

4. fix a time monitor bug reporting incorrect processor time (CPU time

computed from clock() calls)

5. comment out the assert on the size of NATable cache not decreasing.

This is to fix regression failure with seabase/TEST020. Selva will check

in a formal fix that allocates space for NATable objects from a single

heap, and then reenables this assert check.

Change-Id: I9eeee4f36ba8678e90e0ac68a85bfc733599d932

  1. … 26 more files in changeset.