NATable.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
initial support for returning multiple versions and column timestamps

This feature is not yet externalized.

Support added to:

-- return multiple versions of rows

-- select * from table {versions N | MAX | ALL}

-- get hbase timestamp of a column

-- select hbase_timestamp(col) from t;

-- get version number of a column.

-- select hbase_version(col) from t

Change-Id: I37921681fc606a22c19d2c0cb87a35dee5491e1e

  1. … 48 more files in changeset.
Code change for ESP colocation strategy described in LP 1464306

Change-Id: I838bc57541bf753e3bad90ae7776e870958c930a

  1. … 14 more files in changeset.
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
get indexLevel and blockSize from Hbase metadata to use in costing code.

Change-Id: I7b30364ec83a763d3391ddc39e12adec2ca1bd00

  1. … 9 more files in changeset.
Remove some dead code

Remove dead code concerned with constraint and schema labels.

This is an anachronism from pre-open-source versions of the code.

Most of the code removed is in the compiler, with a small amount

of cli and executor code removed.

Change-Id: Ic8a833bb15d1ca9a0e2e2683f2d4644b44c4f96b

  1. … 13 more files in changeset.
metadata fixes and 'sqlmp' code cleanup

-- NATable struct for metadata was being created multiple

times whenever information for a new table was read

from metadata. That has been fixed.

-- an 'initialize trafodion, drop' followed by 'initialize traf'

from the same session was failing due to priv info not getting

reset. This would show up if 'initialize authorization' was

done earlier. That has been fixed.

-- code cleanup mostly related to sqlmp legacy code and reference.

Change-Id: I346e3f3bbc6c7784b38e7e2e1f11d487854c281c

  1. … 54 more files in changeset.
Snapshot Scan changes

The changes in this delivery include:

-decoupling the snapshot scan from the bulk unload feature. Setup of the

temporary space and folders before running the query and cleanup afterwards

used to be done by the bulk unload operator because snapshot scan was specific

to bulk unload. In order the make snapshot scan indepenednt from bulk unload

and use it in any query the setup and cleanup tasks are now done by the query

itself at run time (the scan and root operators).

-caching of the snapshot information in NATable to optimize compilation time

Rework for chaching: when the user sets TRAF_TABLE_SNAPSHOT_SCAN to LATEST

we flush the metadata and then we set the caching back to on so that metadata

get cached again. If newer snapshots are created after setting the cqd they

won't be seen if they are already cached unless the user issue a command/cqd

to invalidate or flush the cache. One way for doing that can be to issue

"cqd TRAF_TABLE_SNAPSHOT_SCAN 'latest';" again

-code cleanup

below is a description of the CQds used with snapshot scan:

TRAF_TABLE_SNAPSHOT_SCAN

this CQD can be set to :

NONE--> (default)Snapshot scan is disabled and regular scan is used ,

SUFFIX --> Snapshot scan is enabled for the bulk unload (bulk unload

behavior is not changed)

LATEST --> Snapshot Scan is enabled independently from bulk unload and

the latest snapshot is used if it exists. If no snapshot exists

the regular scan is used. For this phase of the project the user

needs to create the snapshots using hbase shell or other tools.

And in the next phase of the project new comands to create,

delete and manage snapshots will be add.

TRAF_TABLE_SNAPSHOT_SCAN_SNAP_SUFFIX

This CQD is used with bulk unload and its value is used to build the

snapshot name as the table name followed by the suffix string

TRAF_TABLE_SNAPSHOT_SCAN_TABLE_SIZE_THRESHOLD

When the estimated table size is below the threshold (in MBs) defined by

this CQD the regular scan is used instead of snapshot scan. This CQD

does not apply to bulk unload which maintains the old behavior

TRAF_TABLE_SNAPSHOT_SCAN_TIMEOUT

The timeout beyond which we give up trying to create the snapshot scanner

TRAF_TABLE_SNAPSHOT_SCAN_TMP_LOCATION

Location for temporary links and files produced by snapshot scan

Change-Id: Ifede88bdf36049bac8452a7522b413fac2205251

  1. … 44 more files in changeset.
Changes to destroy more elements in NATable destructor

Changed to use CmpStatement Heap instead of NATable Heap

while creating the table descrptor.

With these changes, the NATable Heap allocated size

is around 22K at the end of seabase/TEST020 with no table

entries in the cache. Earlier, the allocated size was

around 1.5MB.

Patch Set #5:Addressed all the review comments

Change-Id: I61777dcb02f67f87092340d69086ba0ec0de2493

  1. … 4 more files in changeset.
Specify compiler context when querying natable virtual table interface

select * from table(natablecache('ALL','local')) ;

select * from table(natablecacheentries('meta','local')) ;

select * from table(naroutinecache('user','local')) ;

Following the querycache virtual table interface, now the natable and

naroutine cache virtual table interfaces also support specifying the

name of the context we want to query. The first parameter can be 'all' or

name of context (e.g. 'meta', 'user', 'ustat'). The second parameter

can be local or remote. Parameters are case insensitive.

For remote compiler we only query the context pointed to by activeschemadb.

The column num_entries has been added to natablecache virtual table.

This change can be used to monitor memory growth in these caches.

Patch Set 2:

Address issues found by Dave. Changes in 4 files

sql/arkcmp/NATableSt.cpp

sql/arkcmp/NATableSt.h

sql/arkcmp/QueryCacheSt.cpp

sql/optimizer/NARoutine.cpp

Changes cover

A minor leak in in HQCIterator

returning FALSE in NATable/RoutineCacheStats iterator getNext methods

Change-Id: Icf15c93b9ae3c3f523d0abe1580ce7280c5b0d84

  1. … 20 more files in changeset.
Change to avoid placing large scan results in RegionServer cache

By default the result of every Scan and Get request to HBase is placed in

the RegionServer cache. When a scan returns a lot of rows this can lead to

cache thrashing, causing results which are being shared by other queries

to be flushed out. This change uses cardinality estimates and hbase row size

estimates along with the configured size of region server cache size to

determine when such thrashing may occur. Heap size for region server is

specified through a cqd HBASE_REGION_SERVER_MAX_HEAP_SIZE. The units are in MB.

The fraction of this heap allocated to block cache is read from the config

file once per session through a lightweight (no disk access) JNI call. The

hueristic used is approximate as it does not consider total number of region

servers or that sometimes a scan may be concentrated in one or a few region

servers. We simply do not place rows in RS cache, if the memory used by all

rows in a scan will exceed the cache in a single RS. Change can be overridden

with cqd HBASE_CACHE_BLOCKS 'ON'. The default is now SYSTEM. Change applies

to both count(*) coproc plans and regular scans.

Change-Id: I0afc8da44df981c1dffa3a77fb3efe2f806c3af1

  1. … 20 more files in changeset.
port skew buster to Trafodion

1. add simplified TEST062

2. reuse cached partitioning expression when only doVarCharCast

is the same as the when the expression is created in

TableHashPartitioningFunction::createPartitioningExpressionImp().

3. rework

4. fix a time monitor bug reporting incorrect processor time (CPU time

computed from clock() calls)

5. comment out the assert on the size of NATable cache not decreasing.

This is to fix regression failure with seabase/TEST020. Selva will check

in a formal fix that allocates space for NATable objects from a single

heap, and then reenables this assert check.

Change-Id: I9eeee4f36ba8678e90e0ac68a85bfc733599d932

  1. … 26 more files in changeset.
Fix for NATable Heap leak

Earlier, each NATable entry had its own heap. Now, there is a NATable Heap

created for each compiler context. All the cached NATable object is

created from this heap.

This is done to reduce the memory usage in the compiler context and to

take care of leak of NATable heap.

Change-Id: I3ec362c0b8e5b1466956c3773c2b5dc7dccf0d52

  1. … 3 more files in changeset.
Pass object type to NATableDB:removeNATable

This change fixes a bug which can happen when indexes or

sequence generators are dropped and, due to NATable::objectUID_

not being initialize, the NATableDB:removeNATable method must

lookup the object UID from metadata. Before this change, the

code assumed anything looked up was a base table. Now the

object type is passed to NATableDB:removeNATable.

Change-Id: I4566ee68aa84367ea33c2e2bb1d3d6a346e8ce97

Closes-Bug: #1401999

  1. … 7 more files in changeset.
Lookup object UID for Sequence generator

The NATable object representing a Sequence generator sometimes has

an object UID of 0 at the time privileges are checked, resulting

in failure to locate the object and denial of access. This change

looks up the UID if the corresponding NATable member variable is 0.

Change-Id: I889d2d823b8c0faca5318572250b45e07134b6cf

  1. … 2 more files in changeset.
Fix performance regression due to QI for DDL

This check-in omits object UIDs from query plans for the tables

SB_HISTOGRAMS and SB_HISTOGRAMS_INTERVALS. Previously, when the

code generator tried to add the object UIDs for these, it had to

make a special query to the metadata, since the corresponding

internal cached structure omitted object UIDs when they were

created via methods like Generator::createVirtualTableDesc. The

special query to lookup these object UIDs was shown to be

responsible for a large pathlengh regression.

Change-Id: Id5046c5c55a4fc8dd2ba3f891449ea87d35a5534

Closes-Bug: #1398600

  1. … 7 more files in changeset.
Query Invalidation triggered by DDL, phase 1

This first check-in implements most of the framework which will

be used to complete the QI DDL feature. It redefines the old

security invalidation key (SQL_SIKEY) to handle DDL operations in

addition to REVOKE. In a limited number of DDL operations, the object

UIDs of affected Seabase objects are propagated to all nodes for

use by the compiler to invalidate NATable cache entries, as

well as a limited number of types of cached queries. Later this

month, the framework will be complete by allowing prepared queries

that have already been returned from the compiler to be invalidated.

Then the next step for the framework will be support for invalidating

the HTable cache. Finally an effort will be made to cover all of

the necessary DDL operations and all types of cached queries.

The check-in include a new regression test (executor/TEST122) that

demonstrates the cases that are covered. Specifically, a table will

be dropped and recreated with the same name but different definition

in one sqlci session. In another session, which has already populated

NATable cache and query cache for INSERT, UPDATE, DELETE, SELECT,

SELECT COUNT(*), INVOKE and SHOWDDL statements, those some types

of statements will be resubmitted and correctly compiled.

Change-Id: Ie61ce751089b57ce1894f1764c338e9400bb7b8a

Closes-Bug: #1329358

Implements: blueprint ddl-query-invalidation

  1. … 41 more files in changeset.
Use HBase estimated row count if no stats exist

In the absence of statistics, the row count for an HBase table

is estimated using information in its HFiles. Previouly, this

estimate was calculated and assigned to the NATable's

originalCardinality_ member variable, but this did not result

in it being used by the optimizer (the correct row count did

not appear in the results of an Explain). The code to calculate

the estimate is now invoked from HSHbaseTableDef::getRowCount(),

which has the effect of it being used as intended.

Change-Id: I3794e8368082b6f17e47fe686ec9a004aed27976

Closes-Bug: #1388918

  1. … 5 more files in changeset.
Set up security keys while constructing NATable

This change corrects a problem in NATable by setting up the

secKeySet_ ComSecurityKeySet while the NATable is constructed.

In this way, the NATable::resetAfterStatement method will not

flag it as a leak. This change also includes a new regression

test for REVOKE and query invalidation.

Change-Id: Ie2ac5a2380007a53ab13c95f154e02ad1bde191e

Closes-Bug: #1371199

  1. … 5 more files in changeset.
Ensure HFile.Reader objects are closed

Performance workloads are seeing mxosrvrs with a large accumulation

of open sockets, and a resultant decrease in performance. The cause

was HFile.Reader objects used in estimating a table's row count not

being closed. This fix executes a close() on the Reader in a finally

clause, such that it will always be executed even if an exception

is thrown.

Even with this fix, it was noted that early phases of workloads showed

a significant increase in compile times when row count estimation was

enabled. To address this, the estimated counts were saved so that they

would not be repeated as long as the NATable remained in cache. Also,

estimation was avoided completely for metadata tables. With these

changes, my own tests showed a very low overhead for estimation, on

the order of 60-70ms per user table in the workload, regardless of

query complexity or how many queries referenced the table. However,

the results of performance benchmarks executed on clusters did not

improve, and so this change also includes turning off the cqd that enables

row count estimation for the optimizer (a separate cqd controls use

of the feature for Update Statistics, and it is still on by default).

Future work will attempt to address the performance impact of turning

on the cqd.

Change-Id: Ied0369c8def5062d69766198155f8e309bae1ff8

  1. … 3 more files in changeset.
Merge "Support for SALTED index."

  1. … 4 more files in changeset.
Support for SALTED index.

CREATE INDEX supports a new clause "SALT LIKE TABLE".

This causes SALT column to be leading column in the index

Duplicate columns in the index table are now eliminated. This is a bugfix.

Showddl and Invoke will show the SALT syntax and column respectively for

the index. CREATE index also supports HBASE_OPTIONS clause.

A bug seen when NULLABLE partitioning columns are used is also fixed.

Patch Set 2 : All rework from Patch Set 1, excpt for NATable.cpp

Patch Set 3 : Rework in NATable.cpp. Thanks to Hans for all help. Nullable

partition key columns will now generate evenly distributed dataflow through

esps.

Patch Set 4: Fix 3 issues found for the work done in Patch Set 3. Change is only in NATable.cpp

Change-Id: If378ffca29ee83dd4b7928c784b8d34d76f50049

  1. … 25 more files in changeset.
Estmate NHase row count when stats not available

When statistics are not available for a table, give the optimizer

a better estimate of its cardinaly than the default value by

reading summary information from the trailer block of the table's

HFiles.

blueprint estimate-rowcount

Change-Id: Ie4a44bc7c87385c551eef96478147117b151ab9b

  1. … 6 more files in changeset.
Ensure UID returned for MD tables is non-null

When an NATable is created for a metadata table, the UID is not known.

Under certain conditions, this creates a problem whereby the AUTHS

table can not be read, and access to a table is incorrectly denied.

In this fix, the NATable::objectUid() function is changed to do a

lookup of the UID for a metadata table if it is 0, and to store the

result in NATable for future calls.

Change-Id: I5eb58871070e585a7242f92becde7e6a24a8f5f5

Closes-Bug: #1370289

  1. … 3 more files in changeset.
Security changes to support authorization

Added support for authorization commands:

- initialize authorization [, drop]

- create/drop roles

- register/unregister components

- create/drop component operations

- grant/revoke object privileges

- grant/revoke role privileges

- grant/revoke component privileges

- updates to GET and SHOWDDL statements

- checking of privileges for DML requests

- checking of privileges for DDL requests

- regression tests added to catman1 library

Fixed a testware problem in catman1 TEST135 and TEST139

Fixed a parser problem introduced by compGeneral/TESTTOK2 which was recently

introduced.

More details:

This delivery was part of code worked on by many people for several

months on a remote branch. This team held bi-weekly meetings

for several months to design and implement these features. These

meetings also included extensive code reviews.

The security features which include authentication (which was delivered

in June) and authorization is turned off by default. The

traf_authentication_setup script located in $MY_SQROOT/sql/scripts needs

to be run to enable both authentication and authorization. This

procedure is described on the Trafodion Twiki page and will be updated once this

delivery completed to include authorzation.

Delivery updates:

Updated traf_authentication_setup to return consistent error messages

and added a comment to ComSmallDefs.h to address a buf size issue for

metadata tables.

Change-Id: I896f1ee006590284653b2c9882901c05b5f2ba22

  1. … 100 more files in changeset.
Launchpad fixes_2

-- hbase access errors were not being returned during process bringup phase.

arkcmp/CmpContext.*, bin/SqlciErrors.txt

sqlcomp/CmpSeabaseDDLcommon.cpp, CmpSeabaseDDLtable.cpp, nadefaults.cpp

launchpad #1343061

-- showddl now shows hbase options, salted partitions.

-- hbase_options and salt info is kept in metadata.

comexe/ComTdb.h, optimizer/NATable.cpp, NATable.h

parser/sqlparser.y

sqlcat/desc.h, sqlcomp/CmpDescribe.cpp, CmpSeabaseDDLcommon.cpp

CmpSeabaseDDL.h, CmpSeabaseDDLTable.cpp

launchpad #1342465

launchpad #1342996

-- added support for update where current of cursor.

generator/GenRelScan.cpp, GenRelUpdate.cpp,

launchpad #1324679

-- check constrs in update and views

generator/GenRelUpdate.cpp, optimizer/BindRelExpr.cpp

Launchpad #1320034, #1321479

-- long column name returned assertion.

optimizer/ObjectNames.cpp

launchpad #1324689

-- purgedata does not recreate with the original salt and hbase options.

optimizer/RelExeUtil.cpp, sqlcomp/CmpSeabaseDDLcommon.cpp

launchpad #1322400

-- long keys got assertion failure:

optimizer/ValueDesc.cpp

launchpad #1332607

-- cqd hide_indexes was causing constraint creation to fail.

sqlcomp/CmpSeabaseDDLcommon.cpp

launchpad #1340385

-- current_timestamp/time functions are now non-nullable.

Change-Id: Ib3a071894d11d0e3719b98f0cfddfc5ce8624519

  1. … 26 more files in changeset.
fix bug 1343661(cleanup HBASE partitioning), bug 1347819 and bug 1343566

1343661: please refer to sql/sqlcomp/DefaultConstConstants.h for the

definition of CQD HBASE_PARTITIONING.

1347819: the change is in ExExeUtilLoad.cpp to disable HASH2 for fast load.

1343566: method NADefaults::getTotalNumOfESPsInCluster() now returns the

correct value if CQD PARALLEL_NUM_ESPS is set to an integer value.

Rework 1 to address Dave's review comments.

Rework 2 to address Khaled's review comments as follows. A Boolean flag

(isTrafLoadPrep_) is added to class BinWA to better control the type of

partitioning functions needed for the traf preparation step. When we are

binding all nodes, the flag is set to TRUE which instructs createNAFileSet()

not to create hash2.

Rework 3 to address Hans's review comments. If force to have hash2 and

the partitioning function in the cached table is range, do not return the

cached object.

Rework 4 to address seabase/TEST015 core, which is reported in bug

1349990. Bug 1349990 is fixed in this rework.

Change-Id: I9f886ce5361ff4ede16c592e8a3c4e5ca89d76b7

  1. … 19 more files in changeset.
metadata struct updates and fixes.

-- second arkcmp is no longer started to read hist tables.

-- metadata index change got lost in a previous change. That index has been added.

Will require a metadata upgrade from v2.2 to v2.3

-- metadata structs cleanup (no functionality change).

-- object_owner correctly set for indexes.

Change-Id: I78e06c70b67661b2881724b10ede8f6c188c65db

  1. … 16 more files in changeset.
Initial code drop of Trafodion

  1. … 4886 more files in changeset.