NATable.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-3280] Reduce path length in Trafodion for improved performance and scalability

There are many places in Trafodion code getenv is called repeatedly. Cleaned up code so that

repeated getenv calls are avoided during prepare and execute operations.

  1. … 11 more files in changeset.
jira TRAFODION-3157 Add support for BINARY/VARBINARY datatype

Spec attached to jira.

new test regress/seabase/TEST004 added.

  1. … 86 more files in changeset.
[TRAFODION-3234] Refactor hive meta calls to be less resource intensive to support hive partitions if needed

There was yet another method that called HiveMetaStoreClient.listPartitions API. This method is

called whenever the hive table entry is picked from the NATable cache.

Replaced listParitions call with Trafodion specific HDFSClient.getHiveTableModificationTS call

that is less expensive.

HiveMetaStoreClient is now made as ThreadLocal variable to ensure that HiveMetaStoreClient methods

can be invoked in a thread safe manner.

  1. … 6 more files in changeset.
[TRAFODION-3234] Add support for hive partitioned tables

Refactored the interaction with hive metastore to use the efficient APIs to obtain the necessary

information to construct the hive table descriptors. This refactoring is expected to reduce the

memory requirements, efficient and high performing even when there are thousands of partitions.

  1. … 14 more files in changeset.
TRAFODION - 3218 User still has privilege after user's role has been revoked ...

Partial support for column level privileges with QI support for:

column select

column insert

column references

column update

Also, as part of this, updated privilege code in a couple of areas:

Changed object caching code in NATable and NARoutine to store all privileges

assigned to the object when the object is cached (privDescs_). During the load

operation, the code creates bitmaps (privInfo_) for the current user. Privilege

checks are performed against the user bitmaps (privInfo_). This is in

anticipation for some performance updates when connecting to Trafodion (mxosrvr)

with different users.

Change getRoleList to include the roleID and the granteeID that granted the

privilege. The grantee can be a user or a role.

When a privilege is revoked from a role, send QI keys for every user that has

been granted to role.

  1. … 40 more files in changeset.
Merge remote branch 'origin/master' into ansharma_nnpk_br

Conflicts:

core/sql/regress/seabase/EXPECTED031

core/sql/sqlcomp/CmpDescribe.cpp

core/sql/sqlcomp/CmpSeabaseDDLtable.cpp

core/sql/sqlcomp/DefaultConstants.h

  1. … 12 more files in changeset.
nonnull pkey, commit #1

  1. … 22 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 90 more files in changeset.
merge [TRAFODION-3046] pr 1561 Privilege support for native HBase tables

  1. … 1 more file in changeset.
TRAFODION-3046: Privilege support for native HBase tables

-- Grants and revokes against native HBase tables are enforced similar to hive

-- Privilege checking added when creating and dropping native HBase tables

-- Removing dependent Trafodion metadata when native HBase tables are dropped

-- Added regression test (privs2/TEST146)

-- Reorg - split PrivMgrComponents into 2 files:

PrivMgrComponents and PrivMgrUserPrivs

An hbase table can be referenced using one of the following types:

"_CELL_" - references cell data

"_ROW_" - references row data

"_MAP_" - references data defined by the mapped (external) table

Privileges are granted against each of these types, so if you:

select * from hbase."_CELL_".hbase1;

You must have the select privileges on this table

For example: "grant select on hbase."_CELL_".hbase1 to user1"

Likewise for other hbase types, grants are required to gain accessibility

grant select on hbase."_MAP_".hbase1 to user1

grant select on hbase."_ROW_".hbase1 to role1

  1. … 25 more files in changeset.
[TRAFODION-3052] Fix core in PCode evaluator

  1. … 3 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or error in

all the utility commands except load. In case of load, the ComDiagsArea

is allocated in advance to report error rows count.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

[TRAFODION-3017] Simplify the hive client access in Trafodion

Hive Client functions are now moved to a new file HiveClient_JNI.h and

HiveClient_JNI.cpp. Most of the HiveClient functions are static functions

allowing to use HiveClient in Trafodion with ease.

  1. … 33 more files in changeset.
[TRAFODION-2821] Trafodion core code base needs to be thread safe

Java exceptions thrown while calling the java methods from JNI in

Trafodion are stored in the current context. However in a multi-threaded

ESP environment, these exceptions should be stored in thread specific

variable to enable error handling to be thread safe. Otherwise, the JNI

errors could be overwritten by the JNI errors from another thread.

Also streamlined JNI error handling by extending the getExceptionDetails()

methods to log the errors also.

Incorporated error handling in SequenceFileReader JNI methods.

  1. … 22 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 15 more files in changeset.
fix the bufoverrun Critical error checked by TScanCode

  1. … 20 more files in changeset.
[TRAFODION-2838] Add error check to NATableDB::get

[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 4. Remove legacy/obsolete pragmas

  1. … 392 more files in changeset.
lcov: commit #1

  1. … 333 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
[TRAFODION-2727] Memory leak in the compiler part of the code in Trafodion

Compiler objects were refactored to use the heap infrastructure to ensure

that it is accounted and allocated within the trafodion memory management.

The commonly used NAType and derived classes are mostly created via heap,

but the heap was not passed in the constructor. The embedded objects

used the system heap though the container object is allocated in the compiler

heap. This could result in memory leak when the container object is

destroyed.

  1. … 63 more files in changeset.
[TRAFODION-2617] Optionally push estimate row count logic into HBase coproc

  1. … 10 more files in changeset.
[TRAFODION-2628] SQL engine work for Elasticity

It turns out that the compiler already kept a list of node ids,

not assuming that node ids are consecutively numbered 0...n-1.

However, this list was contained in lots of other data structures

that are dead code, coming from the time before Trafodion was

open-sourced. The change does two things: a) getting rid of obsolete

info and b) maintining a list of node ids in the NAClusterInfo

class.

This change does not include executor modifications. The executor

might work just fine, or we may need to add a fix (or optimization)

for cases where we pick a node id for a new process at runtime.

  1. … 12 more files in changeset.
[TRAFODION-2674] Create index sees error 2006 internal assertion

RangePartitionFunction was missing region boundaries for indexes and using table's

boundaries incorrectly

Turned OFF OPT_PCODE_CACHE to avoid mem leak.

Unrelated testware change for compGeneral/TEST023

  1. … 7 more files in changeset.
ra TRAFODION-2584 Add support to register hive schemas in traf MD

This checkin extends jira TRAFODION-2584 to add support to register

hive schemas(referred to as databases in hive) in trafodion metadata.

Until now, only hive tables and views were registered.

Registering hive schemas would allow certain operations, like

granting/revoking privileges, to be done on a hive schema/database.

Syntax to register/unregister hive schema

-- register/unregister hive schema hive.sch;

Syntax to show registred hive schemas

-- get hive registered schemas in catalog trafodion

"showddl schema hive.sch" will now show the ddl for hive database

creation and whether this schema is registered.

showddl and 'get tables in schema' will now return error if that

schema doesn't exist.

regress/hive/TEST007 has been extended with additional tests.

  1. … 15 more files in changeset.
[TRAFODION-2635] Fix two bugs concerning HBASE CELL access on Traf objects

  1. … 13 more files in changeset.
TRAFODION-2610 Register native HBase tables in traf MD plus couple more

-- support for registering hbase native cell/row format tables in traf MD

Syntax:

register [internal] hbase table [if not exists] <table-name>

unregister [internal] hbase table [if exists] <table-name>

-- support to get info related to registered hbase tables

Syntax: get hbase registered table in catalog trafodion

-- enhancements to HBase mapped tables and bug fixes

-- disabling of creation of external tables on native hbase

tables unless they are mapped tables

-- support for case-insensitive format string in TO_DATE function

(exp/exp_datetime.h)

-- regress/seabase/TEST022 enhanced with additional tests

  1. … 48 more files in changeset.
[TRAFODION-2584] Add support to register hive objects in traf metadata

Syntax:

register hive table/view [if not exists] <object-name> [cascade]

unregister hive table/view [if exists] <object-name> [cascade]

cascade option: register/unregister all underlying objects that are

part of the specified view

update statistics, grant/revoke, traf views or external table creation

on hive objects will automatically and internally register those objects

in trafodion metadata.

invoke/showddl will show if this object is registered and whether

that registration was internal or explicit.

Get command extensions:

get hive registered tables/view/objects in catalog trafodion;

get hive external tables in catalog trafodion;

Cleanup command extensions:

cleanup metadata command will clean up inconsistent hive objects

(underlying hive object is missing but object is registered or

an external table exists)

cleanup [hive table | hive view] on <object-name>;

Existing hive objects that had implicit or explicit external tables

created prior to this checkin will have no change in behavior.

ObjectUID of those external tables will continue to be used for

relevant operations.

One can drop those external tables and explicitly register them,

or a subsequent operation(upd stats, grant, etc) that needs objectUID will

automatically register them.

minor changes based on review comments of previous checkin

get all objects command on hive metadata no longer fails.

get views on objects return 3-part name that could be used to

differentiate between a traf and hive view.

regress/hive/TEST007 has been extended.

TBD: Add register/unregister privileges

  1. … 52 more files in changeset.
[TRAFODION-2583] Remove deprecated CQD HIVE_MAX_STRING_LENGTH

  1. … 29 more files in changeset.
[TRAFODION-2529] Pass proper table name to getRowCount for _CELL_ and _ROW_

  1. … 2 more files in changeset.