ComTdbExeUtil.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix for TRAFODION-3112

Internal error: get ... for user/role

Heading incorrect for libraries

Parser error: get procedures/table_mapping functions/functions for user/role

  1. … 11 more files in changeset.
TRAFODION-3086 Further enhancements to 'DDL on Hive objects'

-- Hive CTAS enhancements

-- CTAS will do create through Hive and insert...select in Traf

-- a cqd can be set if CTAS need to be passed in completely to Hive

-- CTAS 'no load' option can be used to create a Hive

table LIKE a traf table.

Older 'create hive like traf' is removed.

-- explain for CTAS to show ddl, insert...select, upd...stats stmts

-- Truncate enhancements.

-- traf: purgedata, truncate, truncate table

-- Hive: truncate, truncate table

-- 'if exists' option has been added

-- Hive truncate is through new exeutil operator

-- older operator uses LOB interface to clear data.

Maintained as Legacy operator, will be removed.

-- new operator uses Hive to truncate tables

-- explain for truncate to show trunc query that will be sent

-- removed obsolete parallel label op

-- removed obsolete purgedata (fast delete) op

  1. … 30 more files in changeset.
Merge remote branch 'origin/master' into ansharma_hiveddl_br

  1. … 7 more files in changeset.
Merge remote branch 'origin/pr/1586/head' into merge_1586

  1. … 5 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 90 more files in changeset.
Support to provide a locking mechanism for LOB insert/update operations

  1. … 28 more files in changeset.
TRAFODION-1573: Additional GET commands for privileges TRAFODION-3074: Failed to register/unregister user when security disabled

TRAFODION-1573 changes:

- Added support for the following commands:

get privileges on <object>

<object>: [LIBRARY | PROCEDURE | FUNCTION | TABLE_MAPPING FUNCTION | SEQUENCE]

- Added support for the FOR CLAUSE on all supported objects

Removed the need to specify keyword 'USER" before username. If USER is

included, then it is ignored.

get privileges on <object> FOR [USER] <user or role name>

<objects>: [TABLES, VIEWS, LIBRARIES, PROCEDURES, FUNCTIONS,

TABLE_MAPPING FUNCTIONS, SEQUENCES]

- The following get command can only be run by DB__ROOT or a user that has been

granted the DB__ROOTROLE or DB__HIVEROLE role

get <objects> in schema hive.xx.xx;

<objects>: [TABLES, OBJECTS, VIEWS]

- The following get command can only be run by DB__ROOT or a user that has been

granted the DB__ROOTROLE or DB__HBASEROLE role

get external hbase objects;

- The following get commands retrieve privilege details from Trafodion metadata;

users can only see objects where they have been granted at least one privilege

get hive registered tables in catalog trafodion;

get hbase registered tables in catalog trafodion;

- get privileges commands now return owner's privileges in output

- Cleaned up code in the parser.

TRAFODION-3074 changes

- register user - fixed query to find next available authID

- unregister user - added checks to not read privilege metadata if authorization

is not enabled

  1. … 17 more files in changeset.
Merge remote branch 'origin/pr/1538/head' into merge_1538

  1. … 2 more files in changeset.
Removing unneeded CLI call to get lob location and instead using the CQD value - LOB_STORAGE_LOCATION to get/save it in Tdb for use later at runtime.

  1. … 3 more files in changeset.
Merge [TRAFODION-2175] PR 1512 Get statement enhancements

  1. … 2 more files in changeset.
Get statement enhancements

Added support and privilege checks for the following commands:

get functions for library

get procedures for library

get table_mapping functions for library

get indexes on table

get objects on table

get views on table

get views on view

get libraries in schema

get objects in view

get tables in view

get views in view

get indexes for user

get tables for user

get libraries for user

get views for user

Changed "get libraries for schema" to include libraries where the current user

has execute privilege on one of the libraries routines (functions, procedures,

or table_mapping functions).

Addressed a performance issue when determining if the user has column level

privileges. If the user has granted privileges against native Hive tables

through EsgynDB, we need to get the column name from Hive. The call to get the

column, by calling hivemd, is very expensive. This change checks to see if the

requested user has been granted any column level privileges on a hive table.

If so, we will go ahead and do the mapping (call hivemd). If not, then we will

not include the hivemd fragment for the query. Since we are scanning the column

privileges table anyway, we also see if the requested user (or their roles) has

been granted any privileges. If so, we include the column privileges check in

the query.

Commented out get statements that we do not support at this time.

  1. … 8 more files in changeset.
review changes from PR 1502, plus jenkins compGeneral crash fix

  1. … 4 more files in changeset.
various fixes

-- JIRA 2980 support for INSTR function.

INSTR(source_string, pattern, startPos, occurrence)

Search for pattern in source_string.

Start at startPos'th character. Optional, default 1

Return occurrence'th occurrence. Optional, default 1

(exp/exp_function.cpp/h, exp/ExpPCodeClauseGen.cpp,

generator/GenItemFunc.cpp, optimizer/BindItemExpr.cpp, ItemExpr.cpp,

optimizer/ItemFunc.h, SynthType.cpp, parser/sqlparser.y)

-- hivemd returns precision, scale, display_datatype

(comexe/ComTdbExeUtil.h, common/BaseTypes.cpp,NAType.cpp/h,

executor/ExExeUtilGet.cpp, ExExeUtil.h)

-- hive decimal type is treated as sql NUMERIC for performance.

(common/NAType.cpp)

-- hive select from hivemd with prepare and multiple executes work

(executor/ExExeUtilGet.cpp)

-- fix for crash in sql buffer pool when allocating large size rows.

(executor/sql_buffer.cpp)

-- SIGN(op) now returns error if op is not numeric

(BindItemExpr.cpp)

-- strings of length zero are no longer cached. Caching them causes problem

during backpatching.

(optimizer/ItemCache.cpp)

--- compile time predicates on constants were not being created correctly

in some cases. That has been fixed.

(ValueDesc.cpp)

-- trim now supported on scaled numeric, float and bignum datatypes

(BindItemExpr.cpp)

-- CASE is now supported on scaled numerics, float, bignum

(BindItemExpr.cpp)

-- char_length is now supported on numerics

(ItemFunc.h, BindItemExpr.cpp)

-- DAYOFMONTH now allowed only on datetime datatypes

(common/OperTypeEnum.h, sqlparser.y, BindItemExpr.cpp, ItemExpr.cpp)

-- TO_TIME now allowed only on 'datetime with time' or character datatypes.

(BindItemExpr.cpp, bin/SqlciErrors.txt)

-- updated expected file for compGeneral/test042

  1. … 31 more files in changeset.
New syntax to retrieve the LOB HDFS filename for both external and internal LOBs . Also added syntax to return starting offset of a particular LOB handle in the LOB Hdfs data file.

  1. … 15 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 4. Remove legacy/obsolete pragmas

  1. … 392 more files in changeset.
[TRAFODION-2754] Get statistics cores sqlci or mxosrvr at str_sprintf()

Switched str_sprinf to use the standard sprintf function to ensure

that the format specification and the passed in parameters are consistent

  1. … 65 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
Few fixes, details listed below.

-- fix an issue where multiple values inserted from a list would return

error but each value inserted on its own would succeed.

ex: create table ts (a timestamp);

insert into ts values ('2017-01-01 10:10:10'), ('2018-01-01 10:10:10');

-- sometimes errors returned from child during hive inserts were not

being returned. That has been fixed.

-- TRAFODION-2683 extension.

added a 'p' (prune) option which would cleanse and filter unneeded

explain output. This helps in reducing output especially

for larger explains.

Ex:

>>explain option 'p' select * from dual;

------------------------------------------------------ PLAN SUMMARY

STATEMENT_NAME ........... NOT NAMED

STATEMENT ................ select * from dual;

------------------------------------------------------- NODE LISTING

ROOT ====================================== SEQ_NO 2 ONLY CHILD 1

DESCRIPTION

fragment_id ............ 0

parent_frag ............ (none)

fragment_type .......... master

xn_access_mode ......... read_only

auto_query_retry ....... enabled

embedded_arkcmp ........ used

select_list ............ %(0)

input_variables ........ %(0), %(0), %(0)

VALUES ==================================== SEQ_NO 1 NO CHILDREN

DESCRIPTION

fragment_id ............ 0

parent_frag ............ (none)

fragment_type .......... master

tuple_expr ............. %(0)

--- SQL operation complete.

>>

  1. … 19 more files in changeset.
TRAFODION-2683 add a new explain option to mask variant fields in output

--Syntax extension to cleanse and mask:

explain options 'c' select ...

explain options 'fc' select...

prepare s from select ...

explain option 'c' s;

--Masked fields will show up as "###" in explain output.

--Both option and options keyword can be used in explain and showplan.

-- explain option/options ...

-- showplan option/options ...

-- showplan can be used on explain statement.

-- Here is an example of what the output will look like:

>>explain option 'c' select * from dual;

----------------------------------------------- PLAN SUMMARY

MODULE_NAME .............. DYNAMICALLY COMPILED

STATEMENT_NAME ........... NOT NAMED

PLAN_ID ................ ###

ROWS_OUT ............... ###

EST_TOTAL_COST ......... ###

STATEMENT ................ select * from dual;

------------------------------------------- NODE LISTING

ROOT ================================ SEQ_NO 2 ONLY CHILD 1

REQUESTS_IN ............ ###

ROWS_OUT ............... ###

EST_OPER_COST .......... ###

EST_TOTAL_COST ......... ###

DESCRIPTION

max_card_est ......... ###

fragment_id ............ 0

parent_frag ............ (none)

fragment_type .......... master

statement_index ........ 0

affinity_value ....... ###

max_max_cardinality ###

total_overflow_size ###

xn_access_mode ......... read_only

xn_autoabort_interval 0

auto_query_retry ....... enabled

plan_version ....... 2,600

embedded_arkcmp ........ used

select_list ............ %(0)

input_variables ........ %(0), %(0), %(0)

VALUES ======================== SEQ_NO 1 NO CHILDREN

REQUESTS_IN ............ ###

ROWS_OUT ............... ###

EST_OPER_COST .......... ###

EST_TOTAL_COST ......... ###

DESCRIPTION

max_card_est ......... ###

fragment_id ............ 0

parent_frag ............ (none)

fragment_type .......... master

tuple_expr ............. %(0)

--- SQL operation complete.

>>

  1. … 21 more files in changeset.
ra TRAFODION-2584 Add support to register hive schemas in traf MD

This checkin extends jira TRAFODION-2584 to add support to register

hive schemas(referred to as databases in hive) in trafodion metadata.

Until now, only hive tables and views were registered.

Registering hive schemas would allow certain operations, like

granting/revoking privileges, to be done on a hive schema/database.

Syntax to register/unregister hive schema

-- register/unregister hive schema hive.sch;

Syntax to show registred hive schemas

-- get hive registered schemas in catalog trafodion

"showddl schema hive.sch" will now show the ddl for hive database

creation and whether this schema is registered.

showddl and 'get tables in schema' will now return error if that

schema doesn't exist.

regress/hive/TEST007 has been extended with additional tests.

  1. … 15 more files in changeset.
TRAFODION-2610 Register native HBase tables in traf MD plus couple more

-- support for registering hbase native cell/row format tables in traf MD

Syntax:

register [internal] hbase table [if not exists] <table-name>

unregister [internal] hbase table [if exists] <table-name>

-- support to get info related to registered hbase tables

Syntax: get hbase registered table in catalog trafodion

-- enhancements to HBase mapped tables and bug fixes

-- disabling of creation of external tables on native hbase

tables unless they are mapped tables

-- support for case-insensitive format string in TO_DATE function

(exp/exp_datetime.h)

-- regress/seabase/TEST022 enhanced with additional tests

  1. … 48 more files in changeset.
[TRAFODION-2584] Add support to register hive objects in traf metadata

Syntax:

register hive table/view [if not exists] <object-name> [cascade]

unregister hive table/view [if exists] <object-name> [cascade]

cascade option: register/unregister all underlying objects that are

part of the specified view

update statistics, grant/revoke, traf views or external table creation

on hive objects will automatically and internally register those objects

in trafodion metadata.

invoke/showddl will show if this object is registered and whether

that registration was internal or explicit.

Get command extensions:

get hive registered tables/view/objects in catalog trafodion;

get hive external tables in catalog trafodion;

Cleanup command extensions:

cleanup metadata command will clean up inconsistent hive objects

(underlying hive object is missing but object is registered or

an external table exists)

cleanup [hive table | hive view] on <object-name>;

Existing hive objects that had implicit or explicit external tables

created prior to this checkin will have no change in behavior.

ObjectUID of those external tables will continue to be used for

relevant operations.

One can drop those external tables and explicitly register them,

or a subsequent operation(upd stats, grant, etc) that needs objectUID will

automatically register them.

minor changes based on review comments of previous checkin

get all objects command on hive metadata no longer fails.

get views on objects return 3-part name that could be used to

differentiate between a traf and hive view.

regress/hive/TEST007 has been extended.

TBD: Add register/unregister privileges

  1. … 52 more files in changeset.
Changed return lobtype iafter prepare during describe, to be FS type 160.Changed a few limits for lob chunksize to handle large files faster. Changed sqlci to handle param input as LOB.

  1. … 10 more files in changeset.
Rework the fix for [TRAFODION-2516] to returntype for param as LOB instead of coercing the type by default to char type since it confuses JDBC and Catalog APIs. The expected input param still is expected as a varchar format. Also added testcases both for the regression as well as the clitestdriver program.

  1. … 11 more files in changeset.
[TRAFODION-2420] RMS enhancements Introduced 2 new counters HeapWM: - "SQL Heap WM" - Tracks the high water mark of the used memory including space buffer and bmo operator memory. It is enough to monitor this counter alone to track the memory utilization of the query.

scrIOTime: "Scr. IO Time" - Tracks the IO time taken to read/write to

the scratch files at BMO operator level

Introduced a 'Single line Format' option to display BMO stats and SE stats

in a single line format to enable exporting the output to Excel easily.

GET STATISTICS FOR QID CURRENT PROGRESS, OPTIONS 'SL'

Removed and cleaned up the code further

Also fixed memory corruption issues seen with GET STATISTICS command.

  1. … 16 more files in changeset.
TRAFODION-2498 Add support to run hive stmts from traf interface

Syntax:

process hive statement '<string>';

<string>: hive statement starting with create/drop/alter/truncate.

These are the only stmts currently supported.

Ex:

>>process hive statement 'create database trafsch';

will create hive database 'trafsch'

>>process hive statement 'create table trafsch.t (a int)';

will create hive table 't' in hive schema 'trafsch'.

'process hive statment ..' could be issued from any traf interface

(sqlci/trafci/jdbc...)

  1. … 24 more files in changeset.
Merge [TRAFODION-2492] PR-976 Support for native hive views, plus few more changes

  1. … 3 more files in changeset.
TRAFODION-2492 Support for native hive views, plus few more changes

-- added support for native hive views created externally. These views

can be accessed in a traf sql query.

There are some restrictions as listed below:

-- privileges support has not yet been added. That will be done, as

needed, in the next checkin.

-- usage info for tables contained in a native hive view is not maintained.

-- native hive views cannot be inserted/updated/deleted

-- cannot update stats on a native hive view

-- support for GET statements to retrieve hive data

get tables/views/objects in schema hive.sch;

-- this will return all tables/views in specified schema(hive database)

get tables/views/objects in catalog hive;

-- this will return all tables/views in hive

get schemas in catalog hive;

-- this will return all schemas(databases) that exist in hive

-- GET statements to retrieve hive data now supports optional MATCH clause.

This can be used to filter results.

-- showddl will show if an external table on hive was created implicitly

for internal usage (upd stats, privileges, views) or explicitly by user.

Internal creation of external table now use 'create implicit external table'

syntax and that info is stored in metadata.

-- drop external table will drop that table even if underlying hive table

is missing. Currently it returns an error.

-- at runtime, accessing a missing hive table now returns a better error

message instead of an internal "data mod check failed" message.

-- regress/tools/runallsb has 2 new options:

-basetests (this will run core,compGeneral,executor,seabase,hive)

-othertests (this will run all other tests suites)

-- optimizer/NATable.cpp was incorrectly indented by some checkin from

a few days back. TABs were used instead of spaces which caused code

to become all crooked. That has been fixed.

-- regress/hive/TEST007 has been updated with additional hive view tests

  1. … 26 more files in changeset.
TRAFODION-2441 user has only select privilege on a table can do ...

This is the second delivery that updates "get" commands so users can only view

information where they have been granted privileges. This delivery handles:

get components;

get privileges on component;

get privileges on component for <role>;

get privileges on component for <user> [cascade];

cascade returns privileges for user plus any roles the user has been granted

  1. … 5 more files in changeset.
TRAFODION-2175 a user should only see specific schemas/tables that he has privs TRAFODION-1573 Additional GET commands for privileges

This is one of several deliveries to improve GET commands:

- return details about objects that are only visible to the current_user.

- support additional statements

This delivery adds visibility checking to to the following commands:

get users;

get roles;

get users for role <role>;

get roles for user <user>;

It now supports the following GET statements:

get privileges for user <user>;

get privileges for role <role>;

When getting privileges for a user or role, the output is as follows:

<privileges> <object> <optional column name>

<privileges> output is a character string containing granted privileges:

char 1: SELECT_PRIV - "S"

char 2: INSERT_PRIV - "I"

char 3: DELETE_PRIV - "D"

char 4: UPDATE_PRIV - "U"

char 5: USAGE_PRIV - "G"

char 6: REFERENCES_PRIV - "R"

char 7: EXECUTE_PRIV - "E"

If the user/role has no privilege then "-" is specified for the character

Example: get privileges for user userx;

SI--G-E TRAFODION.MYSCH.MYTABLE1

S------ TRAFODION.MYSCH.MYTABLE2 <Column> column1

S--U--- TRAFODION.MYSCH.MYTABLE2 <Column> column2

When getting privileges for a role, only those privileges granted to the role

are returned.

When getting privileges for a user, privileges directly granted to the user

plus privileges granted to any roles associated with the user are returned.

A new regression test - privs1/TEST123 has been added to test these changes.

In addition, unused methods were removed from Get code.

  1. … 4 more files in changeset.