ExExeUtil.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
fix regression failures

  1. … 7 more files in changeset.
fix issues for local var and support complex get command

  1. … 1 more file in changeset.
TRAFODION-3121 Enhance 'initialize trafodion' to return progress status

-- init traf will now return status during various steps.

'Started' indication will be returned when that step starts.

'Completed' will be returned when it completes successfully.

'Failed' will be returned if that step fails.

Elapsed time for each step will be shown in {ET hh:mm:ss} format.

-- method CmpSeabaseDDL::initSeabaseMD is rewritten and moved from

file CmpSeabaseDDLcommon.cpp to new file CmpSeabaseDDLinitraf.cpp.

-- priv mgr tables are now always created during init traf.

-- a 'no return status' option has been added if status is not to be

returned during init traf. This is used when init traf is called

internally, for ex, during upgrade.

It is not an externalized option.

-- Output of init traf is shown below.

Note: output is not a printf/cout to stdout.

It is returned as selected rows to the caller.

Apache Trafodion Conversational Interface 2.3.0

Copyright (c) 2015-2017 Apache Software Foundation

>>initialize trafodion;

Initialize Trafodion: Started

Verify User: Started

Verify User: Completed {ET: 00:00:00}

Version Check: Started

Version Check: Completed {ET: 00:00:00}

Create Metadata Tables: Started

Create Metadata Tables: Completed {ET: 00:00:49}

Update Metadata Tables: Started

Update Metadata Tables: Completed {ET: 00:00:06}

Create Schema Objects: Started

Create Schema Objects: Completed {ET: 00:00:20}

Create Metadata Views: Started

Create Metadata Views: Completed {ET: 00:00:05}

Create Repository Tables: Started

Create Repository Tables: Completed {ET: 00:00:16}

Create PrivMgr Tables: Started

Create PrivMgr Tables: Completed {ET: 00:00:29}

Create Library Manager: Started

Create Library Manager: Completed {ET: 00:00:51}

Initialize Trafodion: Completed {ET: 00:03:09}

--- SQL operation complete.

  1. … 36 more files in changeset.
TRAFODION-3086 additional fixes

  1. … 7 more files in changeset.
TRAFODION-3086 Further enhancements to 'DDL on Hive objects'

-- Hive CTAS enhancements

-- CTAS will do create through Hive and insert...select in Traf

-- a cqd can be set if CTAS need to be passed in completely to Hive

-- CTAS 'no load' option can be used to create a Hive

table LIKE a traf table.

Older 'create hive like traf' is removed.

-- explain for CTAS to show ddl, insert...select, upd...stats stmts

-- Truncate enhancements.

-- traf: purgedata, truncate, truncate table

-- Hive: truncate, truncate table

-- 'if exists' option has been added

-- Hive truncate is through new exeutil operator

-- older operator uses LOB interface to clear data.

Maintained as Legacy operator, will be removed.

-- new operator uses Hive to truncate tables

-- explain for truncate to show trunc query that will be sent

-- removed obsolete parallel label op

-- removed obsolete purgedata (fast delete) op

  1. … 30 more files in changeset.
Merge remote branch 'origin/master' into ansharma_hiveddl_br

  1. … 7 more files in changeset.
Address review comments

  1. … 5 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 90 more files in changeset.
Support to provide a locking mechanism for LOB insert/update operations

  1. … 28 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or errors in

all the utility commands.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

  1. … 14 more files in changeset.
Get statement enhancements

Added support and privilege checks for the following commands:

get functions for library

get procedures for library

get table_mapping functions for library

get indexes on table

get objects on table

get views on table

get views on view

get libraries in schema

get objects in view

get tables in view

get views in view

get indexes for user

get tables for user

get libraries for user

get views for user

Changed "get libraries for schema" to include libraries where the current user

has execute privilege on one of the libraries routines (functions, procedures,

or table_mapping functions).

Addressed a performance issue when determining if the user has column level

privileges. If the user has granted privileges against native Hive tables

through EsgynDB, we need to get the column name from Hive. The call to get the

column, by calling hivemd, is very expensive. This change checks to see if the

requested user has been granted any column level privileges on a hive table.

If so, we will go ahead and do the mapping (call hivemd). If not, then we will

not include the hivemd fragment for the query. Since we are scanning the column

privileges table anyway, we also see if the requested user (or their roles) has

been granted any privileges. If so, we include the column privileges check in

the query.

Commented out get statements that we do not support at this time.

  1. … 8 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

Fix for hive/TEST007 failure

Merge branch 'master' of github.com:apache/trafodion into trafodion-3009_1

Conflicts:

core/sql/executor/ExExeUtilGet.cpp

  1. … 3 more files in changeset.
various fixes

-- JIRA 2980 support for INSTR function.

INSTR(source_string, pattern, startPos, occurrence)

Search for pattern in source_string.

Start at startPos'th character. Optional, default 1

Return occurrence'th occurrence. Optional, default 1

(exp/exp_function.cpp/h, exp/ExpPCodeClauseGen.cpp,

generator/GenItemFunc.cpp, optimizer/BindItemExpr.cpp, ItemExpr.cpp,

optimizer/ItemFunc.h, SynthType.cpp, parser/sqlparser.y)

-- hivemd returns precision, scale, display_datatype

(comexe/ComTdbExeUtil.h, common/BaseTypes.cpp,NAType.cpp/h,

executor/ExExeUtilGet.cpp, ExExeUtil.h)

-- hive decimal type is treated as sql NUMERIC for performance.

(common/NAType.cpp)

-- hive select from hivemd with prepare and multiple executes work

(executor/ExExeUtilGet.cpp)

-- fix for crash in sql buffer pool when allocating large size rows.

(executor/sql_buffer.cpp)

-- SIGN(op) now returns error if op is not numeric

(BindItemExpr.cpp)

-- strings of length zero are no longer cached. Caching them causes problem

during backpatching.

(optimizer/ItemCache.cpp)

--- compile time predicates on constants were not being created correctly

in some cases. That has been fixed.

(ValueDesc.cpp)

-- trim now supported on scaled numeric, float and bignum datatypes

(BindItemExpr.cpp)

-- CASE is now supported on scaled numerics, float, bignum

(BindItemExpr.cpp)

-- char_length is now supported on numerics

(ItemFunc.h, BindItemExpr.cpp)

-- DAYOFMONTH now allowed only on datetime datatypes

(common/OperTypeEnum.h, sqlparser.y, BindItemExpr.cpp, ItemExpr.cpp)

-- TO_TIME now allowed only on 'datetime with time' or character datatypes.

(BindItemExpr.cpp, bin/SqlciErrors.txt)

-- updated expected file for compGeneral/test042

  1. … 31 more files in changeset.
[TRAFODION-3009] Streamline error handling in Executor utility commands

ComDiagsArea is now allocated only when there are warnings or error in

all the utility commands except load. In case of load, the ComDiagsArea

is allocated in advance to report error rows count.

This requires all the executor utility commands to use a new version of

ExRaiseSqlError to populate diagnostics area.

[TRAFODION-3017] Simplify the hive client access in Trafodion

Hive Client functions are now moved to a new file HiveClient_JNI.h and

HiveClient_JNI.cpp. Most of the HiveClient functions are static functions

allowing to use HiveClient in Trafodion with ease.

  1. … 33 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 41 more files in changeset.
Merge PR 1417 [TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 9 more files in changeset.
New syntax to retrieve the LOB HDFS filename for both external and internal LOBs . Also added syntax to return starting offset of a particular LOB handle in the LOB Hdfs data file.

  1. … 15 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-1 changes.

Created a new class org.trafodion.sql.HDFSClient. Any direct HDFS access

will be routed to this class via JNI instead of using libhdfs.

Modified the existing code expect for the following to route the HDFS request via this class

1. LOB access

2. Direct HDFS scan of the table

3. Sample data creation during update stats

Added a new class org.trafodio.sql.HdfsScan for scanning one or many ranges of a Hive

text formatted table. This class will be used for Direct HDFS scan in near future.

  1. … 21 more files in changeset.
TRAFODION-2873. Fix for avoiding cores during expression eval for inserts into LOB tables. The fix involves cleaning up the LOBGlibals and eliminating unneeded datamemebers and keeping only the ExLobGlobals members. Also cleaned up all LobLoad related code since it is obsolete and worked only with DP2.

Conflicts:

core/sql/executor/ExExeUtilLoad.cpp

  1. … 9 more files in changeset.
[TRAFODION-2420] RMS enhancements

The float type counters such as Estimated Accesssed Rows, Estimated Rows used and

Estimated Memory are not formatted correctly after sprintf changes.

This resulted in RMS output to be messed up making it diffcult to read and analyze

the output.

(cherry picked from commit 41c0ca7067c1e0097c05e1c823ea26eb2a828591)

  1. … 1 more file in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
Merge remote branch 'origin/pr/1216/head' into merge_1216

  1. … 11 more files in changeset.
Main change:

Added a method for JDBC driver to be able to close an open cursor abruptly by passing in a 0 length to the extract syntax. This will ensure internal opens are not left around if the application does not retrieve all the data with the stream API.

This abrupt close can be called when the result set is closed or the connection is closed by keeping track of all the open Blob objects. (in the driver)

Added a freeResources method and destructor to LobExtract and LobUpdate Tcbs.

Minor changes and fixes :

Refactored the ExLob::closeCursor

Removed the LOB_CLI_SELECT_CLOSE call from ::fetchCursor so it gets closed only when ExLob::closeCursor is called. Makes the code consistent and readable.

Took care of a few memory allocations and deletions.

Refactored the LOB length calculation into on ecommon method so it works correctly for both regular LOB dat aas well as external LOB files. Earlier the external lobs length was being returned incorrectly. Now it reports the length of the external HDFS file .

Fixed one place in generator where the CQD value was not being multiplied by the MB units correctly.

  1. … 10 more files in changeset.
Changes to take care of closing internal cursor statement when extract operation is done and the exact length of data is passed in to be extracted.

  1. … 2 more files in changeset.
TRAFODION-2683 add a new explain option to mask variant fields in output

--Syntax extension to cleanse and mask:

explain options 'c' select ...

explain options 'fc' select...

prepare s from select ...

explain option 'c' s;

--Masked fields will show up as "###" in explain output.

--Both option and options keyword can be used in explain and showplan.

-- explain option/options ...

-- showplan option/options ...

-- showplan can be used on explain statement.

-- Here is an example of what the output will look like:

>>explain option 'c' select * from dual;

----------------------------------------------- PLAN SUMMARY

MODULE_NAME .............. DYNAMICALLY COMPILED

STATEMENT_NAME ........... NOT NAMED

PLAN_ID ................ ###

ROWS_OUT ............... ###

EST_TOTAL_COST ......... ###

STATEMENT ................ select * from dual;

------------------------------------------- NODE LISTING

ROOT ================================ SEQ_NO 2 ONLY CHILD 1

REQUESTS_IN ............ ###

ROWS_OUT ............... ###

EST_OPER_COST .......... ###

EST_TOTAL_COST ......... ###

DESCRIPTION

max_card_est ......... ###

fragment_id ............ 0

parent_frag ............ (none)

fragment_type .......... master

statement_index ........ 0

affinity_value ....... ###

max_max_cardinality ###

total_overflow_size ###

xn_access_mode ......... read_only

xn_autoabort_interval 0

auto_query_retry ....... enabled

plan_version ....... 2,600

embedded_arkcmp ........ used

select_list ............ %(0)

input_variables ........ %(0), %(0), %(0)

VALUES ======================== SEQ_NO 1 NO CHILDREN

REQUESTS_IN ............ ###

ROWS_OUT ............... ###

EST_OPER_COST .......... ###

EST_TOTAL_COST ......... ###

DESCRIPTION

max_card_est ......... ###

fragment_id ............ 0

parent_frag ............ (none)

fragment_type .......... master

tuple_expr ............. %(0)

--- SQL operation complete.

>>

  1. … 21 more files in changeset.
Merge PR 1143 Fix for [TRAFODION-2654] and [TRAFODION-2653]

TRAFODION [2641] User who has MANAGE_STATISTICS privilege can't do update statistics on HIVE tables

TRAFODION [2175] a user should only see specific schemas/tables that he has

privileges to

Updated the following get commands:

get schemas (in catalog)

get tables, indexes (in schema)

get sequences, views (in schema, in catalog)

get libraries, procedures, functions, table mapping functions

TRAFODION [1573] Additional GET commands for privileges

get privileges on table

get privileges on view

New regression test privs1/TEST125

Fixed bug: user granted MANAGE privilege does not have MANAGE sub-privs

Changed REGISTER_HIVE_OBJECT to be treated as a sub-priv under CREATE

Changed UNREGISTER_HIVE_OBJECT to be treaed as a sub-priv under DROP

  1. … 6 more files in changeset.
[TRAFODION-2654] Change the location of trafodion-site.xml from $TRAF_HOME/etc to config [TRAFODION-2653] Sort operator loops at times

All HBase client connections from Trafodion uses Trafodion configuration

concept. Any client side property can now be set in trafodion-site.xml.

The scratch files are now open in blocking mode. Hence any scratch file

operation shouldn't return EAGAIN.

The internal table TDDL of table is now created without any table

coprocessors attached to it.

  1. … 23 more files in changeset.
[TRAFODION-2420] RMS enhancements

New counters introduced as part of BMO

InterimRowCount - Counter to keep track intermediate rows

produced by hash join and for intermediate

runs in sort

phase - BMO Phase for hash-join and sort

get statistics for statement s1, options 'cs' now works.

Removed attaching BMO stats to Non-root Paritial and leaf partition hash group by operators.

These operators are non-blocking BMOs.

Introduced 3 phases for hash grby operator to be visible in RMS Stats

options 'sl' now works with get statistics for qid.

The relevant new counters are now part of accumulated statistics type.

Extended the support upto 512 opeartors from 256 operators in RMS

Fixed both Type 2 and mxosrvr code to obtain all the externalized counters in accumualted stats correctly.

  1. … 27 more files in changeset.