comexe

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
1. JULIANTIMESTAMP now uses clock_gettime with CLOCK_REALTIME by default. It can be switched to earlier implementaion via setting env variable USE_LINUX_JULIANTIMESTAMP=0 2. Remove unwanted CQDs set by mxosrvr. GENERATE_EXPLAIN is not turned ON by mxosrvr.

  1. … 13 more files in changeset.
Merge remote branch 'origin/pr/1721/head' into merge_1721

  1. … 11 more files in changeset.
[TRAFODION-3220] Executor GUI

Adding the ability to display query execution with the "display"

command in Trafodion (sqlci only, requires an X Windows display,

just as the compiler GUI does).

  1. … 44 more files in changeset.
TRAFODION - 3218 User still has privilege after user's role has been revoked ...

Partial support for column level privileges with QI support for:

column select

column insert

column references

column update

Also, as part of this, updated privilege code in a couple of areas:

Changed object caching code in NATable and NARoutine to store all privileges

assigned to the object when the object is cached (privDescs_). During the load

operation, the code creates bitmaps (privInfo_) for the current user. Privilege

checks are performed against the user bitmaps (privInfo_). This is in

anticipation for some performance updates when connecting to Trafodion (mxosrvr)

with different users.

Change getRoleList to include the roleID and the granteeID that granted the

privilege. The grantee can be a user or a role.

When a privilege is revoked from a role, send QI keys for every user that has

been granted to role.

  1. … 40 more files in changeset.
Fixees to support stroring Udr libraries s Blobs.

  1. … 43 more files in changeset.
Fix for TRAFODION-3112

Internal error: get ... for user/role

Heading incorrect for libraries

Parser error: get procedures/table_mapping functions/functions for user/role

  1. … 11 more files in changeset.
TRAFODION-3086 DDL on Hive objs: use database and msck support

-- traf "set schema hive.<sch>" now behaves the same as Hive "use <db>".

1) <sch> is validated to exist at set schema time.

2) <sch> is set in Hive environment for that session. Any unqualified

object reference in ddl uses that default schema (database)

-- added support for msck (meta store check) command.

This command repairs database by refreshing untracked files/partitions

in hive metadata.

  1. … 14 more files in changeset.
TRAFODION-3121 Enhance 'initialize trafodion' to return progress status

-- init traf will now return status during various steps.

'Started' indication will be returned when that step starts.

'Completed' will be returned when it completes successfully.

'Failed' will be returned if that step fails.

Elapsed time for each step will be shown in {ET hh:mm:ss} format.

-- method CmpSeabaseDDL::initSeabaseMD is rewritten and moved from

file CmpSeabaseDDLcommon.cpp to new file CmpSeabaseDDLinitraf.cpp.

-- priv mgr tables are now always created during init traf.

-- a 'no return status' option has been added if status is not to be

returned during init traf. This is used when init traf is called

internally, for ex, during upgrade.

It is not an externalized option.

-- Output of init traf is shown below.

Note: output is not a printf/cout to stdout.

It is returned as selected rows to the caller.

Apache Trafodion Conversational Interface 2.3.0

Copyright (c) 2015-2017 Apache Software Foundation

>>initialize trafodion;

Initialize Trafodion: Started

Verify User: Started

Verify User: Completed {ET: 00:00:00}

Version Check: Started

Version Check: Completed {ET: 00:00:00}

Create Metadata Tables: Started

Create Metadata Tables: Completed {ET: 00:00:49}

Update Metadata Tables: Started

Update Metadata Tables: Completed {ET: 00:00:06}

Create Schema Objects: Started

Create Schema Objects: Completed {ET: 00:00:20}

Create Metadata Views: Started

Create Metadata Views: Completed {ET: 00:00:05}

Create Repository Tables: Started

Create Repository Tables: Completed {ET: 00:00:16}

Create PrivMgr Tables: Started

Create PrivMgr Tables: Completed {ET: 00:00:29}

Create Library Manager: Started

Create Library Manager: Completed {ET: 00:00:51}

Initialize Trafodion: Completed {ET: 00:03:09}

--- SQL operation complete.

  1. … 34 more files in changeset.
Merge branch 'master' of github.com:apache/trafodion into trafodion_2917_1

  1. … 6 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

Ensured that the CQD USE_LIBHDFS 'ON' works as expected via the

older implementation

  1. … 2 more files in changeset.
[TRAFODION-3110] Refactor LOB access to use the new implementation of HdfsClient

Used a single CQD USE_LIBHDFS to switch to older implementation of

using libhdfs for both hdfs scan and LOB access. The CQD USE_LIBHDS_SCAN is

retired.

In addition fixed the following:

1. Fixed a memory leak of LOB Heap structures

2. Possible fix for memory corruption at times

3. Avoid unnecessary creation of LOB threads when not needed

  1. … 23 more files in changeset.
TRAFODION-3086 Further enhancements to 'DDL on Hive objects'

-- Hive CTAS enhancements

-- CTAS will do create through Hive and insert...select in Traf

-- a cqd can be set if CTAS need to be passed in completely to Hive

-- CTAS 'no load' option can be used to create a Hive

table LIKE a traf table.

Older 'create hive like traf' is removed.

-- explain for CTAS to show ddl, insert...select, upd...stats stmts

-- Truncate enhancements.

-- traf: purgedata, truncate, truncate table

-- Hive: truncate, truncate table

-- 'if exists' option has been added

-- Hive truncate is through new exeutil operator

-- older operator uses LOB interface to clear data.

Maintained as Legacy operator, will be removed.

-- new operator uses Hive to truncate tables

-- explain for truncate to show trunc query that will be sent

-- removed obsolete parallel label op

-- removed obsolete purgedata (fast delete) op

  1. … 27 more files in changeset.
Merge remote branch 'origin/master' into ansharma_hiveddl_br

  1. … 7 more files in changeset.
Merge remote branch 'origin/pr/1586/head' into merge_1586

  1. … 5 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 89 more files in changeset.
Support to provide a locking mechanism for LOB insert/update operations

  1. … 28 more files in changeset.
TRAFODION-1573: Additional GET commands for privileges TRAFODION-3074: Failed to register/unregister user when security disabled

TRAFODION-1573 changes:

- Added support for the following commands:

get privileges on <object>

<object>: [LIBRARY | PROCEDURE | FUNCTION | TABLE_MAPPING FUNCTION | SEQUENCE]

- Added support for the FOR CLAUSE on all supported objects

Removed the need to specify keyword 'USER" before username. If USER is

included, then it is ignored.

get privileges on <object> FOR [USER] <user or role name>

<objects>: [TABLES, VIEWS, LIBRARIES, PROCEDURES, FUNCTIONS,

TABLE_MAPPING FUNCTIONS, SEQUENCES]

- The following get command can only be run by DB__ROOT or a user that has been

granted the DB__ROOTROLE or DB__HIVEROLE role

get <objects> in schema hive.xx.xx;

<objects>: [TABLES, OBJECTS, VIEWS]

- The following get command can only be run by DB__ROOT or a user that has been

granted the DB__ROOTROLE or DB__HBASEROLE role

get external hbase objects;

- The following get commands retrieve privilege details from Trafodion metadata;

users can only see objects where they have been granted at least one privilege

get hive registered tables in catalog trafodion;

get hbase registered tables in catalog trafodion;

- get privileges commands now return owner's privileges in output

- Cleaned up code in the parser.

TRAFODION-3074 changes

- register user - fixed query to find next available authID

- unregister user - added checks to not read privilege metadata if authorization

is not enabled

  1. … 17 more files in changeset.
[TRAFODION-2982] JNI HDFS interface should support varied sized large buffers for read/write

Changed the interim byte array size to be of int type to accoumdate

at least 64MB as per review comments.

    • -0
    • +84
    ./ComCompressionInfo.cpp
    • -0
    • +91
    ./ComCompressionInfo.h
  1. … 6 more files in changeset.
[TRAFODION-3065] Trafodion to support compressed Hive Text formatted tables

Compressed text files are now supported via the new implementation using

HDFS java APIs. When the hadoop is not configured to support a particular type

of compression, an error is thrown.

[TRAFODION-2982] JNI HDFS interface should support varied sized large buffers for read/write

A new CQD HDFS_IO_INTERIM_BYTEARRAY_SIZE_IN_KB is introduced to chunk

the read and write when byteArray is involved.

  1. … 13 more files in changeset.
Merge remote branch 'origin/pr/1538/head' into merge_1538

  1. … 2 more files in changeset.
Removing unneeded CLI call to get lob location and instead using the CQD value - LOB_STORAGE_LOCATION to get/save it in Tdb for use later at runtime.

  1. … 2 more files in changeset.
Merge [TRAFODION-2175] PR 1512 Get statement enhancements

  1. … 2 more files in changeset.
Get statement enhancements

Added support and privilege checks for the following commands:

get functions for library

get procedures for library

get table_mapping functions for library

get indexes on table

get objects on table

get views on table

get views on view

get libraries in schema

get objects in view

get tables in view

get views in view

get indexes for user

get tables for user

get libraries for user

get views for user

Changed "get libraries for schema" to include libraries where the current user

has execute privilege on one of the libraries routines (functions, procedures,

or table_mapping functions).

Addressed a performance issue when determining if the user has column level

privileges. If the user has granted privileges against native Hive tables

through EsgynDB, we need to get the column name from Hive. The call to get the

column, by calling hivemd, is very expensive. This change checks to see if the

requested user has been granted any column level privileges on a hive table.

If so, we will go ahead and do the mapping (call hivemd). If not, then we will

not include the hivemd fragment for the query. Since we are scanning the column

privileges table anyway, we also see if the requested user (or their roles) has

been granted any privileges. If so, we include the column privileges check in

the query.

Commented out get statements that we do not support at this time.

  1. … 8 more files in changeset.
review changes from PR 1502, plus jenkins compGeneral crash fix

  1. … 4 more files in changeset.
various fixes

-- JIRA 2980 support for INSTR function.

INSTR(source_string, pattern, startPos, occurrence)

Search for pattern in source_string.

Start at startPos'th character. Optional, default 1

Return occurrence'th occurrence. Optional, default 1

(exp/exp_function.cpp/h, exp/ExpPCodeClauseGen.cpp,

generator/GenItemFunc.cpp, optimizer/BindItemExpr.cpp, ItemExpr.cpp,

optimizer/ItemFunc.h, SynthType.cpp, parser/sqlparser.y)

-- hivemd returns precision, scale, display_datatype

(comexe/ComTdbExeUtil.h, common/BaseTypes.cpp,NAType.cpp/h,

executor/ExExeUtilGet.cpp, ExExeUtil.h)

-- hive decimal type is treated as sql NUMERIC for performance.

(common/NAType.cpp)

-- hive select from hivemd with prepare and multiple executes work

(executor/ExExeUtilGet.cpp)

-- fix for crash in sql buffer pool when allocating large size rows.

(executor/sql_buffer.cpp)

-- SIGN(op) now returns error if op is not numeric

(BindItemExpr.cpp)

-- strings of length zero are no longer cached. Caching them causes problem

during backpatching.

(optimizer/ItemCache.cpp)

--- compile time predicates on constants were not being created correctly

in some cases. That has been fixed.

(ValueDesc.cpp)

-- trim now supported on scaled numeric, float and bignum datatypes

(BindItemExpr.cpp)

-- CASE is now supported on scaled numerics, float, bignum

(BindItemExpr.cpp)

-- char_length is now supported on numerics

(ItemFunc.h, BindItemExpr.cpp)

-- DAYOFMONTH now allowed only on datetime datatypes

(common/OperTypeEnum.h, sqlparser.y, BindItemExpr.cpp, ItemExpr.cpp)

-- TO_TIME now allowed only on 'datetime with time' or character datatypes.

(BindItemExpr.cpp, bin/SqlciErrors.txt)

-- updated expected file for compGeneral/test042

  1. … 31 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Removed the dependency on libhdfs in Trafodion code in some more portions of the code.

Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:

a) Prefetch the data using double buffering concept

b) Avoid unnecessary data copy

c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.

This HDFS Scan is still being tested and hence it is turned off by default.

Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

  1. … 40 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

  1. … 15 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text format hive tables

Part-6 changes

Removed the dependency of libhdfs while assigning ranges for hdfs scan at runtime

and for sample file creation during bulk load

  1. … 12 more files in changeset.
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

Part-2 changes

Introduced a new CQD USE_LIBHDFS_SCAN 'OFF' to switch to the new implementation

The new implementation details are at executor/ExHdfsScan.h

Fixed a bug that was causing unexpected errors in JVM when the JNI object corresponding

to java class is not initialized correctly

  1. … 27 more files in changeset.
New syntax to retrieve the LOB HDFS filename for both external and internal LOBs . Also added syntax to return starting offset of a particular LOB handle in the LOB Hdfs data file.

  1. … 15 more files in changeset.