CmpSeabaseDDLroutine.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Library manager fixes and LOB alter fix

  1. … 9 more files in changeset.
Fixes for regression tests and expected files

  1. … 5 more files in changeset.
iChanges based on review comments

  1. … 14 more files in changeset.
Fixees to support stroring Udr libraries s Blobs.

    • -215
    • +1585
    ./CmpSeabaseDDLroutine.cpp
  1. … 45 more files in changeset.
TRAFODION-3121 Enhance 'initialize trafodion' to return progress status

-- init traf will now return status during various steps.

'Started' indication will be returned when that step starts.

'Completed' will be returned when it completes successfully.

'Failed' will be returned if that step fails.

Elapsed time for each step will be shown in {ET hh:mm:ss} format.

-- method CmpSeabaseDDL::initSeabaseMD is rewritten and moved from

file CmpSeabaseDDLcommon.cpp to new file CmpSeabaseDDLinitraf.cpp.

-- priv mgr tables are now always created during init traf.

-- a 'no return status' option has been added if status is not to be

returned during init traf. This is used when init traf is called

internally, for ex, during upgrade.

It is not an externalized option.

-- Output of init traf is shown below.

Note: output is not a printf/cout to stdout.

It is returned as selected rows to the caller.

Apache Trafodion Conversational Interface 2.3.0

Copyright (c) 2015-2017 Apache Software Foundation

>>initialize trafodion;

Initialize Trafodion: Started

Verify User: Started

Verify User: Completed {ET: 00:00:00}

Version Check: Started

Version Check: Completed {ET: 00:00:00}

Create Metadata Tables: Started

Create Metadata Tables: Completed {ET: 00:00:49}

Update Metadata Tables: Started

Update Metadata Tables: Completed {ET: 00:00:06}

Create Schema Objects: Started

Create Schema Objects: Completed {ET: 00:00:20}

Create Metadata Views: Started

Create Metadata Views: Completed {ET: 00:00:05}

Create Repository Tables: Started

Create Repository Tables: Completed {ET: 00:00:16}

Create PrivMgr Tables: Started

Create PrivMgr Tables: Completed {ET: 00:00:29}

Create Library Manager: Started

Create Library Manager: Completed {ET: 00:00:51}

Initialize Trafodion: Completed {ET: 00:03:09}

--- SQL operation complete.

  1. … 36 more files in changeset.
[TRAFODION-2249] Cannot use library management SPJs after an upgrade

Updated initialize trafodion, upgrade library management to modify the

jar/dll file locations for all system libraries to the new location.

Changed python installer to call ..., upgrade library management during

an upgrade operation

  1. … 2 more files in changeset.
[TRAFODION-2974] Make event log reader and JDBC real TMUDFs

Roberta pointed out that we have two predefined UDFs, EVENT_LOG_READER

and JDBC, where the system administrator should have the ability to

control who can execute these functions.

To do this, these two UDFs cannot be "predefined" UDFs anymore, since

those don't have the metadata that's required for doing grant and

revoke.

Roberta also pointed out that the JDBC UDF should refuse to connect to

the T2 driver, for security reasons.

The fix leaves the predefined TMUDFs in place, for now, they will be

removed in R2.4 (see TRAFODION-2975).

The new "real" TMUDFs are in the "_LIBMGR_" schema, mostly for

convenience, as this schema has other UDFs that are created when

Trafodion is initialized.

  1. … 10 more files in changeset.
[TRAFODION-2768] Make Trafodion code base to compile in RH7

  1. … 126 more files in changeset.
[TRAFODION-2752] Fix error message when UDR method not found

The error message displayed only the signature, but not the

method name.

  1. … 2 more files in changeset.
[TRAFODION-2754] Get statistics cores sqlci or mxosrvr at str_sprintf()

Switched str_sprinf to use the standard sprintf function to ensure

that the format specification and the passed in parameters are consistent

  1. … 65 more files in changeset.
[TRAFODION-2637] Library management with Cloudera parcels

Also in this commit:

[TRAFODION-2581] Combine JVM startup code of SQL executor and UDR server

We have library management SPJs that allow a user to deploy library

files from a client machine. These SPJs make use of the pdcp command

that is installed on many clusters, but not always. The new method

uses temporary HDFS files and a TMUDF to distribute the library

file to all nodes of the cluster.

For now I left the old PUT method in place and added a new PUTFILE

method. We can remove the old method later.

A few other things fixed in the process:

- Make sure that when a UDR server calls the executor and uses a JVM,

we share the same JVM. To do this, we need to use the same startup

method, and I needed to change the executor startup method to accept

the extra options used for the UDR server.

- Got rid of the "HBase compatibility mode" in JavaObjectInterface.

- A common method to execute SQL in the library management code.

- removed the obsolete init_libmgmt.sh file.

- We have special code that makes sure a UDR can request to run in

parallel, including a request to run on all nodes. Expanded this

special code from predefined UDFs to any UDF.

  1. … 25 more files in changeset.
Fixes addressing multiple failure in SQL Regressions (core 116).

  1. … 6 more files in changeset.
TRAFODION-2441 user has only select privilege on a table can do ... TRAFODION-2409 support privilege control(column privileges) for hive tables TRAFODION-2423 any user can perform 'initialize trafodion, drop' TRAFODION-2435 Any user can perform TRUNCATE on native Hive tables. TRAFODION-2463 Hive: Any user can do update statistics for hive tables

Fixed issues found while testing privileges with native Hive.

TRAFODION-2441:

changed code that initializes owner privileges for views.

TRAFODION-2409:

returning error message 1328 during attempt to grant unsupported column level

privilege on hive table.

TRAFODION 2423:

added privilege checks for all initialize commands, error 1017 is returned if

not DB__ROOT

TRAFODION-2435:

Returning error 1051 if TRUNCATE is attempted on a hive table where the

current user has no privilege

TRAFODION-2463:

Privilege checks added for Hive table during update statistics

  1. … 25 more files in changeset.
convert all MY_SQROOT to TRAF_HOME

  1. … 294 more files in changeset.
metadata access performance improvement for accessing traf objects

Packed version of descriptor structures is now stored in traf TEXT

table during various DDL operations.

During dml, if packed descriptors exist, then they are read and used.

If they dont exist, then descriptors are generated on the fly.

Descriptor classes are defined in file sqlcat/TrafDDLdesc.h and

are based off the older desc structures from desc.h.

These are simplified and class'ified version of the older structures

and are versioned. Changes to these classes should be done carefully

by either using filler bytes or versioning them.

2 new cqds, traf_store_object_descriptor and traf_read_object_descriptor,

are added to control if packed descs should be stored/read.

These are currently set to OFF by default but are enabled during

regressions run. Once this feature is tested, these cqds will be turned

on by default.

In addition, some helper commands have been added to manipulate descrs.

Syntax:

alter table t [check|generate|delete|disable|enable] stored descritors

If authorization is enabled, then auth/priv specific info

is generated on the fly and currently not stored in metadata.

In a later checkin, these will also be stored in metadata.

  1. … 74 more files in changeset.
Merge in latest origin/master

  1. … 4 more files in changeset.
Merge [TRAFODION-2039] PR 563 Fix mistake in PR 530

Merge remote branch 'hegdean/wrkbrnch' into Trafodion2037

  1. … 5 more files in changeset.
[TRAFODION-2037] Improve DDL concurrency

  1. … 10 more files in changeset.
Merge [TRAFODION-2039] PR 530 Add support for ALTER LIBRARY FILE '<file-name>'

  1. … 1 more file in changeset.
JIRA TRAFODION-2060 Add support for tinyint datatype

-- added support for tinyint datatype to be used in create

and cast statements

-- added support for returning tinyint in select list and

input params

-- currently tinyint select list and input params is only enabled for

queries from sqlci. Some support for it has been added to jdbc T4

but need to be completed. Until then, tinyint select/param support

is not enabled for t4/t2/odbc interfaces

-- removed obsolete tandem floating point datatypes

  1. … 62 more files in changeset.
[TRAFODION-2039] Add support for ALTER LIBRARY FILE '<file-name>'

Location of library file can now be altered. Alter command will check for

existence of new file. Old library definition will be removed from compiler

caches.

  1. … 6 more files in changeset.
Various fixes

-- Implicit xns for ddl stmts are now started in master exe if

autocommit is OFF

-- default values for varchar columns are now handled correctly

-- DATEFORMAT function on timestamp values are now handled correctly

-- CREATE/DROP PROCEDURE stmt are now handled correctly with DDL xns.

-- alter primary key stmt now returns an error if table already has a

user defined primary/clustering key

-- regress/core/TEST116, seabase/TEST030,031 have been updated with new tests

  1. … 4 more files in changeset.
commit #1

  1. … 25 more files in changeset.
TRAFODION [1902] - Showddl continues to show function after it's dropped

Fixed a problem during a drop library request, where functions and procedures

were being dropped but not removed from DBRoutineCache.

Added a new regression test (udr/TEST103) that tests DDL operations related to

functions, procedures, and libraries. The regression tests were missing.

  1. … 5 more files in changeset.
update

  1. … 298 more files in changeset.
TRAFODION [1696] - drop authorization doesn't drop all roles, and create role will run into internal error

A while back a change was made in the PrivMgr constructors to assume that

authorization is enabled by default instead of the unknown state. If the state

is unknown, privilege manager performs I/O to determine its state, otherwise

no additional checking is performed. This was changed because one, the

authorization check in PrivMgr is expensive and two, the majority of the callers

already perform the authorization check by looking in the compile context set up

during process startup. Role code was not updated to handle this change

correctly as described in TRAFODION-1696. Changes were made to check compiler

context to verify that authorization is enabled for role commands.

Fixed a comment related to queryBuf size for internal library management

operations.

  1. … 1 more file in changeset.
TRAFODION [1879] - Integrate Library Management into Trafodion Metadata

The initial release of library management has been delivered to Trafodion. This

is follow-on work that integrates library management operations into the

existing Trafodion infrastructure. Currently, with the initial release of

library management, the consumer needs to run a special script to set up

everything. This delivery incorporates the steps into existing SQL commands and

removes the need for the script.

This delivery contains:

-> Support for three new INITIALIZE TRAFODION options

-> Support for a new role (plus infrastructure to make it easier to add roles)

-> Change initialize authorization to handle upgrade better

-> Fixed couple issues found while testing

-> New regression test (udr/TEST102)

*** Support for three new INITIALIZE TRAFODION options:

Three new INITIALIZE TRAFODION options have been added:

CREATE LIBRARY MANAGEMENT - create and populates the libmgr schema

DROP LIBRARY MANAGEMENT - drops the libmgr schema

UPGRADE LIBRARY MANAGEMENT - adds new procedures to the libmgr library

Parser was changed to support these new options, a new keyword - MANAGEMENT

was added.

(parser/ParKeyWords.cpp/sqlparser.y)

DDL compiler was changed to recognize the new options and call implementation

methods.

(optimizer/RelExeUtil, sqlcomp/CmpSeabaseDDLcommon.cpp)

The following implementation methods were added:

createSeabaseLibmgr (code for create library management)

dropSeabaseLibmgr (code for drop library management)

upgradeSeabaseLibmgr(code for upgrade library management)

createLibmgrProcs – a helper method called by create and upgrade code to

create libmgr procedures

grantLibmgrPrivs – a helper method called by createLibmgrProcs and

initialize authorization that add grants to procedures for

the new DB__LIBMGRROLE role.

(CmpSeabaseDDL.h/CmpSeabaseDDLroutine.cpp)

Changes were made to define the new schema, library and procedures.

(common/ComSmallDefs.h, CmpSeabaseDDLroutine.h)

All procedures are defined in a new file called sqlcomp/CmpSeabaseDDLroutine.h.

This file is based on similar support for system metadata tables

(CmpSeabaseDDLmd.h). It contains the text of all the procedures. During

"create library management" and "upgrade library management" operations, this

list is used to create/add procedures.

*** Support for a new role:

Initialize authorization code was changed to create the new DB__LIBMGRROLE role.

As part of this change, role infrastructure was modified to make it easier to

add system roles in the future. The initialize code checks to see what roles

have not been added, and adds them.

(common/NAUserId.h, common/ComUser, sqlcomp/PrivMgrMD,

sqlcomp/CmpSeabaseDDLauth, sqlcomp/PrivMgrRoles)

In addition, a check is made – if the SEABASE_LIBMGR_SCHEMA exists, then

grants are performed on the procedures to allow DB__LIBMGRROLE execute

privilege.

(sqlcomp/CmpSeabaseDDLcommon.cpp)

*** Change initialize authorization to handle upgrade better:

Changes were made so initialize authorization performs an upgrade operation if

called and component privileges already exists.

(sqlcomp/PrivMgrComponentOperations, sqlcomp/PrivMgrComponentPrivileges,

sqlcomp/PrivMgrMD)

*** Fixed couple of issues found while testing:

Invalid role ID generation. Role ID generation code was using the wrong range

to determine the next role ID.

(CmpSeabaseDDLauth.cpp)

Drop schema issues with libraries. Drop library cascade fails when the order

of libraries and functions is incorrect. If you drop function, drop library it

fails with missing procedure.

(CmpSeabaseDDLschema.cpp)

  1. … 26 more files in changeset.
JIRA TRAFODION-1798 (ddl xns) and few other fixes, details below.

-- support for sql part of ddl xns. Section 1 of JIRA TRAFODION-1798

-- cqd ddl_transactions to enable or disable ddl xns.

Default is currently off. Once it is tested, it will be turned on.

Dev regressions are run with cqd set to ON

-- get stmts run with read committed to get changes in current xns

-- support for where preds with get stmts

-- scan to pass in transid even if running with read uncommitted access.

This enables rows modified in current xn to be returned.

-- cleanup no longer return multiple duplicate error messages if

objects id is not found.

-- cleanup no longer includes internallay created schemas (_HV_ , _HB_)

during cleanup operations.

-- Correct error msg was not getting returned if an invalid index

existed in table and the same index was created again.

-- init traf, drop md views was giving an error if views didnt exist.

That has been fixed.

-- regressions with -diff option now show original file timestamps

instead of the timestamp when the diff command was run.

  1. … 72 more files in changeset.