ustat

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Turned on privilege features, reorg'd PrivMgr code

Turned on GRANTED BY, WITH GRANT OPTION, GRANT OPTION FOR clauses

For GRANTED BY -> showddl now displays the GRANTED BY clause when

--> the current user is not the object owner and

--> the current user is not DB ROOT

added object_owner and schema_owner to the SeabaseLibraryDesc

ComTdb.h

Generator.cpp

desc.h

CmpSeabaseDDLtable.cpp ->getSeabaseLibraryDesc

added object owner in calls to PrivMgrPrivileges::getPrivTextForObject

CmpDescribe.cpp

PrivMgrCommands.cpp/.h

PrivMgrPrivileges.cpp/.h

Changed object grant and revoke to store the list of privileges associated

with the object and columns in the PrivMgrPrivileges class.

--> added new methods generateObjectRowList and generateColumnRowList,

changed the destructor to remove these lists, changed code to call

these new methods, and removed extra I/Os

--> removed member trafMetadataLocation_ (it is already stored in parent)

For WITH GRANT OPTION clause at GRANT time:

--> added checks at grant time to:

--> check for potential circular grants (error 1036)

--> added new method getTreeOfGrants to get list of grantors

that have previously granted to the current grantee

For GRANT OPTION FOR clause at REVOKE time:

--> changed error messages returned to be more meaningful

--> moved and activated call checkRevokeRestrict after call to

getAffectedObjects

Added new columns to the COLUMN_PRIVILEGES and SCHEMA_PRIVILEGES tables

to include the object_name, grantor_name, and grantee_name to match

OBJECT_PRIVILEGES, ROLE_USAGES, and COMPONENT_PRIVILEGES;

Reorganized the contents of PrivMgr files:

--> PrivMgr document exists that describes the .h/.cpp structure

--> Added new files PrivMgr.h/PrivMgr.cpp that describes the parent

class for all PrivMgr requests

--> moved existing defines, classes, etc around to match the PrivMgr

document

Fixed a couple of issues:

--> Fixed a bug in initialize authorization where the WGO was not set up

correctly for UDR's

--> Fixed a bug in PrivMgrObject::selectAllWhere where an error condition

was not returned

--> Fixed a bug in seabaseGrantRevoke where the incorrect object type was

sent for views

--> Fixed a bug in update statistics privilege checking that was not

handling HBase tables correctly

Added two regression tests (skipped until catman1 test directory is split up)

--> TEST132 - tests for privilege checking on libraries, populate index,

showddl, invoke, update statistics, and showstats

--> TEST140 - tests for WITH GRANT OPTION and GRANTED by option

Fixed expected result for catman1/TEST137

Change-Id: Iaf523aef763b0bce2101fedae0ee701606c369c7

  1. … 51 more files in changeset.
Fix 1450193: User ... plans generated without using histogram stats

The issue occurs because the person compiling the query does not have select

privilege on the HISTOGRAMS table so statistics cannot be retrieved.

We support a special interface to allow metadata tables to be accessed without

privs for operations performed during compilations. However, this interface is

not being used for HISTOGRAMS tables during query compilation. The change is to

use this interface when gathering statistics during a query compilation.

Added exception handler around setting/resetting the parserflag.

Change-Id: I13f9a4bd870b3978e29aa1dad251c24ed400f8ff

Stop sending user CQDs at embedded compile switch

Do not send user entered CQDs when switching to embedded META CmpContext

to compile metadata queries.

Also do the holdAndSet needed CQDs only when the embedded (META)

CmpContext is first switched to. These CQDs won't be restored when

switching back so that we won't send the same set of CQDs again next

time the CmpContext is used.

LP bug 1438372, Compiler returns internal error from

../optimizer/opt.cpp line 6907

LP bug 1444739, compGeneral/TEST006 failed with tmp fix for bug 1438372

Change-Id: I92398aeeaf53820d3f8af1fb73e9c1d6f449ce5e

  1. … 7 more files in changeset.
various lp and other fixes, details below.

-- added support for self referencing constraints

-- limit clause can now be specified as a param

(select * from t limit ?)

-- lp 1448261. alter table add identity col is not allowed and now

returns an error

-- error is returned if a specified constraint in an alter/create statement

exists on any table

-- lp 1447343. cannot have more than one identity columns.

-- embedded compiler is now used to get priv info during invoke/showddl.

-- auth info is is not reread if already initialized

-- sequence value function is now cacheable

-- lp 1448257. inserts in volatile table with identity column now work

-- lp 1447346. inserts with identity col default now work if inserted

in a salted table.

-- only one compiler is now needed to process ddl operations with or

without authorization enabled

-- query cache in embedded compiler is now cleared if user id changes

-- pre-created default schema 'SEABASE' can no longer be dropped

-- default schema 'SCH' is automatically created if running regressions

and it doesn't exist.

-- improvements in regressions run.

-- regressions run no longer call a script from another sqlci session

to init auth, create default schema

and insert into defaults table before every regr script

-- switched the order of regression runs

-- updates from review comments.

Change-Id: Ifb96d9c45b7ef60c67aedbeefd40889fb902a131

  1. … 69 more files in changeset.
Security fixes for 144553, 1414125, and 1393529

1445583: showstats command performance slow with security enabled

Several changes were made to improve performance:

Performance optimization:

NATable.cpp: NATable::setupPrivs

- If the current user is the object owner, then default the privilege bitmap

to object Owner values - no need to call PrivMgr to get privileges

Caching optimization:

We are now caching privmgr metadata tables in compiler cache when the compiler

context is instantiated. This avoids a metadata lookup for these tables.

- Added new methods that return if the table is part of the PrivMgr schema

- Adjusted CmpSeabaseDDL::createMDdescs to include privmgr metadata in the

cached entries

- Adjusted CmpSeabaseDDL::getMDtableInfo to check for privmgr metadata tables

from the cached entries

- Removed obsolete code CmpSeabaseDDL::alterSeabaseDropColumn

- changed CmpSeabaseDDL::getSeabaseTableDesc to check for both system and

privmgr metadata from compiler cache

- added new method CmpSeabaseDDL::getPKeyInfoForTable that returns the

primary key name and UID for a table. This is needed when dropping privmgr

metadata tables

Removed extraneous recompilations of HISTOGRAM structures:

Today, update statistics and showstats are reloading NATable entries

for HISTOGRAM tables on every access. This is because the parserflag

ALLOW_SPECIALTABLETYPE is turned on. When this flag is turned, the compiler

always reloads the cache entries - see code from CmpMain::sqlcomp:

//if using special tables e.g. using index as base table

//select * from table (index_table T018ibc);

//then refresh metadata cache

if(Get_SqlParser_Flags(ALLOW_SPECIALTABLETYPE) &&

CmpCommon::context()->schemaDB_->getNATableDB()->cachingMetaData())

CmpCommon::context()->schemaDB_->getNATableDB()->refreshCacheInThisStatement();

Changed code to not set ALLOW_SPECIALTABLETYPE and ALLOW_PHONYCHARACTERS

parserflags by default. Individual statements are setting these flags as needed.

1414125: User without priv can view data in metadata tables

The problem is that a user with priv cannot view data in metadata tables.

Even when a user had SELECT privilege on a system or privmgr metadata table,

the request failed.

The problem is that parameter 2 sent to CmpDescribeIsAuthorized in

hs_globals.cpp is NULL so SELECT priv is not checked. If the user has SHOW

component privilege, it works. A call was added to getPrivileges for metadata

tables before calling CmpDescribeIsAuthorized.

1393529: Core dump accessing MD table descriptors

When "UPDATE STATISTICS LOG [ON, OFF, CLEAR]" is specified by a non DB__ROOT

user, a core dump occurred. This happens because the isAuthorized check is

performed expecting a NATable structure. This command does not need any

special security checks.

Updated traf_authentication_setup script to support a new installation option

Change-Id: If7dbf3ec66e5beb7d88bda61ef32611401dd97b9

  1. … 8 more files in changeset.
Eliminate manual steps in load/ustat integration

The fix achieves full integration of the bulk load utility with

Update Statistics. The Hive backing sample table is now creeated

automatically (formerly, we only wrote the HDFS files to be

used by the Hive external table), the correct sampling percentage

for the sample table is calculated, and the ustat command is

launched fro1m the executor as one of the steps in execution of

the bulk load utility.

Change-Id: I9d5600c65f0752cbc7386b1c78cd10a091903015

Closes-Bug: #1436939

  1. … 23 more files in changeset.
New ustat algorithm and bulk load integration

This is the initial phase of the Update Statistics change to use

counting Bloom filters and a Hive backing sample table created

during bulk load and amended whenever a new HFile for a table

is created. All changes are currently disabled pending some

needed fixes and testing.

blueprint ustat-bulk-load

Change-Id: I32af5ce110b0f6359daa5d49a3b787ab518295fa

    • -0
    • +225
    ./hs_faststats.cpp
  1. … 13 more files in changeset.
Enable authorization by default for regress, plus

Patch 1:

Added TEST138 to catman1 - skipped files

Fixed wording in the traf_authentication_setup script from reviewer comments.

Original delivery:

change 1 - Enable authorization during development regression tests

change 2 – Added support for create schema IF NOT EXISTS and drop schema IF EXISTS

change 3 - Changed traf_authentication_setup script to support a new installation option

change 1 - Enable authorization during development regression tests

Authorization will be enabled during regressions runs

Since regressions run mostly as DB__ROOT, there should be few visible differences.

Developers may see GRANT statements displayed as part of SHOWDDL requests.

This can be controlled by a new CQD:SHOWDDL_DISPLAY_PRIVILEGE_GRANTS

SHOWDDL_DISPLAY_PRIVILEGE_GRANTS

ON - display GRANTS if authorization is enabled

OFF - do not display GRANTS

SYSTEM

if running with SQLMX_REGRESS set, do not display grants

otherwise, display grants

regress/tools/init_sb_regr_sql -- execute initialize authorization

regress/tools/runregr_catman1.ksh -- turn on TEST138

regress/catman1 -- various test and expected files to set the new SHOWDDL CQD

"Initialize authorization, drop;" can be performed to disable authorization

files:

sql/regress/catman1/EXPECTED135

sql/regress/catman1/EXPECTED137

sql/regress/catman1/EXPECTED138

sql/regress/catman1/TEST133

sql/regress/catman1/TEST135

sql/regress/catman1/TEST136

sql/regress/catman1/TEST137

sql/regress/catman1/TEST138

sql/regress/catman1/TEST139

sql/regress/tools/init_sb_regr.sql

sql/regress/tools/runregr_catman1.ksh

sql/sqlcomp/CmpDescribe.cpp

sql/sqlcomp/CmpSeabaseDDLauth.cpp

sql/sqlcomp/DefaultConstants.h

sql/sqlcomp/nadefaults.cpp

change 2: Added support for create schema IF NOT EXISTS and drop schema IF EXISTS

Added support for new schema syntax. Change update stats for HIVE tables to use this syntax

files:

sql/parser/StmtDDLCreate.cpp

sql/parser/StmtDDLCreateSchema.h

sql/parser/StmtDDLDrop.cpp

sql/parser/StmtDDLDropSchema.h

sql/parser/sqlparser.y

sql/sqlcomp/CmpSeabaseDDL.h

sql/sqlcomp/CmpSeabaseDDLcommon.cpp

sql/sqlcomp/CmpSeabaseDDLschema.cpp

sql/ustat/hs_globals.cpp

change 3: Changed traf_authentication_setup script

This file was changed to support a new option "--setup" that only enables authentication

This will be used by the installation script when the customer chooses not to

initialize trafodion.

sqf/sql/scripts/traf_authentication_setup

traf_authentication_setup --help

This script enables or disables security features for Trafodion

Usage: traf_authentication_setup [options]

Options:

--file <loc> Optional location of the OpenLDAP configuration file

--help Prints this message

--off Disables authentication and authorization

--on Enables authentication and authorization

--setup Enables authentication

--status Returns status of authentication enablement

Change-Id: Ia9a66364a6d74955a0833088874e0aaca044eae3

  1. … 24 more files in changeset.
OSIM (Optimizer Simulator) redesign 1.

Simulate query plan generation of production cluster on dev workstation,

by collecting information from production cluster, and restore it on dev worksation.

--runnig on production clusters, collect table DDLs, statistics, CQD, to osim-directory,

--the directory path can either full(absolute) or relative.

osim capture location '<osim-directory>'[, force];

--runing queries on cluster

osim capture stop;

--restore DDLs, CQDs, statistics and cluster information.

osim load from '<osim-directory>';

--setup runtime information, like cpu number, node number.

osim simulate start|continue '<osim-directory>';

Change-Id: I30882e87a6ea0f08c9aa64685705eebebcbb3bf0

  1. … 37 more files in changeset.
Misleading error when Hivestats can't be created

When Update Statistics is performed on a Hive table, the

trafodion.hivestats schema is created if it doesn't already

exist. If the user issuing the ustat statement does not

have the create_schema component privilege, this will fail, but

the code as it existed ignored the error, assuming it was

a "schema already exists" error, which is expected in most

cases. If the schema does not exist, an error is then given

when an attempt is made to create the histograms table in

the non-existing schema.

This has been changed to check the error when the schema

creation attempt fails, so the correct error is generated.

This is a partial fix, because a user with manage_statistics

privilege should not be prevented from executing Update

Statistics just because the trafodion.hivestats schema did

not already exist. The likely full solution would be to

create the schema automatically at Trafodion initialization.

Change-Id: I26561100bab9f6a59f3f5d0d56ce9d2aab380fa3

Partial-Bug: #1425748

  1. … 1 more file in changeset.
Merge "Fixes for security gaps"

  1. … 1 more file in changeset.
Errors reported for Update Stats on Hive tables

Attempts to execute Update Statistics statements on Hive

tables fail with error 1002 (catalog does not exist). The

catalog used to store Hive stats was no longer being

successfully created on demand. The Trafodion catalog is

now used instead, with a schema for Hive stats created

in it on first use. The tables for histograms and

histogram intervals for Hive tables are created in that

schema on first use.

Change-Id: Ib57d0af4a3da6f52f0544d6c2fce3e77d2e823c1

Closes-Bug: #1320397

  1. … 1 more file in changeset.
Fixes for security gaps

Fix summary:

1389791 – Create table with 128 character-long schema & table names hangs on HortonWorks

fix 1 - Privilege checks not working for UDRs

fix 2 - QI not working when UDR's are involved

fix 3 - Routines are not being removed from NARoutineDB cache

Code cleanup

Miscellaneous changes

1389791: Create table with 128 character-long schema & table names hangs on HortonWorks

Check to make sure the total name length is not longer than supported value,

see: https://issues.apache.org/jira/browse/HDFS-6055

bin/SqlciErrors.txt - new error message

sqlcomp/CmpCatSqlErrorCodes.h - new error message

sqlcomp/CmpSeabaseDDLmd.h - new literal describing length of generated HBase name

sqlcomp/CmpSeabaseDDLcommon.cpp - new check for maxmum HBase name length

fix 1: privilege checks are not working correctly for UDR's

The method RelRoot::checkPrivileges is called to verify privileges for all object types.

However, some UDR objects checks were skipped because they were not added to the UDR Stoi list.

optimizer/BindItemExpr.cpp - add function to Stoi list

optimizer/BindRelExpr.cpp - add procedures to Stoi List

optimzier/RelMisc.h - signature changes for privilege related work

optimizer/BindRelExpr.cpp - rewrote checkPrivileges

optimizer/NARoutine.h/NARoutineDB.cpp - added method

moveRoutineToDeleteList

fix 2: QI is not working when UDR's are dropped

Code to drop items from NARoutineDB cache was missing.

Code to set security keys for the user in the plan was missing

Code to set objectUIDs in the plan was missing

When security keys were added, they were incorrect

sqlcomp/CmpMain.h (.cpp) - added calls to compare invalidation keys with objects stored in

NARoutineDB cache; if found, then remove item from cache by

calling helper methods in NARoutineDB class.

optimizer/NARoutineDB.h (NARoutine.cpp) - added helper method to remove entries from the cache

free_entries_with_QI_key - based off of similar method for table cache

ComSecurityKey.h (.cpp) - new method to check invalidation keys shared by tables/routines

qiCheckForInvalidObject

optimizer/NATable.cpp - rewrote table invalidation code so it could be shared with routines.

generator/GenUdr.cpp - add the routine's object UID to the query plan

sqlcomp/CmpSeabaseDDLroutine.cpp - code to send invalidations keys during drop routine

common/ComSmallDefs.h - new QI actions for USAGE and REFERENCES

common/ComDistribution.cpp - add EXECUTE as a privilege for QI, also added USAGE and REFERENCES

sqlcomp/PrivMgrPrivileges.cpp - not generating correct security keys

fix 3: Routines were not being removed from NARoutineDB cache

Added new fields to the various routine structures for objectOwnerID, schemaOwnerID, and privInfo.

Set up the correct routineID in various routine structures

At drop time, made sure routine was removed from NARoutineDB cache

comexe/ComTdb.h - added new fields to routine descriptor and TDB

generator/Generator.cpp - new fields for routines

optimizer/NARoutine.h (.cpp) - new fields for routines

removeNARoutine - based off similar method for table cache

optimizer/NARoutine.cpp - added new field to store privilege information in NARoutine,

which also gets security keys needed for query invalidation

sqlcat/desc.h - new fields for routines

sqlcomp/CmpSeabaseDDLtable.cpp - set up new values in NARoutine structure

sqlcomp/CmpSeabaseDDLroutine.cpp - code to remove entries from cache at drop time

Other changes:

sqlcomp/PrivMgrCommand.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrMD.h (.cpp) - performance change, don't check authorization enabled

sqlcomp/PrivMgrDesc.cpp - missing object_type

parser/sqlparser.y - incorrect object type set for grant/revoke on UDRs

ustat/hs_globals.cpp - incorrect error returned

Code cleanup:

cli/Statement.h - remove obsolete code

cli/Statement.cpp - remove obsolete code

common/Collections.h - remove obsolete code

generator/GenRelMisc.cpp - remove obsolete code

optimizer/ItemCache.cpp - remove obsolete code

optimizer/RelCache.cpp - remove obsolete code

optimizer/NARoutine.h - remove obsolete code

optimizer/NARoutine.cpp - remove obsolete code

executor/SqlTableOpenInfo.h - new helper methods to check privileges

sqlcomp/PrivMgrMD.h - new helper methods to check privileges and get text for error

sqlcomp/PrivMgrDefs.h - simplification of code for checkPrivileges method

Change-Id: I981ad7f094b79a25f5e0aca30dedea4601b424ea

  1. … 39 more files in changeset.
Fix for LP bugs 1404442 & 1401683

These are duplicate problems.

A QA test suite (arkcase/arkt1112.log) contains some update statistics

tests which are starting a transaction but the transaction is never

ended.

Since the rogue transaction is started in a secondary arkcmp, the master

arkcmp is unaware of this transaction.

What differentiates this from other tests is that QA test turns off

autocommit before performing a series of update statistics.

The fix is to encapsulate a begin/commit work around DDL operations

executed by update statistics.

Change-Id: Ibbed2133e5a79f7393c9ec1b25c884a7dd8b0c3e

Fixed security issues with showstats & update stats

Related to LP bug 1401330

As part of fixing bug 1401330, found two issues:

-- core dump while performing showstats by a non DB__ROOT user

-- an indentity with MANAGE_STATISTICS privilege gets unexpected error

during an update statistics command.

Patch set 1:

Since the authentication check was move, now it needs to

deallocate the EHI structure.

Change-Id: Ib295064e45aec26a3650f712131d6f803203494a

  1. … 2 more files in changeset.
Fixes for SQL security

LP bugs fixed:

1392805 – DB_ROOT incorrectly gets “NOT AUTHORIZED” messages

1398546 – revoke priv from role fails when view is present

1401233 – USAGE privilege not checked when creating procedure (and

revoking privileges)

1403995 – Update stats failures due to schema PUBLIC_ACCESS_SCHEMA

1401683 – (Partial) DDLoperations see error 8841 about transaction

started by SQL

Regressions updated:

catman1/TEST135 & EXPECTED138

catman1/EXPECTED138 (fix in common/ComUser.cpp)

Bug descriptions:

1392805:

Changed create view code to allow DB__ROOT to create views. Some

reorganization required to make sure create view sets the updatatable

and insertable privilege correctly. This also fixed the problem where

the incorrect privileges were set when created by DB__ROOT.

Sqlcomp/CmpSeabaseDDLview.cpp

Sqlcomp/PrivMgrPrivileges.h (sets default privileges)

1398546:

The check to see if the "select" privilege is still in existence needed

to be changed until after all the privilege descriptors were analyzed.

Sqlcomp/PrivMgrPrivileges.cpp (gatherViewPrivileges)

Sqlcomp/PrivMgrDesc.h

1401233:

Missing checks at create UDR and revoke USAGE privilege were added.

Sqlcomp/CmpSeabaseDDLroutine.cpp

Sqlcomp/PrivMgrMD (getUdrsThatReferenceLibraries)

Sqlcomp/PrivMgrPrivileges.cpp (dealWithUdrs)

1403995:

This is a critical case QA filed because the PUBLIC_ACCESS_SCHEMA does

not exist for temporary sample tables during Update Statistics. If the

PUBLIC_ACCESS_SCHEMA does not exist, the temporary sample table will be

created in the same schema as the source table. Also fixed an issue for

private schemas not owned by DB__ROOT to make the histogram table's

owner the current user.

ustat/hs_cli.cpp

ustat/hs_globals.cpp

1401683:

There are several 8841 issues being detected. This is a fix for one of

them related to Update Statistics where an embedded "get" statement

causes a transaction to be started in a child tdm_arkcmp process. The

fix is to not automatically start a transaction for the get request.

generator/GenRelScan.cpp

Change-Id: Ied42fdea6c6f8c43f29dab661b06b74f0f07ff99

  1. … 13 more files in changeset.
Fixed typos in user-visible messages and sample code

Fixed spellings and minor grammar in strings used in

in output, and sample code supplied to users.

One possibly harmful misspelling was found in source code,

file sqf/monitor/linux/montim.cxx used "Contianer_ExitProcess"

where it should have used "Container_ExitProcess".

Five files which could be changed for this reason are being held back

because they are also in open change 875. They will be checked in after

that change is merged to the main branch:

conn/odbc/src/odbc/dependencies/linux/smxoevl.h

conn/trafci/src/org/trafodion/ci/InterfaceQuery.java

sqf/src/sp_proxy/sp_common.cpp

sqf/src/tm/tmlibmsg.h

sqf/src/tm/tmtimer.cpp

In patch set 2, modified these files as notde by reviewer:

conn/jdbc_type2/samples/CreateDataSource.java

sqf/monitor/test/montestutil.cxx

sql/cli/Statement.h

sql/optimizer/RelExpr.h

sql/optimizer/RelScan.h

sql/optimizer/ScmCostMethod.cpp

Change-Id: I3761e9e1518ab39415806bba0f4ed93d14e1d41c

  1. … 58 more files in changeset.
Ustat fails with duplicate CLI statement name

Under certain circumstances, processing of the ON NECESSARY

COLUMNS clause will occur while another dynamic cursor is

still open, causing the default statement name used by

HSCursor to be duplicated. When instantiating HSCursor for

NECESSARY, a unique statement name is now supplied to the

constructor.

Another unrelated change is included; the access clause for

the query to retrieve necessary columns now specifies FOR

COMMITTED ACCESS rather than FOR UNCOMMITTED ACCESS. This

is a response to a code review comment for the delivery of

ustat automation that was not included in that delivery

because of Git problems.

Change-Id: I574b62d0d665fd4b6fd79fd0c6184050bef82d63

Merge "ANSI Schema changes"

ANSI Schema changes

ANSI Schema

Implements the changes to support ANSI schemas. For more information

see the blueprint at:

https://blueprints.launchpad.net/trafodion/+spec/security-ansi-schemas

The syntax changes for REGISTER USER and CREATE ROLE were not

implemented in this delivery.

NOTE: This code was reviewed internally prior to merging with the

main branch.

Change-Id: I1c7937dbcd067e792dcacb65f12c43e4f84a25ad

Change-Id: I98395eeef1e8bde424d9e83f96928358f0b1991b

  1. … 75 more files in changeset.
Automated collection of necessary statistics

When the optimizer requests a histogram for a given column, and that

histogram does not exist, it may (depending on the cqds in effect) register

a request for the histogram to be created at a later time, or utilize a

small sample to generate a rudimentary histogram on the fly. In either

case, when a subsequent Update Statistics statements specifies the ON

NECESSARY COLUMNS clause, any column of the target table that has been

the subject of one of these actions will have a bona fide histogram

created.

blueprint ustat-automation

Change-Id: Ieceac3e1d84bf8091a5bf340dc1739a447ad2436

  1. … 1 more file in changeset.
Fixes and removal of obsolete code.

-- LP 1400556 'get tables in schema' is not supported on external

hbase tables. An error is now returned.

-- LP 1400553 Insert into external hbase tables in _ROW_ format must use

column_create function and VALUES clause to create rows.

An error is returned otherwise.

-- a bug with that prevented a boundary case when sequence increment value

was one less than largeint max has been fixed.

-- error message to indicate what options can be used during alter sequence

has been updated

-- create table as select stmt now returns an error if running within a user

transaction. This is the same behavior as other DDL operations.

This will be

removed once we have transaction support for DDL stmts.

-- create table as select now uses non-transactional 'upsert using load' to

populate target table instead of transactional 'insert...select' stmt.

-- hive/test020 has been enabled. This tests for access to ORC files.

-- obsolete sidetree insert and NVT user load code has been removed.

Change-Id: I14d321deaa52321777acd1d8ca55420f1e973367

  1. … 31 more files in changeset.
remove WITH DIVISION clause

Change-Id: I986fe56a8559f88f2781adf700933d5b03c9e697

Authorization checks for DDL & utilities

Fixed issues from code comments.

LOAD/UNLOAD authorization checks:

Code was added during code generation to make sure user has privileges,

if the user had necessary privileges, then the EXEUTIL parser flag is

turned on to avoid further privilege checks. When load/unload

completes, the parser flag is reset.

Update/showstats Statistics authorization checks:

Added a new error message

Changed hs_globals to support a new isAuthorized method and store

parser flags when class is instantiated and reset them when done

Changed hs_cli.cpp to use new IF NOT EXISTS syntax when creating

histogram tables, make owner of histogram tables DB__ROOT

(will need to adjust when schema privileges happen), and clean up

CreateHistTables method to remove old authorization mechanism

Changed hs_update.cpp which controls the update and showstats operation

to add authorization checks

Purgedata and populate index changes:

Changed CmpSeabaseDDLcommon.cpp to check privileges for purgedata

Changed CmpSeabaseDDLindex.cpp to check privileges for popindex

Additional component privileges and checks:

Added support for new component privileges in PrivMgrMD.h/.cpp

Added support for MANAGE_COMPONENTS

Added support for CREATE_INDEX and DROP_INDEX component privs

Fixes from last delivery that were postponed:

Context.cpp - fix for previous code review

CmpSeabaseDDLtable - added calls to deallocEHI

PrivMgrMD - fixed wording in a comment

Miscellaneous changes:

ComUser - added new convenience method - isRootUserID()

NATable.cpp (setupPrivInfo) to always set up privInfo_ and to call

the embedded compiler while extracting privileges

Privilege adjustments to take advantage of privInfo stored in NATable:

Added code to mark and rewind errors in diags.

Fix for LP bug 1392895

Change-Id: I6f7245ae7e66086769c0e92d901399c99e8f2af3

  1. … 28 more files in changeset.
Merge "Use HBase estimated row count if no stats exist"

  1. … 2 more files in changeset.
Change HSTableDef to recognize volatile tables

The redefinition of the isVolatile() virtual function for

some subclasses of HSTableDef originally returned FALSE, and

had not been updated since the addition of volatile tables.

Now the function always determines whether the table is volatile

based on the name of the containing schema.

The symptom of the bug was a core file produced for certain

operations on a virtual table, such as bulk load.

Change-Id: I1d592935d4922e8ce8202bf7429f929275ebae0f

Closes-Bug: #1388207

Use HBase estimated row count if no stats exist

In the absence of statistics, the row count for an HBase table

is estimated using information in its HFiles. Previouly, this

estimate was calculated and assigned to the NATable's

originalCardinality_ member variable, but this did not result

in it being used by the optimizer (the correct row count did

not appear in the results of an Explain). The code to calculate

the estimate is now invoked from HSHbaseTableDef::getRowCount(),

which has the effect of it being used as intended.

Change-Id: I3794e8368082b6f17e47fe686ec9a004aed27976

Closes-Bug: #1388918

  1. … 3 more files in changeset.
Merge "Changes to support OSS poc."

  1. … 6 more files in changeset.
Changes to support OSS poc.

This checkin contains multiple changes that were added to support OSS poc.

These changes are enabled through a special cqd mode_special_4 and not

yet externalized for general use.

A separate spec contains details of these changes.

These changes have been contributed and pre-reviewed by Suresh, Jim C,

Ravisha, Mike H, Selva and Khaled.

All dev regressions have been run and passed.

Change-Id: I2281c1b4ce7e7e6a251bbea3bf6dc391168f3ca3

  1. … 143 more files in changeset.
DBSecurity: REVOKE ROLE, credential propagation, +

Overview

1) Corrects a CLI/Executor overwrite problem and removes workaround

code in PrivMgr. Launchpad bug #1371176.

2) REVOKE ROLE now lists referencing and referenced objects when a

revoke request is refused due to dependencies.

3) REVOKE ROLE now reports that the specified grant cannot be found

when grantor has not granted the role to the user. Previously the

misleading error "Not Authorized" was issued, which as confusing when

the user was DB__ROOT. The same change was made for REVOKE COMPONENT

PRIVILEGE. A similar change will be made in the future for revoking

object privileges.

4) REVOKE ROLE now considers grants to PUBLIC before concluding a

revoke would require a dependent object to be dropped.

5) User credential are now propagated to the the compiler process.

Launchpad bug 1373112.

Externals

If the priv/role, grantee, grantor tuple does not exist, REVOKE

ROLE/REVOKE COMPONENT PRIVILEGE now reports error 1018: Grant of role or

privilege <name> from <grantor> to <grantee> not found, revoke request

ignored.

When REVOKE ROLE detects a dependent object, error message 1364 now

reports the referencing and the referenced object.

Cannot revoke role <role-name>. Object <referencing-object> depends on

privileges on object <referenced-object>.

Details for user credential propagation:

The propagate user credentials code has only been partially implemented.

The existing code sends the user ID to the first compiler process.

Other compiler processes started would not get the connected user ID

instead the DB__ROOT user ID became the user by default. Therefore,

privilege checks are succeeding when they should fail.

User credentials consist of an integer user ID and a username. The

existing code only passed the user ID. The compiler process would

then do a metadata look-up to get the username. If we kept this

model, then we would get into an infinite loop:

When the compiler process received the user ID, it did a

metadata read to get the associated username. After reading the

metadata, both the username and user ID was set in context globals.

The metadata lookup code will start another arkcmp process for the

compilation request. The compilation would then start a compiler

process. That compiler process would start another compiler process,

etc.

The solution is to send both the username and user ID to the compiler

process. Both values are known at the time the compiler process is

started. This alleviates the need for a database look-up when the

compiler process starts. To do this a new session attribute was

created - SESSION_DATABASE_USER. This session attribute sends both the

user ID and username to the compiler process during startup processing.

Once we were able to start a compiler process and store a user ID other

than DB__ROOT in the Context globals, another similar infinite loop

occurred during privilege checking. For example, a showddl command

starts a compiler process when extracting privilege information. The

compiler calls checkPrivileges to make sure the current user has

privileges. The checkPrivileges statement makes a metadata request

that requires a compilation. This starts up another compiler process.

This compiler process is sent the metadata request. When compiling the

metadata request in the new compiler process, checkPrivileges is called

which starts a compiler process, …

This worked previously because the user passed was DB__ROOT, and the code

in checkPrivileges is short circuited and the metadata call is avoided.

A fix to set the parserflag (INTERNAL_QUERY_FROM_EXEUTIL) before the

metadata request was performed. This fix requires that the file

"SqlParserGlobalsCmn.h" be included in additional files. Including this

file needs to be done with care. In order to get everything to compile,

we changed where this file was included in several places.

Once all these changes were made, the envvar: DBUSER_DEBUG now works.

If set, then good details about how users are sent to different

processes is displayed.

Change-Id: If7538eee38178c2345fe418172c6196b25a20b33

  1. … 30 more files in changeset.