Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
TRAFODION - 3218 User still has privilege after user's role has been revoked ...

Partial support for column level privileges with QI support for:

column select

column insert

column references

column update

Also, as part of this, updated privilege code in a couple of areas:

Changed object caching code in NATable and NARoutine to store all privileges

assigned to the object when the object is cached (privDescs_). During the load

operation, the code creates bitmaps (privInfo_) for the current user. Privilege

checks are performed against the user bitmaps (privInfo_). This is in

anticipation for some performance updates when connecting to Trafodion (mxosrvr)

with different users.

Change getRoleList to include the roleID and the granteeID that granted the

privilege. The grantee can be a user or a role.

When a privilege is revoked from a role, send QI keys for every user that has

been granted to role.

  1. … 40 more files in changeset.
Merge remote branch 'origin/master' into ansharma_nnpk_br

Conflicts:

core/sql/regress/seabase/EXPECTED031

core/sql/sqlcomp/CmpDescribe.cpp

core/sql/sqlcomp/CmpSeabaseDDLtable.cpp

core/sql/sqlcomp/DefaultConstants.h

  1. … 12 more files in changeset.
nonnull pkey, commit #1

  1. … 22 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 90 more files in changeset.
TRAFODION-2731 CodeCleanup: Phase 4. Remove legacy/obsolete pragmas

  1. … 392 more files in changeset.
lcov: commit #1

  1. … 333 more files in changeset.
TRAFODION-2731 CodeCleanup: Remove obsolete, legacy and unused code

This phase handles the following:

-- removal of code that dealt with:

-- mpalias, NSK, MP, mploc, resource fork, rfork

-- ARLIB, DISK, VOLUME, PFS, compiler version info

-- interpretasrow/IAR, AuditImage, ExtractColumns functions

-- ARKCMP_SINGLE_PROCESS and oneProcess()

-- recompControl, remoteDefaults, rtdu, module

-- latebind thru nsk defines, guardian names, nametype nsk

-- SHADOW implementation

-- MEASURE

-- older sqlcat ReadTableDef

-- DP2_MEMORY, DP2_SPACE, HGB_DP2_MEMORY_LIMIT

-- internal cli methods no longer used by any caller

Code within the following defines is removed if it is obsolete

or the define itself is removed if that feature is always on:

-- removed NA_EIDPROC

-- removed SQLEXP_LIB_FUNC

-- removed NA_CMPDLL

-- removed SQ_PHANDLE_VERIFIER

-- removed SQ_NEW_PHANDLE

-- removed __EID

-- removed ARKFS_OPEN

-- removed STAND_ALONE

-- removed __TANDEM

-- removed NA_C89

-- removed NA_NSK

-- removed SQLEXPORT_LIB_FUNC

-- removed SQLCLI_LIB_FUNC

-- removed CLI_PRIV_SRL

-- removed PRIV_SRL

-- removed NA_LINUX

-- removed NA_HSC_LINUX

-- removed NA_UNIX

-- removed NA_WINNT

-- removed HAVE_MMAP

-- removed NA_NO_C_RUNTIME

-- removed NA_DEBUG_C_RUNTIME(replaced with _DEBUG)

-- removed NA_64BIT usage except in sqlcli.h

-- removed dg64

-- removed SQLEXPORT_LIB

-- removed NA_ARKFS

-- removed NA_IEEE_FLOAT

-- removed NA_GUARDIAN_MSG

-- removed NA_HSC

-- removed NA_TMFNOTYETSUPPORTED

-- removed ERROR

-- removed ERROR_STATE

-- removed SQLERRORS_LIB_FUNC

Contents of these files have been removed.

Next checkin fill remove the files itself from git:

executor/dmeasql.h

executor/ExMeas.h, ExMeas.cpp

executor/tempfile.h, .cpp

executor/rcb.h

executor/stubs.cpp, stubs2.cpp

exp/srlversion.cpp

cli/rtdu.h, rtdu2.h, rtdu.cpp, rtdu.cpp

cli/VicKeyValuePair.h

cli/CliDll.cpp

cli/CliStubsStaticBuild.cpp

cli/globalsrlversion.cpp

cli/globalstubs.cpp

cli/sqlciSRLStubs.cpp

cli/test.cpp

cli/privsrlversion.cpp

common/SqlExpDllDefines.h

common/SqlExportDllDefines.h

sqlcat/enum.h

sqlcat/ReadTableDef.h, cpp

sqlcat/readRealArk.h, cpp

  1. … 460 more files in changeset.
[TRAFODION-2584] Add support to register hive objects in traf metadata

Syntax:

register hive table/view [if not exists] <object-name> [cascade]

unregister hive table/view [if exists] <object-name> [cascade]

cascade option: register/unregister all underlying objects that are

part of the specified view

update statistics, grant/revoke, traf views or external table creation

on hive objects will automatically and internally register those objects

in trafodion metadata.

invoke/showddl will show if this object is registered and whether

that registration was internal or explicit.

Get command extensions:

get hive registered tables/view/objects in catalog trafodion;

get hive external tables in catalog trafodion;

Cleanup command extensions:

cleanup metadata command will clean up inconsistent hive objects

(underlying hive object is missing but object is registered or

an external table exists)

cleanup [hive table | hive view] on <object-name>;

Existing hive objects that had implicit or explicit external tables

created prior to this checkin will have no change in behavior.

ObjectUID of those external tables will continue to be used for

relevant operations.

One can drop those external tables and explicitly register them,

or a subsequent operation(upd stats, grant, etc) that needs objectUID will

automatically register them.

minor changes based on review comments of previous checkin

get all objects command on hive metadata no longer fails.

get views on objects return 3-part name that could be used to

differentiate between a traf and hive view.

regress/hive/TEST007 has been extended.

TBD: Add register/unregister privileges

  1. … 52 more files in changeset.
[TRAFODION-2583] Remove deprecated CQD HIVE_MAX_STRING_LENGTH

  1. … 29 more files in changeset.
TRAFODION-2492 Support for native hive views, plus few more changes

-- added support for native hive views created externally. These views

can be accessed in a traf sql query.

There are some restrictions as listed below:

-- privileges support has not yet been added. That will be done, as

needed, in the next checkin.

-- usage info for tables contained in a native hive view is not maintained.

-- native hive views cannot be inserted/updated/deleted

-- cannot update stats on a native hive view

-- support for GET statements to retrieve hive data

get tables/views/objects in schema hive.sch;

-- this will return all tables/views in specified schema(hive database)

get tables/views/objects in catalog hive;

-- this will return all tables/views in hive

get schemas in catalog hive;

-- this will return all schemas(databases) that exist in hive

-- GET statements to retrieve hive data now supports optional MATCH clause.

This can be used to filter results.

-- showddl will show if an external table on hive was created implicitly

for internal usage (upd stats, privileges, views) or explicitly by user.

Internal creation of external table now use 'create implicit external table'

syntax and that info is stored in metadata.

-- drop external table will drop that table even if underlying hive table

is missing. Currently it returns an error.

-- at runtime, accessing a missing hive table now returns a better error

message instead of an internal "data mod check failed" message.

-- regress/tools/runallsb has 2 new options:

-basetests (this will run core,compGeneral,executor,seabase,hive)

-othertests (this will run all other tests suites)

-- optimizer/NATable.cpp was incorrectly indented by some checkin from

a few days back. TABs were used instead of spaces which caused code

to become all crooked. That has been fixed.

-- regress/hive/TEST007 has been updated with additional hive view tests

  1. … 26 more files in changeset.
[TRAFODION-2440] Add retry to row count estimation logic

  1. … 16 more files in changeset.
TRAFODION-1586 Add support for external Traf tables mappedd to HBase tables

This is the initial support for HBase mapped tables.

A doc for external manuals will be added later.

regress/seabase/TEST022 contains various testcases.

  1. … 77 more files in changeset.
TRAFODION-2327 Reduce I/O when loading objects into caches

For each authorization ID (user, role, or PUBLIC), a bitmap containing the

accumulated privileges (across all grantors) is stored with the object desc.

When the object desc is loaded into cache, the privilege bitmaps associated

with the current user, PUBLIC, and the current users' roles are extracted and

unioned together to calculate the final set of privileges. This unioned list

is used during privilege checking.

Today, an I/O is performed to retrieve the list of roles granted to the current

user for each object loaded into NATable and NARoutine cache. Since this list

does not change unless the current user changes (a new session with a different

user) or a grant/revoke role for the current user is performed, these extra

I/O's are not needed.

To remove the extra I/O's for each object, the list of roles will be stored in

the ContextCli. Therefore, this in-memory role list can be used instead of

rereading metadata.

This checkin creates two new CLI requests:

- GetRoleList - returns the list of roles associated with the user

If the list exists in ContextCli, it returns the stored values

If the list does not exist, it retrieves them from Metadata, stores

them and returns the values

- ResetRoleList - removes the list of roles from ContextCli

The first time GetRoleList is called in a session, the users' roles are

stored in ContextCli. They remain in memory until the session ends and

restarts as a different user, or another process grants or revokes a role

from the current user.

If another process revokes a role from the current user, a query invalidation

key is created. When the revoke role query invalidation key for the current

user is detected, ResetRoleList is called. The next time GetRoleList is called

an updated role list is retrieved from metadata and stored in ContextCli.

If another process grants a role to the current user, there could be two

outcomes. If the current user already has the privilege from another source

then nothing happens. If the current user does not have the privilege, then

one recompilation is attempted. Prior to performing the retry, code was

added to ResetRoleList. The recompilation then gets the latest role list and

either succeeds or fails depending on the granted privileges.

  1. … 16 more files in changeset.
[TRAFODION-2282][TRAFODION-882] New scheme to invalidate histogram cache

  1. … 27 more files in changeset.
Create NACollection objects on NAHeap. Avoid use of system heap as far as possible

  1. … 50 more files in changeset.
TRAFODION [2137] Improve metadata access time during query compilation

A change was made to return privilege information in the descriptor structure

instead of getting it when the NATable or NARoutine object is instantiated.

For tables, storing privileges in the descriptor structure allows privileges

to be saved with other table attributes in the metadata. This improves metadata

access time during initial query compilations.

Changes:

--> At create time or when the object's DDL changes (redeftime), the compiler

gets the list of privs for all users. If stored descriptors is enabled,

this list is stored as part of the object definition in the TEXT table.

--> PrivMgr returns a list of bitmaps for all users granted any priv

--> the list of privs is transformed into a VirtTable

--> the VirtTable is transformed into TrafDesc

--> a packed form of the TrafDesc is stored in the TEXT table

--> When an NATable or NARoutine is instantiated, the current user's credentials

are extracted from the TrafDesc and stored in the class thereby eliminating

the need to perform I/O to get privs for the user.

  1. … 22 more files in changeset.
Merge branch 'master' into trafodion-1788

Conflicts:

core/sql/common/ComSmallDefs.h

core/sql/generator/Generator.cpp

core/sql/optimizer/NATable.cpp

core/sql/sqlcat/ReadTableDef.cpp

core/sql/sqlcat/desc.h

  1. … 20 more files in changeset.
metadata access performance improvement for accessing traf objects

Packed version of descriptor structures is now stored in traf TEXT

table during various DDL operations.

During dml, if packed descriptors exist, then they are read and used.

If they dont exist, then descriptors are generated on the fly.

Descriptor classes are defined in file sqlcat/TrafDDLdesc.h and

are based off the older desc structures from desc.h.

These are simplified and class'ified version of the older structures

and are versioned. Changes to these classes should be done carefully

by either using filler bytes or versioning them.

2 new cqds, traf_store_object_descriptor and traf_read_object_descriptor,

are added to control if packed descs should be stored/read.

These are currently set to OFF by default but are enabled during

regressions run. Once this feature is tested, these cqds will be turned

on by default.

In addition, some helper commands have been added to manipulate descrs.

Syntax:

alter table t [check|generate|delete|disable|enable] stored descritors

If authorization is enabled, then auth/priv specific info

is generated on the fly and currently not stored in metadata.

In a later checkin, these will also be stored in metadata.

  1. … 74 more files in changeset.
[TRAFODION-1882]: Column Privilege: a user can grant column privilege to ... [TRAFODION-1788]: Grant and Revoke on table columns with referencing views ...

The main issue is that the view-col <=> referenced-col usages were not available

from the metadata.

-- create view was changed to add view-col <=> referenced-col usages to the

TEXT table. This allows NATable and privilege management to retrieve this

information. No upgrade is required

-- Privilege management was changed to see if views could still exist based

only on column level privileges.

-- Grants and revokes on referenced columns for objects are now propagated to

to referencing views

-- Fixed an issue during column grants where column ordinals were not checked

correctly [TRAFODION-1882]

-- Fixed an issue where DB__ROOT, acting as schema owner, was able to grant

privileges that the schema owner was not able to grant,

  1. … 15 more files in changeset.
JIRA TRAFODION-2072 plus few more

-- long varchars (> 32K) are handled for traf (regular and aligned)

and hive tables

-- showddl for hive tables shows traf and hive definitions

-- traf external tables can optionally take columns attrs that would

overwrite corresponding hive col attrs

-- metadata is updated with info no whether col attrs were specified

-- 'repeat' function can optionally take a max length parameter

-- error is returned if an external table is created for non-hive

-- tests added to regress/seabase/TEST031 and hive/TEST009

  1. … 44 more files in changeset.
[TRAFODION-1955] JNI optimization at the time of compilation

Removed ByteArrayList class and used array of byte array instead.

Only one JNI to java transistion is done to getRegionStartKeys

and getRegionEndKeys at the time of query compilation

Also fixed an issue with lib_mgmt Makefile.

  1. … 15 more files in changeset.
TRAFODION-1940 better NATable statistics plus OID, cleanhb fixes

TRAFODION-1939 Remove OID from the list of reserved words

TRAFODION-1940 Improve NATable statistics

- Add the context of the NATable cache

to the natablecache() UDF

- Add the entry size to the

natablecacheentries() UDF

Also changed the ilh_cleanhb script to print a help text

on how to use it and require a parameter 0, 1 or 2.

Still leaving the old, misspelled directory for zookeeper

in the script until everyone has moved to a later version

that puts the zookeeper data under local_hadoop/data/zookeeper.

  1. … 9 more files in changeset.
update

  1. … 298 more files in changeset.
JIRA 1849 Add support for alter column rename (details below)

-- support for 'add column c rename to newc'

-- fixed issue with ddl stmts and aqr, it now works correctly

if ddl_transactions are being used

-- drop table with views now drops them in reverse creation order.

That way dependent views are dropped correctly.

-- views are dropped before and recreated after an alter column

statement.

-- dev regressions tests now show system columns in invoke.

-- index uid is now stored in desc and NATable structs

  1. … 51 more files in changeset.
JIRA TRAFODION-1798 (ddl xns) and few other fixes, details below.

-- support for sql part of ddl xns. Section 1 of JIRA TRAFODION-1798

-- cqd ddl_transactions to enable or disable ddl xns.

Default is currently off. Once it is tested, it will be turned on.

Dev regressions are run with cqd set to ON

-- get stmts run with read committed to get changes in current xns

-- support for where preds with get stmts

-- scan to pass in transid even if running with read uncommitted access.

This enables rows modified in current xn to be returned.

-- cleanup no longer return multiple duplicate error messages if

objects id is not found.

-- cleanup no longer includes internallay created schemas (_HV_ , _HB_)

during cleanup operations.

-- Correct error msg was not getting returned if an invalid index

existed in table and the same index was created again.

-- init traf, drop md views was giving an error if views didnt exist.

That has been fixed.

-- regressions with -diff option now show original file timestamps

instead of the timestamp when the diff command was run.

  1. … 72 more files in changeset.
jira-1443 final.

  1. … 16 more files in changeset.
Merge branch 'master' of github.com:apache/incubator-trafodion

Conflicts:

core/sql/sqlcomp/CmpSeabaseDDL.h

core/sql/sqlcomp/CmpSeabaseDDLtable.cpp

  1. … 12 more files in changeset.
Fixes for Jira's trafodion-19 & trafodion-29

See Jira's for a description.

Also removed obsolete code from the parser

  1. … 46 more files in changeset.
TRAFODION-1457 Issues related to decoding HBase region keys and TEXT table

  1. … 41 more files in changeset.
Merge remote branch 'github/master' into fixJira1425_1432

Conflicts:

core/sql/optimizer/NATable.h

  1. … 4 more files in changeset.