Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Removing an extraneous source tree: sqf/src/seatrans/hbase-trx2

Reason:

We inadvertently created a directory 'hbase-trx2' (when we were working on HBase0.98 in a side-branch). So want to clean that up.

Change-Id: I74c018e179786abefe80c49c5c0804cf0ab269ff

    • -67
    • +0
    /sqf/src/seatrans/hbase-trx2/Makefile
    • -56
    • +0
    /sqf/src/seatrans/hbase-trx2/README.txt
    • -29
    • +0
    /sqf/src/seatrans/hbase-trx2/build_trx
    • -171
    • +0
    /sqf/src/seatrans/hbase-trx2/pom.xml
  1. … 46 more files in changeset.
Fix for initialize authorization failure

and reenabled catman1 TEST133:

Initialize authorization creates a set of metadata tables and then loads

data into the OBJECT_PRIVILEGES table to specify object ownership values

It also loads data into the ROLE_USAGE table to specify role ownership

values. Sometimes these insert..selects fail with an error 8102. This

change does not fix the 8102 problem but fixes the code so the insert ..

select succeeds. Bascially the insert was changed to an upsert and

sanity checks placed around calls to make sure the correct number of

rows were processed.

insertSelect code changes:

Added select count(*) from target table to make sure target table empty

Changed insert to an upsert command

After upsert added select count(*) on target table to get rows found

Compare rows in target table with expected rows -> return an error if

not equal.

This fixes the problem. The upsert ignores duplicate rows so we

avoid the 8102 error. The inserted versus expected number of rows make

sure the correct number of rows were processed.

Change-Id: Id701d031ab9b9c2ebdc0584b01a2b5af9fc02b26

    • -1
    • +1
    /sql/regress/tools/runregr_catman1.ksh
    • -12
    • +43
    /sql/sqlcomp/PrivMgrPrivileges.cpp
Bug 1375902 Incorrect range boundaries for queries using ESPs

Reading unsalted, multi-region tables in parallel was very imbalanced,

except for the very first time a table was accessed in a session.

The problem was that NATable caching involves unparsing the region

boundaries, and we created these region boundary values in a way

that could not be unparsed. So, when we used an NATable object from

the NATable cache, the region boundaries had dummy values instead

of the real ones.

The fix adds code to create valid SQL literals from binary values,

to be used in the unparse method. Moving to HBase 0.98 also required

an additional fix, to handle partial values in HBase region start keys.

A region start key in an HBase 0.98 table can be a prefix of an

actual row key.

There is still a problem with keys that use the interval type,

but it is probably a day 1 issue. Since it is late in the 0.9

release, I'm deferring that to the next release.

Change-Id: Id0788a87641e7201723b5f8014215a36c87b7e23

Merge "a) undo disable txns for DDL change b) fix USE_TRANSACTIONS_SCANNER setup"

Native Hbase access improvments

Native hbase access via Trafodion SQL engine now utilizes the enhancements

made for Trafodion table like pushing mutliple rows to JNI and pre-fetch.

Also removed the dependency on the ResultIterator for the native hbase access.

This reduces yet another object created in HTableClient and improves java

memory usage on the client side.

Cleaned up state changes in the HBase access operators and removed

redundant code in the JNI layer.

Change-Id: I70ab52917aac64b68b3816b8ad834842a4d8745e

    • -187
    • +134
    /sql/executor/ExHbaseIUD.cpp
    • -181
    • +148
    /sql/executor/ExHbaseSelect.cpp
    • -469
    • +146
    /sql/executor/HBaseClient_JNI.cpp
a) undo disable txns for DDL change b) fix USE_TRANSACTIONS_SCANNER setup

Change-Id: Ib37e202b9239305bd1e38e2761b587a4316ee439

removing quasi secure mode from bulk load

in HBase .98 permissions on the /bulkload will controlled

using Access Control Lists (ACL). Quasi secure was introdiced

because HBase .94 did not have ACLs.

Change-Id: I1427d6e7ab639417010875b10a01b60696790dee

    • -83
    • +0
    /sql/executor/TrafBulkLoadClient.java
Disabling dtmci 'status regions'

Disabling 'status regions' functionality from dtmci since it needs

to be upgraded to work for hbase98.

Change-Id: I97679a6d4a71713130e62cf266318eab041fac98

Code to determine HBase version, renaming of scripts

Now using hbase shell to determine HBase version. trafodion_mods will

run the correct subscripts depending on if HBase is 0.98 or 0.94.

Renamed ambari_uninstall script to hortonworks_uninstall to clear up

some confusion and to make deleting stray ambari processes easier.

Deleting more stray rpm's in traf_cloudera_uninstall.

Change-Id: I1cce1cfdfd652ab56e9e292d92f2cbc0488a66bc

    • -131
    • +0
    /installer/tools/ambari_uninstall
    • -0
    • +131
    /installer/tools/hortonworks_uninstall
    • -209
    • +0
    /installer/tools/traf_ambari_uninstall
    • -1
    • +15
    /installer/tools/traf_cloudera_uninstall
    • -0
    • +215
    /installer/tools/traf_hortonworks_uninstall
Delivery of the Migration to HBase 0.98 branch

Change-Id: I410b90e0730f5d16f2e86a63cbffe4abaf9daa5d

    • -0
    • +26
    /conn/jdbc_type2/native/SQLMXCommonFunctions.cpp
    • -1
    • +1
    /conn/jdbc_type2/native/SrvrOthers.cpp
    • -1
    • +1
    /conn/odbc/src/odbc/nsksrvr/SrvrMain.cpp
    • -1
    • +1
    /sqf/export/include/dtm/tmtransaction.h
    • -0
    • +8
    /sqf/sql/scripts/cp_search
    • -0
    • +5
    /sqf/sql/scripts/cp_separator
    • -54
    • +79
    /sqf/sql/scripts/install_local_hadoop
    • -0
    • +269
    /sqf/sql/scripts/orc/OrcFileRead.java
  1. … 276 more files in changeset.
Support for SALTED index.

CREATE INDEX supports a new clause "SALT LIKE TABLE".

This causes SALT column to be leading column in the index

Duplicate columns in the index table are now eliminated. This is a bugfix.

Showddl and Invoke will show the SALT syntax and column respectively for

the index. CREATE index also supports HBASE_OPTIONS clause.

A bug seen when NULLABLE partitioning columns are used is also fixed.

Patch Set 2 : All rework from Patch Set 1, excpt for NATable.cpp

Patch Set 3 : Rework in NATable.cpp. Thanks to Hans for all help. Nullable

partition key columns will now generate evenly distributed dataflow through

esps.

Patch Set 4: Fix 3 issues found for the work done in Patch Set 3. Change is only in NATable.cpp

Change-Id: If378ffca29ee83dd4b7928c784b8d34d76f50049

  1. … 11 more files in changeset.
Merge "Delivering the first version of the trafodion_scanner utility."

Estmate NHase row count when stats not available

When statistics are not available for a table, give the optimizer

a better estimate of its cardinaly than the default value by

reading summary information from the trailer block of the table's

HFiles.

blueprint estimate-rowcount

Change-Id: Ie4a44bc7c87385c551eef96478147117b151ab9b

Bug 1363162.Added a change #2 suggested after code review.

Change-Id: I8acf9b5598cc8f7ac3dbcdec59b7814bc77c680e

Fixed private key file check and odd characters in license prompt.

The prompt for accepting the license had microsoft word quote characters

in it which did not render as quotes on some terminal connections.

change the characters to standard ascii quote characters.

The test for private key file was missing a logical not operator.

Change-Id: Ia628cde5008674f8b7fba85e221a790e664fe6d7

Merge "Fix for Bug #1373530. Zookeeper quorum was wrong in dcs-site.xml file."

Bug 1335477: Group by query returns different results in 2 builds

Reset the noExePreds flag in the MDAM key when we add

partitioning key predicates during preCodeGen. Those added

predicate can contain executor predicates that must not be

suppressed.

Also fixed an issue found in bug 1306836, where the syntax for

selecting specific partitions (salt buckets) was ignored for

coprocessor aggregation (simple count(*)). This syntax now disables

the coprocessor aggregate.

Change-Id: Id8dc0b7769f31a700bb459c424caada2f447a6ae

Delivering the first version of the trafodion_scanner utility.

Change-Id: I65f97dfaf98cf22943c3b3386cd132980ab4c9de

    • -0
    • +218
    /installer/tools/trafodion_scanner
    • -0
    • +84
    /installer/tools/trafodion_scanner.cfg
Merge "Fix for bug 1373179"

Merge "Start RMS processes by default"

Merge "Removed duplicate file from mpi"

Merge "PrivMgr code review rework/fixes"

Fix for bug 1373591, 1370576, fix for port issue

In traf_ambari_mods the script is now parsing for the port number and using

that to pass into the ambari configs.sh script to change HBase settings. The

script will otherwise use the default of 8080 which may be incorrect.

Bug 1373591 uninstall scripts are now removing distro repos.

Bug 1370576 cloudera and ambari setup scripts are now removing previous

installed files

Change-Id: I4160c8f0b9b246f13fb103465fe15566f541bfc2

    • -0
    • +11
    /installer/tools/traf_ambari_uninstall
    • -3
    • +10
    /installer/tools/traf_cloudera_uninstall
Start RMS processes by default

This change configures instance startup to include an mxssmp and

mxsscp process to run on each node. The performance group has

verified that there is no performance regression when RMS processes

are running.

Even though RMS processes are running, the GET STATISTICS commands

are still not able to return statistics for a query id.

Change-Id: If6321b9e756b5ec94e274d5dc0194fe804973882

Implements: blueprint enable-rms-infrastructure

Merge "1.DCS child server state remains as CONNECT_REJECT"

Fix for Bug #1373530. Zookeeper quorum was wrong in dcs-site.xml file.

The dcs_installer script was incorrectly trying to get the zookeeper

quorum node name in the single node cluster case. In the single

node cluster case the zookeeper quorum is simply the node name.

Change-Id: I69b7f857521ac9fd03fdd63742b6bdbf94ee3c6b

PrivMgr code review rework/fixes

This code has been reviewed and/or tested by some of the members of the

security team.

Overview:

1) Replaced calls to sprintf with calls to std::to_string. This avoids the

problem of the buffer potentially being too small for the written data.

2) Eliminated use of STATUS_INTERNAL error return in PrivMgr code. Usage was

inconsistent and could lead to confusion and errors. Internal errors are

now reported as STATUS_ERROR.

3) Previously construction of internal errors was not supplying the filename.

Two defines were created, one for PrivMgr (PRIVMGR_INTERNAL_ERROR) and one

for SeabaseDDL (SEABASEDDL_INTERNAL_ERROR) to generate the error completely,

using the provided string.

4) Code was aligned where previous changes had left it misaligned.

5) The command GET COMPONENT PRIVILEGES ON component FOR authID was

implemented. Also, the header for the related GET COMPONENT PRIVILEGES ON

component was updated.

6) ALTER USER now supports remapping the DB__ROOT user. Also, the command

now correctly checks the REMAP_USER privilege.

7) Checks for ID/Name mapping were not always checking the return code.

Checks were added and errors (sometimes internal) were added.

8) Buffer size for usernames was previously hardcoded, now

MAX_DBUSERNAME_LEN is used.

9) DROP ROLE now checks for granted privileges prior to removing the system

grant for the role.

10) GRANT/REVOKE ROLE now support the MANAGE_ROLES privilege for using the

GRANTED BY clause. Now a non-DB__ROOT user can grant or revoke a role on

behalf of another user, but only if they have the MANAGE_ROLES privilege.

This authority may be moved to a separate privilege in the future.

11) A member variable (myTable_) was not being freed in the destructor.

12) GRANT/REVOKE ROLE is now checking the return from insert, delete, and

update operations.

Externals:

Previously error messages 1356 and 1357 were missing a parameter.

E.g.

*** ERROR 1356 *** Cannot create the component privilege specified.

Component privilege code for the component already exists.

is now:

*** ERROR 1356 *** Cannot create the component privilege specified.

Component privilege code CR for the component already exists.

Error 1357 reports an existing component operation name.

Internals:

/cli

Context.cpp

Only change is alignment changes from a previous review.

/executor

ExExeUtilGet.cpp

o Fixed the getComponentPrivilegesForUser query-missing a parenthesis.

o Updated the headers generated for GET COMPONENT PRIVILEGES ON component and

GET COMPONENT PRIVILEGES ON component FOR authID.

o Within the work() function, corrected code for handling dual GET COMPONENT

PRIVILEGES command based on presence/absence of param1 (authID).

/generator

GenRelExeUtil.cpp

Reverted component operations back to component privileges for use in dual

command.

/sqlcomp

CmpSeabaseDDL.h

Added SEABASEDDL_INTERNAL_ERROR define.

CmpSeabaseDDLauth.cpp

CmpSeabaseDDLauth.h

o verifyAuthority() in the base class was removed, and the version in the user

class was enhanced to support verifying the authority to remap users.

o Internal errors are now generated using the SEABASEDDL_INTERNAL_ERROR define.

o CmpSeabaseDDLuser::alterUser now supports remap the external username for

DB__ROOT. Previously reserved names were prohibited by the ALTER USER

command, now an exception is made for DB__ROOT if the operation is setting a

new external name.

o CmpSeabaseDDLrole::dropRole now checks for privileges granted to a role

before removing the system grant of the role. Previously the system grant

could have been removed leaving the role ungrantable.

CmpSeabaseDDLcommon.cpp

o Internal errors are now generated using the SEABASEDDL_INTERNAL_ERROR define.

o grantRevokeSeabaseRole() now uses the MANAGE_ROLES privilege to determine if

a user can grant or revoke a role on behalf of another user. Previously

this was restricted to DB__ROOT. This authority may be moved to a separate

component privilege in the future.

o GRANTED BY clause now restricted to DB__ROOT for GRANT/REVOKE COMPONENT PRIVILEGE.

PrivMgrCommands.cpp

o Replaced usage of STATUS_INTERNAL with STATUS_ERROR.

o Removed unused grantRoleToCreator() - CREATE ROLE calls PrivMgrRoles class

directly.

PrivMgrComponentOperations.cpp

o Member variable myTable_ is now deleted when the instance is freed.

o Corrected generation of error message for existing component name or code to

use the correct parameter constructor type.

o Internal errors are now generated with PRIVMGR_INTERNAL_ERROR define.

o UIDToString (which calls std::to_string) used instead of sprintf to convert

component UIDs to strings.

PrivMgrComponentPrivileges.cpp

o Member variable myTable_ is now deleted when the instance is freed.

o Internal errors are now generated with PRIVMGR_INTERNAL_ERROR define.

o UIDToString (which calls std::to_string) used instead of sprintf to convert

component UIDs to strings. Similarly, authIDToString is called to convert

granteeIDs and grantorIDs to strings.

o grantPrivilege() no longer removes WITH GRANT OPTION when a privilege a user

has WITH GRANT OPTION is re-granted to the user without WITH GRANT OPTION.

Instead, the grant is now a nop.

PrivMgrComponents.cpp

o Internal errors are now generated with PRIVMGR_INTERNAL_ERROR define.

o UIDToString (which calls std::to_string) used instead of sprintf to convert

component UIDs to strings.

PrivMgrDefs.h

Removed STATUS_INTERNAL from the list of PrivStatus enums.

PrivMgrMD.h

o Added define PRIVMGR_INTERNAL_ERROR.

o Added static inline functions authIDToString and UIDToString. Respectively

they take an int32_t and int64_t and return a std:string. Internally they

both cast the value to long long int as that is the only equivalent type

supported by the current (4.4.6) gcc compiler.

PrivMgrMD.cpp

o Internal errors are now generated with PRIVMGR_INTERNAL_ERROR define.

o UIDToString (which calls std::to_string) used instead of sprintf to convert

component UIDs to strings.

o A grant of select to the AUTHS table to PUBLIC was removed. This was added

as a workaround, but another workaround (allow SELECT on all metadata tables)

superseded this one.

PrivMgrPrivileges.cpp

o Internal errors are now generated with PRIVMGR_INTERNAL_ERROR define.

o Checks for STATUS_INTERNAL were removed.

PrivMgrRoles.cpp

o Member variable myTable_ is now deleted when the instance is freed.

o Internal errors are now generated with PRIVMGR_INTERNAL_ERROR define.

o authToString is called to convert roleIDs, granteeIDs, and grantorIDs to strings.

o grantRole() no longer removes WITH ADMIN OPTION when a privilege a user has

been granted a role WITH ADMIN OPTION and the role is re-granted to the user

without WITH ADMIN OPTION. Instead, the grant is now a nop.

o The error return from insert or update is checked following a grant as well

as from delete and update following a revoke, and an internal error is

reported if the operation fails.

Change-Id: I6d9fc31455222f28cdc5d0db65dc75fc8bb4a99e

    • -91
    • +75
    /sql/sqlcomp/CmpSeabaseDDLauth.cpp
    • -164
    • +154
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
    • -10
    • +13
    /sql/sqlcomp/PrivMgrComponents.cpp
    • -37
    • +14
    /sql/sqlcomp/PrivMgrPrivileges.cpp
    • -101
    • +58
    /sql/sqlcomp/PrivMgrRoles.cpp
Fix for bug 1373179

Native Hbase Table access via Trafodion dumps core.

The error messages passed from Java layer to JNI was not handled

correctly leading to core dump. Fixed the error handling code correctly.

Also flagged three Java Objects in HTableClient for gc earlier to reduce the

java object memory usage.

Separated out the relase and flush HTableClient as two different operations

to ensure that HTableClient can be continued to be used even after flush.

Changed NAString in error routines to use the relevant heap instead of

system heap.

Change-Id: I3fac1b8de4745ec9c6f47fc8f64b483f9cc6ef36

    • -100
    • +16
    /sql/executor/HBaseClient_JNI.cpp
    • -5
    • +26
    /sql/executor/JavaObjectInterface.cpp
Ensure UID returned for MD tables is non-null

When an NATable is created for a metadata table, the UID is not known.

Under certain conditions, this creates a problem whereby the AUTHS

table can not be read, and access to a table is incorrectly denied.

In this fix, the NATable::objectUid() function is changed to do a

lookup of the UID for a metadata table if it is 0, and to store the

result in NATable for future calls.

Change-Id: I5eb58871070e585a7242f92becde7e6a24a8f5f5

Closes-Bug: #1370289

Fix for bug 1356069

Fix for bug 1356069 :

Executor assert may not dump core within embedded compiler

Change-Id: I4fcceb6ba2ae65acc23ff33011e97f8d7c03d2f0