Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Index-join scan trimming heuristics rework II

Change-Id: I9cf63be7967a012559e27dc4ca950bb28b8ccd3b

Merge "Launchpad: 1380826 -- TransactionScanner changes"

Merge "Fix for bug in determining range partition boundary value"

Set authorization enabled/Sequence generator privs

Code to set authorization enabled at startup

Contains changes to check authorization at process startup time and

code review comments from previous deliveries

Description of changes to check authorization at process startup time:

At process && compiler context startup time a check has been added to

see if authorization is enabled. Based on this check a new flag is set

in the compiler context.

Any operation wishing to see what the authorization status is, just need

to look at this flag.

This code has been reviewed internally by the security team.

There will be a subsequent set of changes in the PrivMgr code to return

better errors.

Details:

Arkcmp/CmpContext.h

Added a new flag containing authorization status and methods that get

and set this flag.

Sqlcomp/nadefaults.cpp

In method: NADefaults::readFromSQLTables added code that checks to see

if authorization is enabled and sets the flag in CmpContext.

It calls CmpSeabaseDDL::isPrivMgrMetadataInitialized to determine

privmgr metadata status

Sqlcomp:CmpSeabaseDDL.h/CmpSeabaseDDLcommon.cpp

Implementation of method isPrivMgrMetadataInitialized

Changed isAuthorizationEnabled to look at the CmpContext flag instead of

the flag (which was removed) in the CmpSeabaseDDL class

Changed initSeabaseAuthorization and dropSeabaseAuthorization to change

the flag in the context and kill compiler processes

Changed all calls to PrivMgrnnnn::isAuthorizationEnabled to use the

CmpSeabaseDDL::isAuthorizationEnabled or directly from CmpContext

Bin/SqlciErrors.txt & sqlcomp/CmpDDLCatErrorCodes.h to create a new

error 1234 (currently unused)

Sqlcomp/PrivMgrMD.cpp changed mapping of PrivMDStatus to match what was

done in nadefaults.cpp

Optimizer/BindRelExpr.cpp && sqlcomp/nadefaults.cpp to look in

CmpContext for authorization enabled flag

Check privileges for Sequence generator

Adds the code in compiler to check for usage privilege

for any sequence generators used in a query.

Additional privilege checks, plus

This delivery includes:

Verifying that user had correct privileges to perform all DDL

operations. This is performed through a call to

isDDDLOperationAuthorized. The signature changed to pass the object

owner instead of the object name. This eliminates an I/O and made the

method simpler. All callers were changed to use the new signature and

all DDL operations now call this method after the NATable structure has

been retrieved. A new regression test was added (TEST138).

As part of DDL privilege checking, the ALTER and DELETE component

privilege is no longer granted during initialize authorization.

Updated files to address code review checkin for change ID:

If7538eee38178c2345fe418172c6196b25a20b33.

Fixed a problem where SHOWDDL was not returning an error when user does

not have appropriate privilege.

Made the PRIVMGR_MD schema a reserved schema.

Added code to switch contexts for several PrivMgr operations. This

required a change to not grant owner privileges when creating the

OBJECT_PRIVILEGES table.

Added a KNOWN diff file for TEST133. There is an issue where rows are

not being loaded into OBJECT_PRIVILEGES during an error test.

Change-Id: I7448e7171e5f1f09feb6d1f688470b72dc1f43d4

    • -0
    • +25
    /sql/regress/catman1/DIFF133.KNOWN
    • -37
    • +21
    /sql/regress/catman1/EXPECTED133
    • -0
    • +1525
    /sql/regress/catman1/EXPECTED138
    • -0
    • +308
    /sql/regress/catman1/TEST138
  1. … 12 more files in changeset.
Only create non-existent control point tables

Change-Id: I9da8fdf8191e6480450d2c04c3f1498a47574b86

Launchpad: 1380826 -- TransactionScanner changes

Made changes to limit the cache that is gathered in the

client-side TransactionScanner. Previous implementation

would gather all rows from the server before returning any

Results on the next() call. This change will use the scan.cache

size and gather that number before returning a row to the user.

Tested by reproducing the bug in Launchpad with a 10mln row table

and hitting the GC error. Used the new TransactionalScanner

and a profiler to verify that it does not hit the GC issue and

uses a limited amount of memory.

Change-Id: I479c2a4f375e3f8a1c81ded4d66dd49a8a97bdf6

Fixes bug 1383877

Also removes some obsolete code from the GET statement processing.

Has passed developer regressions and changes have been reviewed by

the Security team.

Change-Id: I4a02b0ddb3fcf612cd69c3e05bfe2f9952e6fe43

    • -722
    • +13
    /sql/executor/ExExeUtilGet.cpp
mxosrvr exit when heapsize become large,dcs part, change after last review

Change-Id: I881aba688404ad704a4c1ea4fe27cae3dd7dda84

Merge "Enabling runtime stats for hbase operators"

mxosrvr exit when heapsize become large, code changed after last review

Related to change https://review.trafodion.org/#/c/643/

Change-Id: I084b3c704e3651503286e8fc9f60b17ca0f18fc7

    • -8
    • +116
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -0
    • +24
    /conn/odbc/src/odbc/nsksrvr/SrvrMain.cpp
Fix packaging of sql regress tests

The exclude file was excluding the sql/regress/tools/make-quoted-pattern.ksh

script used by UDR suite. This caused the test to fail when run outside of a

source tree.

Verified that removing this pattern does not change the number of files packaged

in the client and server product tar files.

Change-Id: I120bf32257a355abf06fad606a191d0fb11698ef

fix #1388277 (optimizer core)

Change-Id: I24a662c86e93451afc514c24210d415811369773

Created pre-defined test subsets

Created the following pre-defined test subsets which can be invoked

by passing QUICK1, QUICK2, PART1, or PART2 to the --tests input

parameter :

QUICK1 - Current tests used by the phoenix_quick job. Takes

approx. 15 minutes to run.

QUICK2 - Tests selected based on whether it had failed before

on elapsed time. Takes approx. 15 minutes to run.

PART1 - First half of the phoenix test divided by time. Takes

approx. 30 minutes to run.

PART2 - Second half of the phoenix test divided by time. Takes

approx. 30 minutes to run.

Also fixed file to conform to Python's PEP8 Style Guide.

Change-Id: I13f452511d821695b2f52b41a749237d4fc628da

Enabling runtime stats for hbase operators

This is the first set of changes to collect the runtime stats info for

hbase tables and operators. It contains:

1) Populate the estimated row count in hbase access TDB.

2) Collect the hbase access time and accessed row count at the JNI layer

(only for select operations now).

Partially reviewed by Mike H. and Selva G.

Removed the part that devides the estimated rows by number of ESPs based on the comments

Change-Id: I5a98a8ae9c4462aa53ad889edfe4cd8563502477

Update to version 0.9.1

Updated version number from 0.9.0 to 0.9.1

Change-Id: I5c101b9cf9dd50e1cb031c80329ee2aa56aff9cc

Ensure HFile.Reader objects are closed

Performance workloads are seeing mxosrvrs with a large accumulation

of open sockets, and a resultant decrease in performance. The cause

was HFile.Reader objects used in estimating a table's row count not

being closed. This fix executes a close() on the Reader in a finally

clause, such that it will always be executed even if an exception

is thrown.

Even with this fix, it was noted that early phases of workloads showed

a significant increase in compile times when row count estimation was

enabled. To address this, the estimated counts were saved so that they

would not be repeated as long as the NATable remained in cache. Also,

estimation was avoided completely for metadata tables. With these

changes, my own tests showed a very low overhead for estimation, on

the order of 60-70ms per user table in the workload, regardless of

query complexity or how many queries referenced the table. However,

the results of performance benchmarks executed on clusters did not

improve, and so this change also includes turning off the cqd that enables

row count estimation for the optimizer (a separate cqd controls use

of the feature for Update Statistics, and it is still on by default).

Future work will attempt to address the performance impact of turning

on the cqd.

Change-Id: Ied0369c8def5062d69766198155f8e309bae1ff8

Bulk unload fixes and rework

- rework

- fix for bug 1387377

Change-Id: I7ad6115ab50f291e2ad97a042ec2b8fbc9d256bf

Fix for bug in determining range partition boundary value

When the boundary value of a range partition function is determined by

decoding the encoded value stored as the split boundary in hbase, some mistakes

were being made. This problem was seen for nullable varchar columns. The

symptom is crash during compile with an overflowing stack. The problem

was that the buffer provided to the runtime code to decode the encoded value

was laid out incorrectly. The null bytes were interchanges with the var length

indicator bytes. Thanks to Hans suggesting t his fix.

Patch Set 2:

Chaged testcase as suggested by Hans. Also including a new fix.

This is a fix that that salted indexes to be created when the corresponding

base table has key columns of character type. The fix also allows salted index

to be created when the index is created on columns whose column number

in the base table is larger than the number of columns in the index itself.

Change-Id: Id0bd2b187500d283860d0d12ecb9d8d743b429e9

    • -6197
    • +6234
    /sql/regress/seabase/EXPECTED010
    • -5
    • +10
    /sql/sqlcomp/CmpSeabaseDDLcommon.cpp
Fixed external links and CSS stylesheet for the DCS docbook

Cherry picking changes submitted to proposed/0.8.3 by Susan

In the following XML source files in /dcs/src/docbkx, I updated

the external links (adding xlink:show="new") so that the links

now open in a separate window from the DCS book:

-- book.xml

-- configuration.xml

-- preface.xml

-- troubleshooting.xml

In book.xml, I removed unnecessary and confusing links from the

book's title and from the "Trafodion DCS" logo in the subtitle.

I changed the link in the Abstract so that the word "Trafodion"

links to the Trafodion wiki. I also added a missing period at

the end of that paragraph. I removed the question mark (?) from

the title "DCS" in the Overview section.

In preface.xml, I removed the link from "DCS version" to the

"Connectivity Subsystem" section in the Trafodion wiki (given

that the wiki content is subject to change, resulting in a broken

link) and replaced it with a more appropriate cross-reference to

the download site for the DCS package.

Per Arvind's advice, I updated the "Run modes" and "Example

Configurations" sections in configuration.xml to include the

alternate approach of specifying the host name and number of

servers.

I also updated the freebsd_docbook.css stylesheet in

/dcs/src/site/resources/css so that the DCS docbook automatically

appears in sans serif typeface (such as Arial), improving its

readability and look-and-feel.

(cherry picked from commit 4730742c501f5149fb453c9e7bfe5b4cc0e831c4)

Removed dos eol from freebsd_docbook.css as suggested by Susan.

Change-Id: Idd6f20123d628c7fd3a90b523af785575fa87ffe

Bulk load/unload fixes

- changes to sqenvcom.sh to support native compressions

for Clouder and Hortonworks distributions (tested on

cluster)

- rework from provious checkin.

- fix for bug 1387202 which cause bulk unload to hang

when target loaction is invalid.

Change-Id: Ia6046dfb2b5ff2f986b8306c26a991991a3da780

Merge "Changes to support OSS poc."

    • -13
    • +251
    /sql/optimizer/BindRelExpr.cpp
Change jdbc_test directory structure

Change jdbc_test directory structure to org/trafodion/jdbc_test

Change-Id: Ifb6e3a8e96c766c3b0c16656a14f6caf5c411f90

Merge "Make corrections to HFile path"

Merge "DBSecurity: REVOKE ROLE, credential propagation, +"

Merge "Update TOOLSDIR references for Hadoop, HBase, Hive dependencies"

Merge "fix #lp1384430: full sort order for plan1 of NJ"

Make corrections to HFile path

To estimate the row count for an HBase table, the HFiles are

accessed directly from the HDFS file system. When the Trafodion

name of the table includes a delimited id, the quotes must not

be included in the node of the HDFS path representing the

qualified table name. In addition, the hbase.rootdir property

in hbase-site.xml may consist solely of a path rather than a

full URL. It was previously assumed that a full URL would be

present, and the value of the property was used to construct

a java.net.URI object. When a string consisting of only a file

path is passed to the URI constructor, a NullPointerException

is thrown (instead of the expected URISyntaxException), causing

the path we construct to the HFile to be incorrect. The code

was changed to utilize either a file path or a valid URI as the

value of the property.

Change-Id: If35d9da7aaab815a9c1d550bc505d86f0cbcf611

Closes-Bug: 1384959

Changes to support OSS poc.

This checkin contains multiple changes that were added to support OSS poc.

These changes are enabled through a special cqd mode_special_4 and not

yet externalized for general use.

A separate spec contains details of these changes.

These changes have been contributed and pre-reviewed by Suresh, Jim C,

Ravisha, Mike H, Selva and Khaled.

All dev regressions have been run and passed.

Change-Id: I2281c1b4ce7e7e6a251bbea3bf6dc391168f3ca3

  1. … 129 more files in changeset.
Merge "Eliminate minor estimated nullcount inaccuracies"

fix #lp1384430: full sort order for plan1 of NJ

Change-Id: I7f3162af34d16bc5e801305c24438d098a9b09ef

    • -70
    • +72
    /sql/optimizer/OptPhysRelExpr.cpp