Trafodion

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Update Statistics performance improved by sampling in HBase

Update Statistics is much slower on HBase tables than it was for

Seaquest. A recent performance analysis revealed that much of the

deficit is due to the time spent retrieving the data from HBase that

is used to derive the histograms. Typically, Update Statistics uses a

1% random sample of a table’s rows for this purpose. All rows of the

table are retrieved from HBase, and the random selection of which rows

to use is done in Trafodion.

To reduce the number of rows flowing from Hbase to Trafodion for queries

using a SAMPLE clause that specifies random sampling, the sampling logic

was pushed into the HBase layer using a RandomRowFilter, one of the

built-in filtersrovided by HBase. In the typical case of a 1% sample,

this reduces the number of rows passed from HBase to Trafodion by 99%.

After the fix was implemented, Update Stats on various tables was 2 to 4

times faster than before, when using a 1% random sample.

Change-Id: Icd40e4db1dde444dec76165c215596755afae96c

Move Seapilot from qpid::client to qpid::messaging API.

We need to do this because the qpid::client API is no longer supported

and is no longer part of QPID versions after 0.26.

Change-Id: I87c242b85dc9171a07b18c533f7a596d819a9fd1

    • -1
    • +1
    /sqf/seapilot/source/chameleon/Makefile
    • -3
    • +0
    /sqf/seapilot/source/wrapper/ProducerWrapper.cpp
  1. … 7 more files in changeset.
CREATE PROCEDURE generates core (bug 1335186 - attempt 2)

Fixes bug 1335186 introduced after Directory structure change.

Previous fix handled unversioned files Vproc.cpp and Vproc.h

along with DriverInfo template file.

This fix removes generated file DriverInfo.cpp from source control.

Since the file was existing in the directory, ANT copy didn't generate

the file from template leading to the version error.

Change-Id: I4c8bd220f489805723aa6725c3884b5a923ace39

Corrects problems w/authetication error reporting

1) Error not reported when authentication fails (LP 1321059)

2) Authentication error reporting should be moved out of CLI (US

1036)

In addition, the template authentication configuration file refers to

the wrong script for enabling authentication. (LP 1327302)

Analysis and Design:

When support for Zookeeper was added to MXOSRVR, authentication was not

active. The code to retrieve and handle errors was moved later in the

connection logic. Unfortunately, by the time it was invoked, the error

was no longer available, and a generic error was reported by an

exception handler. The check is move back to its previous location.

Previously the CLI would load authentication error codes and text into

the diags area, and then MXOSRVR would extract it. Now, MXOSRVR simply

generates its own error, although currently it looks a lot like the old

error (minus the CLI reference).

After the authentication configuration template was delivered, the name

of the script to enable authentication was chosen. The template was not

updated due to time constraints.

Externals:

Previously the error messages from authentication were of the form:

*** ERROR[8837] CLI Authentication : User: external-username : invalid

username or password

The new format is:

*** ERROR[8837] Invalid username or password. User: external-username

The format of this message will likely change, but for now the original

CLI range error is retained.

Change-Id: I8ab58404941aaf50319883071bb4142cbaafb557

    • -22
    • +30
    /conn/odbc/src/odbc/nsksrvr/SrvrConnect.cpp
    • -6
    • +2
    /conn/odbc/src/odbc/nsksrvrcore/SQLWrapper.cpp
    • -3
    • +1
    /conn/odbc/src/odbc/nsksrvrcore/SQLWrapper.h
    • -27
    • +69
    /conn/odbc/src/odbc/nsksrvrcore/srvrothers.cpp
enable embedded compiler support for fetch histogram queries

The fetch histogram logic executes queries against the user metadata

tables HISTOGAMS and HISTORAGM_INTERVALS that require the spawning of

external arkcmp processes. The work in this check-in enables embedded

compiler for these queries so the spawning of external arkcmps is avoided.

The change introduces a "switch context" to a meta data context when

compiling the user metadata queries and "switch back" to the user context

prior to fetching the rows from the cursors on the user metadata tables.

The main change is implemented in the "readHistogram" function in

hs_read.cpp file.

A new method (switchBackContext) was added to class CmpContext class

to enable switching back to a previous context. The purpose of the new

method is to ensure that thread globals are appropriately reset when

reverting back to a previous context.

The checkin also turns OFF CQD HBASE_SERIALIZATION in file regress/tools/sbdefs

as the hbase serialization feature is not enabled yet. The feature (CQD)

was enabled during regression testing prior to this checkin.

Change-Id: Iee1b8152dee3fea0ce3d3354719b5b7aa8fb198f

Enabling thread independent JNI objects

In addition the comments raised while this change was in zulu are also being addressed. See zulu for details using the earlier link. Also, the changes made for those comments were lost while moving to Jenkins. I have redone those changes again and committing it now

Change-Id: I5b709e1e01f474ade306b19ab21b53ee35997f25

bulk load changes (rework)

the changes include:

- renaming the hbase-extensions.jar to trafodion-hbase-extensions.jar

- changes to bulk load test and expected files

- changes to HBulkLoadClient.java

- changes to makefile

Change-Id: Ib807951fb78030219456b8c1949190d9065101f9

    • -51
    • +84
    /sql/executor/HBulkLoadClient.java
Change sqlci to NOT route stdout through monitor.

Change-Id: Ifa677c978b7b4f0471e72c89339844ceb6981c6e

Migrating to review.trafodion.org

Change-Id: I6b772ec92fdbda83fdaeedcbb8ff2a9bc91a6066

Migrating to review.trafodion.org

Change-Id: I9af54575966d74360e15270fc9b173760ecde868

Migrating to review.trafodion.org

Change-Id: Ia6a35c451feda3c5b395e7435006d5aa91dbb2b8

Migrating to review.trafodion.org

Change-Id: I5f43847971cc7291af8d28ee859635db4cf5fd83

Migrating to review.trafodion.org

Change-Id: I0a314b9f3e698744432e119ad880f42507a38e2a

Fix for bug 1335186

Core seen during CREATE PROCEDURE.

Happended due to version mismatch between Type 2 JDBC and JNI files.

Template file had an old package name.

Change-Id: I51323059f0dc1091f841c847bf6be4bbf7a69868

Merge "Squashed commit of the following:"

Removed extra underscore in variable name AMBARI__URL

This made it impossible for the suer to override the default

Ambari url with a different port.

Change-Id: Iee4c55c6bc80866870e4b4653177d4a92a362d77

Merge "Using embedded compiler will not get error 2031"

Merge "Fix for query cache issue for Hive selects."

Squashed commit of the following:

commit 1b8106079000418f4afa6ce0e247e81f3f5b2e2c

Merge: 7b7c311 e79bbdf

Author: Justin Du <justin.du@hp.com>

Date: Thu Jun 26 13:41:50 2014 -0800

Merge remote branch 'gerrit/master' into bp/reentrant_cmp

Conflicts:

sql/common/arkcmp_proc.cpp

Change-Id: I56a9cd33544e863c233a1b4e6a811bfb15efe27e

commit 7b7c3110c55f837672240f25dd5496540212ba17

Author: Justin Du <justin.du@hp.com>

Date: Mon Jun 23 13:33:27 2014 -0800

Expect file changes for executor/TEST013

To report the expected error (8193) when the schema name has reserved

word as perfix for either types of statements.

Change-Id: I8ce1854afa1d916eecd01c70d27a89857fb6c2a9

commit 376666657d99b200fbcf8cd5e177096619d44c0e

Author: Justin Du <justin.du@hp.com>

Date: Thu Jun 19 13:03:06 2014 -0800

Pass diags info between CmpContext instances

1) Preserve the diags info when restoring cqds and controls after meta

data access.

2) Pass diags info from current CmpContext to the previous CmpContext at

the CmpContext switch back call.

Change-Id: Ibac9ff19c82f8dc17f278ec4327f39837085503e

commit b22407a2b78ff8c2b7f1ea14c908833fe8e98540

Author: Justin Du <justin.du@hp.com>

Date: Fri Jun 13 17:08:24 2014 -0800

Rework after review

Added warning (SQLCODE 2032) and assert when improperly using CmpContext

switch logic

Change-Id: I32e6e4f2168e3cf52dc58ec85da1555f4fd29051

commit 2870519283471f6b3c5ea844875546b0dda32a04

Author: Justin Du <justin.du@hp.com>

Date: Thu Jun 12 10:14:12 2014 -0800

Bug fixes related to CmpContext switch

1. CmpContext switch takes place only if the embedded compiler is

involved.

2. Fix for TMF error 75 (process doesn't have the active transaction)

see in DDL operations.

Change-Id: Icdca929c4c782464e1f8267d497bf91518a8b3a1

commit 1cebd4b2c6134097a2b4bd419b00fdee71d93f1f

Merge: 8145ba5 0862f91

Author: Justin Du <justin.du@hp.com>

Date: Tue Jun 10 11:21:11 2014 -0800

Merge remote branch 'gerrit/master' into bp/reentrant_cmp

Conflicts:

sql/arkcmp/CmpStatement.cpp

Change-Id: Iba12fa3fb64809f9c9393d06db0215d7bbf6ee7d

commit 8145ba58927d0d954b68183031747a77367e000f

Author: Justin Du <justin.du@hp.com>

Date: Tue Jun 10 10:12:59 2014 -0800

Associate global empty input LogProp with CmpContext

1. Changed GLOBAL_EMPTY_INPUT_LOGPROP as alias (via #define) instead of

thread pointer to access the default input LogProp from curent

CmpContext.

2. Restored recursion counter for embedded compiler so that histogram

access is done by external compiler

Change-Id: I32560e3b0b1dfe5fc2b1ee9a839d75bfdb57fa9a

commit 80b36c32a0db1cf0dc5ae5cead2a04522da1231e

Author: Justin Du <justin.du@hp.com>

Date: Mon Jun 2 20:58:47 2014 -0800

First set of recursive compilation with CmpContext switch

1. Disable recursion counter when entering compileDirect

2. Starting or reuse CmpContext for metadata access during compilation

3. Fixed few problems in CmpContext switching code.

Change-Id: Iff66309319e989a247b80d92ea8e3e32a35e1755

Change-Id: Ib096a46288a8616fc26c9f52474873778e76ed8b

  1. … 4 more files in changeset.
Added creation of working directory on all nodes, not just first node.

The lack of the working directory on all nodes was causing setup to

fail during copying files to all nodes.

Change-Id: I69533fe1c56e5d503dc37c30aa38b2b827f297c8

Added --assumeyes to yum remove commands so they don't prompt and hang

Change-Id: I6baefee7c5174ecbb19cf004ca7e344cd5e7a173

    • -10
    • +10
    /installer/tools/traf_ambari_uninstall
Merge "JDBC T4 samples README file cleanup"

Using embedded compiler will not get error 2031

There is no need to raise error 2031 (Could not use embedded compiler to

compile the statement: ...) when failed to compile a query. Removed the

AQR entry for SQL error 2031.

Change-Id: I3979574ea9405d0b35c0c7924e30ad64475c6c10

Merge "trafodion bulk load changes (disabled))"

Merge "Fix for bug - Zookeeper not in connecting state"

Fix for query cache issue for Hive selects.

Two changes

1)Fix for bug #1293816.

2)Discontinue linking in libprotobuf.so, since it is currently unused.

For the hive query cache bug, the issue was that any change to a HDFS file

in a Hive directory, or to the directory itself (add/drop a file), was not

reflected in the query cache key. So the compiler could give a plan with an

incorrect list of HDFS files to the HDFSScan operator. The fix is to add

max(fileInfo.mLastMod) to the query cache key. The max is taken over all

files for a given Hive table. The number of files for a given Hive table has

also been added to the query cache key to cover cases where a file is deleted

from a Hive directory. Both query cache and query text cache are addressed.

The mLastMod time for each file and the number of files are determined through

the libHdfs call hdfsListDirectory(), which we already make.

Linking in libprotobuf.so is causing issues on certain MapR clusters since

MapR also uses this library and sometimes the version used by MapR is

different from what Trafodion uses. Since this library is not being used by

Trafodion stack right now, we will no longer link in this library in SQL or

connectivity layers. When a fix is found for the version incompatibility

issue, this change will be reversed.

Patche Set 2.

Thank you Dave for catching these issues. They have been resolved in

Patch Set 2.

Change-Id: Idbe599a876fdcaf77d2bdb9fdbf4b77a3f431e46

    • -1
    • +4
    /conn/odbc/src/odbc/nsksrvrcore/Makefile
    • -0
    • +2
    /sql/nskgmake/tdm_sqlmxevents/Makefile
Fix for 1318788 - Create salt table keyed on interval columns generates core

When creating column desc the leading precision for the interval

datatype is not being set correctly. Comments from definition of struct

ComTdbVirtTableTableInfo in ComTdb.h inidcate that for interval type the

leading precision should be the precision.

Change-Id: I2aa7da5f5ba3bbef084b5180125c2fe99195a2ed

Merge "Bump version number"

Merge "Fix to handle install on MapR clusters."

Bump version number

Change-Id: Iaf0634775c1b0c4c6252696d395e663683008e41