ExpHbaseDefs.h

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Part 2 of ALTER TABLE/INDEX ALTER HBASE_OPTIONS support

blueprint alter-table-hbase-options

This set of changes includes non-transactional HBase semantics

for ALTER TABLE/INDEX ALTER HBASE_OPTIONS. As part of this

change, two sets of logic that manipulate HBase options in

the CREATE TABLE code path were refactored so that logic

could be reused by ALTER TABLE/INDEX. In both these cases,

the logic in question was taken out of a create-side method

and moved into a separate method.

Still to come is the equivalent transactional support, and

also a regression test script that exercises this feature.

With this change, ALTER TABLE/INDEX ALTER HBASE_OPTIONS works

but only in a non-transactional manner. That is, if the

operation is aborted after the request has been issued to HBase,

the change to the Trafodion metadata may be rolled back but

the change to the underlying HBase object may still complete.

Change-Id: Ied11d0b2ac7b7d65110d39a1e5a7fbbac1e1f827

  1. … 9 more files in changeset.
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
Part 1 of ALTER TABLE/INDEX ALTER HBASE_OPTIONS support

blueprint alter-table-hbase-options

This set of changes includes the following:

1. Syntax for ALTER TABLE/INDEX ALTER HBASE_OPTIONS

2. Compiler binding and generation logic

3. Run-time logic to update the metadata TEXT table with the

new options.

Still to come is:

1. Logic to actually change the HBase object

2. Transactional logic to actually change the HBase object

The functionality in this set of changes is only marginally

useful. If you manually change hbase options on a Trafodion

object using the hbase shell, you could use this ALTER

TABLE/INDEX command to update the Trafodion metadata. (Of

course some care would have to be taken to use the same

options!).

Change-Id: Id0a5513fe80853c06acdbbf6cc9fd50492fd07b2

  1. … 18 more files in changeset.
Adding more run-time memory allocations from NAHeap

This set of changes moves some of the string vector variables in HBase

access operators from standard string template to our NAList and

NAString (or HbaseStr for row IDs). In the process, allocationis of the

objects will be from our HAHeap instead of the system heap. This would

help us tracking memory usage and detecting leaks easier.

In addition, a change in ExHbaseAccessTcb::setupListOfColNames()

prevents unnecessary allocations to populate the columns list unless it

is empty. The Google profiling tools helped us on identifying this

problem.

also, removed ExHbaseAccessDeleteTcb operator which was not used.

Change-Id: I87ab674ab8e3d291f2fc9563718d88de537ae96b

  1. … 10 more files in changeset.
Reducing the path length for scan, single row get, batch get operations

Earlier, Key-values per row is buffered as a ByteBuffer and shipped to JNI side.There were 2 C->java transitions to get the Hbase row into tuple format.

Now, we avoid this copy on the Java side and ship the length and offsets of

the Key-Value for all rows in a batch along with reference to the

Key-Value buffer. The data is directly moved from Hbase Buffers to

row format buffer. There are only 2 C->java transitions per batch of row.

This has shown 3-4 times reduction in path length in Trafodion SQL processes

while scanning a table with 12 million rows.

Trafodion SQL processes dump core when ex_assert function is called or when it

exits with non-zero value. These processes will also dump core when an

environment variable ABORT_ON_ERROR is set in release mode.

Fix for seabase/TEST010 failure with the patchSet 1

Rowset Select was dumping core sometimes. A wrong tuple was used to calculate the row length. It caused the proces to dump core sometimes.

Change-Id: I0ec4669b54971b6a0c699c0fa0662c85f68bd25d

  1. … 13 more files in changeset.
Bug 1343615: Duplicated rows for parallel scan on salted table

- In preCodeGen, add partitioning key predicates to scan node

if it uses a single subset key and HASH2 partitioning function

- Handle partitioning key preds in FileScan::preCodeGen, move

code from HbaseAccess::preCodeGen.

- Make a special partitioning key predicate for salted tables

with a HASH2 function: "_SALT_" between :pivLo and :pivHi

This will lead to an efficient access path for the ESP to read

only the data it is supposed to read.

- Salted tables have a HASH2 partitioning key that does not

include the "_SALT_" column. So, the partitioning key is

not a prefix of the clustering key. However, we need to apply

the partitioning key predicates to the clustering key of the

table, since that's the only key we have. This is different

from a "partition access" node. See

TableHashPartitioningFunction::createSearchKey() in file

PartFunc.cpp.

- Moved a method to create a partitioning key predicate of

the form <some expr> between :pivLo and :pivHi up in the

class hierarchy, to be able to use it in both

HashPartitioningFunction and TableHashPartitioningFunction

Also added support for the KeyPrefixRegionSplitPolicy. This might

be useful in the future when we push things like GROUP BY into the

region servers. It can ensure that keys with the same prefix of a

given length stay in the same region. Example for a table with

this split policy:

-- make sure all the line items for an order (first 4 bytes of the key)

-- stay within the same region

create table lineitems(orderno int not null,

lineno int not null,

comment char(10),

primary key (orderno, lineno))

hbase_options ( SPLIT_POLICY = 'org.apache.hadoop.hbase.regionserver.KeyPrefixRegionSplitPolicy',

PREFIX_LENGTH_KEY = '4');

Removed ENCODE_ON_DISK table property because it is deprecated and does nothing.

Patch set 2: Changed comment in ExpHbaseDefs.h.

Change-Id: I57fafe2f854475261313abcf5bd2c81013f43756

  1. … 7 more files in changeset.
Table-specific split policy for salted tables:

Added a table-specific split policy when we are creating salted

tables. Also added a max file size for salted tables, if the max file

size split policy is used. Both split policy and max file size are

controlled via 2 new CQDs:

HBASE_SALTED_TABLE_MAX_FILE_SIZE (default: 0 - Use HBase default (10 GB))

HBASE_SALTED_TABLE_SET_SPLIT_POLICY (default: ON)

- ON means that salted tables will use ConstantSizeRegionSplitPolicy.

- OFF means that tables (salted or not) will use the default HBase split

policy, unless specified in the HBASE_OPTIONS clause.

Examples:

CREATE TABLE T ... SALT ...

- Uses ConstantSizeRegionSplitPolicy with the default max file size

(10 GB unless changed in hbase-site.xml)

CREATE TABLE T ... <no salt option>

- Uses the default HBase split policy

(IncreasingToUpperBoundRegionSplitPolicy, unless changed in

hbase-site.xml). I’ll work with Marvin to stop changing this

default in our installer.

CREATE TABLE T ... SALT ... HBASE_OPTIONS (SPLIT_POLICY =

'org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy');

- Uses IncreasingToUpperBoundRegionSplitPolicy, overriding the CQD

CQD HBASE_SALTED_TABLE_SET_SPLIT_POLICY OFF;

CREATE TABLE T ... SALT ...

- Uses the default HBase split policy

Change-Id: I4e221479e2c05c97d69b4865fefffba7e3e561fe

  1. … 6 more files in changeset.
Initial code drop of Trafodion

  1. … 4886 more files in changeset.