Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
White space adjustment

Adding apache license to generated protobuf files

Fix for [Trafodion-3244] Need to provide support for HDP 2.5.x and 2.6.x Various files needed to correctly build modified protos and remain backward compatible

  1. … 75 more files in changeset.
TRAFODION-3165 HDFS storage option not work well if table use namespace

refactor the retry logic

refactor the code for review comments

add retry logic

add license header

use coprocessor to set storage policy

  1. … 3 more files in changeset.
[TRAFODION-2733] Provide an improved memory quota assignment for big memory operators (BMO)

Added SplitBanceHelper.java. This file was deleted inadvertently

Following changes are done in BMO memory quota

1) Enabled memory quota per node. The CQD BMO_MEMORY_LIMIT_PER_NODE

(renamed from EXE_MEMORY_LIMIT_PER_CPU) is set to 10240 MB by default.

Old attribute Old value Renamed Attribute New value

EXE_MEMORY_LIMIT_PER_CPU 0 BMO_MEMORY_LIMIT_PER_CPU 10240

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHGROUPBY 25

EXE_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 10 BMO_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN 25

EXE_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 10 BMO_MEMORY_LIMIT_LOWER_BOUND_PROBE_CACHE 25

EXE_MEMORY_LIMIT_LOWER_BOUND_SORT 10 BMO_MEMORY_LIMIT_LOWER_BOUND_SORT 200

2) Changes in EXPLAIN

Estimated memory per node for all BMOs at ROOT operator

Estimated memory per instance for every BMO operator

Memory quota per instance for every BMO operator

3) BMO TDB contains the memory quota per esp instance now.

4) Root TDB now contains the limit per node and estimated memory per node.

This can be used by WMS to change the memory allocation during

runtime without compilation. - Not yet implemented.

4) Added a CQD BMO_MEMORY_LIMIT_UPPER_BOUND to gap the memory

consumed by BMO by the same queries with less number of

bmos.

5) The unused memory quota is yielded to other fragments in the process

also.

6) Removed the code to limit the ESPs from being assigned to a fragement

based on the BMO memory quota.

7) Added a new CQD BMO_MEMORY_ESTIMATE_RATIO_CAP to gap the memory

estimate skew by any one BMO operator to 0.7.

8) To disable the memory quota per node, set BMO_MEMORY_LIMIT_PER_NODE to 0.

9) This memory quota is distributed proportionally based on the estimated memory

taking into consideration the number of bmo instances per operator and

the number of nodes available in the cluster to host these instances.

Hence, this memory quota should be valid in multi-fragments independent of the

number of fragments in an ESP.

10) Removed the CQD EXE_MEMORY_LIMIT_NONBMOS_PERCENT and EXE_MEMORY_RESERVED_FOR_MXOSRVR_IN_MB

11) Fixed BMO stats WM to be at least the allocated memory.

12) Changed the sort operator to account the bmo memory correctly.

(cherry picked from commit ba19c04a58890fdd845b03f8d915abdd487b6407)

Conflicts:

core/sqf/src/seatrans/hbase-trx/src/main/java/org/apache/hadoop/hbase/coprocessor/transactional/SplitBalanceHelper.java

core/sql/cli/Context.cpp

core/sql/executor/ex_frag_rt.cpp

core/sql/executor/ex_sort.cpp

core/sql/regress/executor/EXPECTED131

core/sql/regress/executor/EXPECTED140

core/sql/regress/hive/EXPECTED009

core/sql/regress/hive/EXPECTED030

core/sql/regress/hive/FILTER009

core/sql/regress/seabase/EXPECTED010

core/sql/regress/seabase/EXPECTED011

core/sql/regress/seabase/EXPECTED016

core/sql/sqlcomp/DefaultConstants.h

  1. … 44 more files in changeset.
Reworks, addressing Sean's and Selva's comments

  1. … 1 more file in changeset.
Put back Apache license in generated file

[TRAFODION-2617] Optionally push estimate row count logic into HBase coproc

  1. … 9 more files in changeset.
[TRAFODION-2654] Change the location of trafodion-site.xml from $TRAF_HOME/etc to config [TRAFODION-2653] Sort operator loops at times

All HBase client connections from Trafodion uses Trafodion configuration

concept. Any client side property can now be set in trafodion-site.xml.

The scratch files are now open in blocking mode. Hence any scratch file

operation shouldn't return EAGAIN.

The internal table TDDL of table is now created without any table

coprocessors attached to it.

  1. … 22 more files in changeset.
Merge remote branch 'origin/master' into TRAFODION-2001

Conflicts:

core/sqf/src/seatrans/hbase-trx/src/main/java/org/apache/hadoop/hbase/regionserver/transactional/TrxTransactionState.java.tmpl

Fix for [Trafodion-2614] HBaseTxClient throws TableNotFoundException for TRAFODION._DTM_.TDDL

[Trafodion-2608] Skips file flush in case of insert drop same transaction.

Manual merge fixes

  1. … 1 more file in changeset.
Restored nodeAdd scripts to support new node add logic. Fixed 'shell node add' logic in Trafodion Configuration library module.

  1. … 7 more files in changeset.
Added missed template file

Merge fixes

Added copyright to the generated protobufs

This is a large contribution of changes from Esgyn TransactionManager and libraries that are collectively much better tested and hardened than Trafodion, but are too numerous and complex to cherry pick individually.

The changes include:

Dozens of bug fixes

Improved recovery after HBase failures

Better error reporting and tracing, which includes region keys for easier debugging

Reduced UnknownTransactionExceptions resulting from duplicate transaction registration

Better TLOG management including reducing the default number of tables

  1. … 7 more files in changeset.
[TRAFODION-2448] Change TM SplitHelper to get configured parent znode instead of hard-coded /hbase

[TRAFODION-2468] DDL HA , step 2 changes(please refer jira).

  1. … 1 more file in changeset.
[TRAFODION-2466]Handle sortReclen limit in sort overflow.

  1. … 3 more files in changeset.
[TRAFODION-2150] and [TRAFODION-2151] TM long prepare and chore thread issues.

  1. … 1 more file in changeset.
additional changes to address review feedback.

  1. … 3 more files in changeset.
[TRAFODION-2414] enhance 8616 commit conflict error message to include tx conflict details.

  1. … 27 more files in changeset.