install_local_hadoop

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[TRAFODION-3281] Fix syntax error introduced in previous change

Earlier changes for this jira introduced syntax error in script

with spaces, where tabs are expected.

[TRAFODION-3281] Remove virtual nodes configuration

Changed workstation mode (aka install_local_hadoop aka ILH) to use

single-node configuration instead of virtual nodes.

* ILH script generates sqconfig for current node.

* template sqconfig file moved to config template dir.

* Updated expected files accordingly.

** Filed jira [TRAFODION-3283] for bug found.

Removed setting and use of SQ_VIRTUAL_NODES variable.

Removed virtual nodes support in sqgen.

  1. … 18 more files in changeset.
[TRAFODION-3272] Change installer settings for TRAF_LOG, TRAF_CONF, TRAF_VAR

Changes python-installer and ambari-installer.

Installers set TRAF_CONF to /etc/trafodion/conf

Installers allow user to specify TRAF_VAR, but defaults to /var/lib/trafodion

Installers allow user to specify TRAF_LOG, but defaults to /var/log/trafodion

Changes DCS conf directory from DCS_INSTALL_DIR/conf to TRAF_CONF/dcs

Changes REST conf directory from REST_INSTALL_DIR/conf to TRAF_CONF/rest

  1. … 53 more files in changeset.
[TRAFODION-2626] Change path of sqconf, ms.env, sqcert, udr and related files

This changeset covers the remaining instances of hard-coded paths based

on TRAF_HOME for dynamic files that should be in specific paths such as

TRAF_CONF or TRAF_VAR.

All files referenced via TRAF_HOME should be read-only software, not written

during run-time.

  1. … 59 more files in changeset.
Merge remote branch 'origin/pr/1721/head' into merge_1721

  1. … 11 more files in changeset.
Fixees to support stroring Udr libraries s Blobs.

  1. … 45 more files in changeset.
[TRAFODION-3038] fix HDP download link in install_local_hadoop script

review comment + fix to executor/test020 from a different checkin.

  1. … 2 more files in changeset.
TRAFODION-3086 Traf support for DDL operations on Hive objects

-- Support for TRAFODION-3086. Details in document attached to jira.

Other changes:

-- support for "if not exists", "if exists" clause for create/drop view

-- Support for: truncate T, truncate table T.

-- same as purgedata

-- showddl <tab>, detail

-- unregister hive schema <sch>

-- will unregister all objects in specified schema

-- Support for "if not registered", "if registered" clauss for

register/unregister command.

  1. … 90 more files in changeset.
[TRAFODION-3037] install_local_hadoop script out of date (APACHE)

[TRAFODION-2958] Consolidate the master and backup files in dcs conf folder to a single file (masters)

  1. … 22 more files in changeset.
[TRAFODION-2897] Invalid name in MySQL log file when using install_local_hadoop

Bug reported on the dev list. MySQL does not understand shell syntax.

[TRAFODION-2654] Change the location of trafodion-site.xml from $TRAF_HOME/etc to config [TRAFODION-2653] Sort operator loops at times

All HBase client connections from Trafodion uses Trafodion configuration

concept. Any client side property can now be set in trafodion-site.xml.

The scratch files are now open in blocking mode. Hence any scratch file

operation shouldn't return EAGAIN.

The internal table TDDL of table is now created without any table

coprocessors attached to it.

  1. … 23 more files in changeset.
Changes to move all Trafodion created hdfs files under /user/trafodion

  1. … 19 more files in changeset.
Various fixes, details below

-- max length limited to 16777216 bytes (16M)

for char cols and functions (repeat, concat).

(optimizer/SynthType.cpp, common/ComSmallDefs.h, sqlcomp/nadefaults.cpp)

-- previous max length change requires adding of a new hbase property

called hbase.client.keyvalue.maxsize so large key/values could be

handled in hbase cell.

Following scripts have been updated to handle that.

(sqf/sql/scripts/install_local_hadoop,

install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/

service_advisor.py

install/installer/traf_hortonworks_mods

install/python-installer/configs/mod_cfgs.json)

Developers can also update hbase-site.xml with this property if

they dont want to reinstall local hadoop.

<property>

<name>hbase.client.keyvalue.maxsize</name>

<value>0</value>

</property>

-- while accessing a hive table as an external table, the hive table and

corresponding external table definitions are validated to be the

same. This validates that corresponding columns have the same

data attributes (type, length, scale, etc).

This check causes failures if hive column is of 'string' datatype.

That is because hive 'string' column length can be changed by

a cqd but the corresponding external table has predefined length

set when the table is created.

The validation check now ignores the length attribute if the hive column is

of 'string' datatype.

(optimizer/BindRelExpr.cpp, common/CharType.*, NAType.*)

-- data moved into direct buffer would sometimes cause overflow and crash.

The max direct buffer length used to send/retrieve hbase data

is now limited to 1G (executor/ExHbaseAccess.cpp)

-- errors during vsbb upsert are now handled correctly

(executor/ExHbaseIUD.cpp)

-- support for GET CATALOGS command

(generator/GenRelExeUtil.cpp, executor/ExExeUtilGet.cpp)

-- An incorrect computation would sometimes cause group by rollup to

crash compiler in NAHeap::unlinkLargeFragment()

(generator/GenRelGrby.cpp)

-- sort of a large row that exceeded sort pre-set buffer size would crash

if the row size exceeded the max buffer size.

That has been fixed by allocating space for atleast one row.

(generator/GenRelMisc.cpp)

-- purgedata on a delimited name was failing. That has been fixed.

(optimizer/RelExeUtil.h)

-- regress/tools/runregr_privs1/privs2 fixed to handle running of

subset of tests

-- regress/seabase/TEST031 updated with new tests

  1. … 30 more files in changeset.
[TRAFODION-2426] Do not set hbase master log splitting property

This changes all install methods to no longer touch the log splitting

property.

  1. … 9 more files in changeset.
convert all MY_SQROOT to TRAF_HOME

  1. … 294 more files in changeset.
Merge [TRAFODION-2293] remove minor version for the hbase-trx-apache_*.jar

Update install_local_hadoop

Update install_local_hadoop

Merge [TRAFODION-2293] remove minor version for the hbase-trx-apache_*.jar

[TRAFODION-2306] Trafodion customization using its own configuration file

Missed out this change as part of merge of PR #777

[TRAFODION-2293] add restrict in install_local_hadoop

[TRAFODION-2306] Trafodion customization using its own configuration

  1. … 3 more files in changeset.
[TRAFODION-2293] remove minor version for the hbase-trx-apache_*.jar

  1. … 5 more files in changeset.
[TRAFODION-2306] Trafodion customization using its own configuration file

When sqgen is run, the customization resource file for Trafodion trafodion-site.xml

with pre-defined properties, can be found at $MY_SQROOT/etc.

You can add client side hbase properties with 'trafodion.' prefixed. These properties

become effective in the client Trafodion processes with 'trafodion.' stripped off.

When you modify this file, it needs to be copied to all nodes of Trafodion cluster

and Trafodion restarted.

This closes PR #777

  1. … 10 more files in changeset.
[TRAFODION-2306] Trafodion customization using its own configuration file

See JIRA for details

  1. … 8 more files in changeset.
[TRAFODION-2303] Limit the heap size of hbase processes for co-operative developer environment

In the developer environment, the HBase processes are started without setting

max heap size. Then, HBase java processes uses 1/4 of the physical memory as

the max heap size. This results in memory pressure in a cooperative developer

environment shared by many developers.

JIRA [TRAFODION-2236] TM crashes during startup

  1. … 2 more files in changeset.