HDFSHook.cpp

Clone Tools
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move core into subdir to combine repos

  1. … 10768 more files in changeset.
Move core into subdir to combine repos

  1. … 10622 more files in changeset.
Move core into subdir to combine repos

Use: git log --follow -- <file>

to view file history thru renames.

  1. … 10837 more files in changeset.
fix lp 1392099 (cqd hive_max_varchar_length)

There are two contribuing factors.

1. A fully specified partition requirement (partition of 1) in

FileScan::synthHiveScanPhysicalProperty() is not recognized

and a modification of the ESPs renders the partition requirement

not satisfiable.

2. The reduction in CQD value for hive_max_varchar_length reduces

the dop from 5 to 1. In Exchange::createContextForChild(), a dop of

1 renders the parallel version of plan infeasible (see line near 2004

on test "rppForMe->getCountOfPipelines() > 1").

The fix is to address issue #1.

Added a safety check for ApproximatelyNPartitions and improvement

for hive sampling logic.

remove update to cg/TEST011

Change-Id: Idbf24be0b00b336dbac93afdddb6896ed0937e23

  1. … 1 more file in changeset.
Initial changes for ORC file support.

Access to ORC (optimized row columnar) format tables is not enabled by

default yet. This checkin is initial and infrastructure changes for

that support.

Change-Id: I683c1b63c502dd4d2c736181952cb40f9f299cfd

  1. … 53 more files in changeset.
Fix for bug 1336106: HDFS files without permission causes core dump

Missing HDFS permissions caused Hive queries in Trafodion to crash

instead of giving an error message. Added a workaround to the bug

in libhdfs and also added more error handling to the HDFS statistics

code.

7/17: Rework from comments for patch set 1.

7/20: Rework from comments for patch set 2.

Change-Id: Ibb8cfe4ce0aca389598bc6724dbd0702961b5a30

  1. … 3 more files in changeset.
Fix for query cache issue for Hive selects.

Two changes

1)Fix for bug #1293816.

2)Discontinue linking in libprotobuf.so, since it is currently unused.

For the hive query cache bug, the issue was that any change to a HDFS file

in a Hive directory, or to the directory itself (add/drop a file), was not

reflected in the query cache key. So the compiler could give a plan with an

incorrect list of HDFS files to the HDFSScan operator. The fix is to add

max(fileInfo.mLastMod) to the query cache key. The max is taken over all

files for a given Hive table. The number of files for a given Hive table has

also been added to the query cache key to cover cases where a file is deleted

from a Hive directory. Both query cache and query text cache are addressed.

The mLastMod time for each file and the number of files are determined through

the libHdfs call hdfsListDirectory(), which we already make.

Linking in libprotobuf.so is causing issues on certain MapR clusters since

MapR also uses this library and sometimes the version used by MapR is

different from what Trafodion uses. Since this library is not being used by

Trafodion stack right now, we will no longer link in this library in SQL or

connectivity layers. When a fix is found for the version incompatibility

issue, this change will be reversed.

Patche Set 2.

Thank you Dave for catching these issues. They have been resolved in

Patch Set 2.

Change-Id: Idbe599a876fdcaf77d2bdb9fdbf4b77a3f431e46

  1. … 10 more files in changeset.
Fix for a few Hive issues

a) Improved Hive file drectory parsing to include a MapR format with no hostName or portNum. The logic to parse the directory name was previously repeated in three places in code, now we have one common place to encapsulate this parsing logic. Most files have been touched due to this refactoring

b) Fix for Launchpad bug #1320344. Hive Delete will now raise an error

c) Fix for Launchpad bug #1320385. Hive Update will now raise an error

d) Hive Merge will also raise an error. A query cache problem described in #1320344 and #1320385 has not been resolved in this delivery.

Patch Set 2:

This patch set addresses the two issues that Hans found. Thank you.

Patch Set 3:

This patch set addresses the two issues that Dave found. Thank you.

Change-Id: I19764227663dbb5c2d410608864a000592ce964a

  1. … 18 more files in changeset.
Initial code drop of Trafodion

    • -0
    • +1008
    ./HDFSHook.cpp
  1. … 4886 more files in changeset.