drill

Clone Tools
  • last updated 27 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6090: While connecting to drill-bits using JDBC Driver through Zookeeper, a lot of "Curator-Framework-0" threads are created if connection to drill-bit is not successful(no drill-bits are up/reachable)

I am using Drill JDBC driver 1.12.0 to connect to MapR-DB. I am finding the available drill-bits using Zookeepers. When drill-bits are not up or not reachable, the connection is failed with exception: "Failure in connecting to Drill: oadd.org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client", which is expected, but number of threads created by ZKClusterCoordinator just keeps on increasing.

Steps to reproduce the issue

Setup a connection with a drill-bit using Apache Drill JDBC driver 1.12.0 through Zookeeper hosts(port 5181)

Now stop the drill-bit services or block the drill-bit IPs using iptable rules

Truncate catalina logs

Try to connect to the drill-bit/hit a code path that requires connection to drill-bits.

Take thread dump using kill -QUIT <java process id>

grep -c "Curator-Framework-0" catalina.out

Observe that the curator framework thread just keep on accumulating

RCA:

ZKClusterCoordinator creates curator threads in the constructor

ZKClusterCoordinator is instantiated by DrillClient.connect

DrillClient.connect is called in DrillConnectionImpl constructor

Fix:

Call DrillConnectionImpl .cleanup() from all the catch blocks in the DrillConnectionImpl constructor.

close apache/drill#1094

DRILL-6093 : Account for simple columns in project cpu costing

close apache/drill#1093

DRILL-6085: Fixed spontaneous vm exits on Travis.

close apache/drill#1095

DRILL-5993: Fixed copyEntry for var length nullable vectors.

DRILL-3993: Changes after review:

Moved checks for OOM into spillIfNeeded method to avoid excessive call of chooseAPartitionToFlush method.

Ignored unit tests until DRILL-6018 is fixed.

These failures appears because new Calcite tries to simplify expressions from the query when applying ReduceExpressionsRule and fails with NFE.

Similar problem, but with old version of Calcite was described in DRILL-6018, therefore these unit tests are marked as ignored until DRILL-6018 is fixed.

DRILL-6088: MainLoginPageModel errors out when http.auth.mechanisms is not configured

DRILL-6088: Update based on review feedback

close apache/drill#1092

DRILL-5741: Automatically manage memory allocations during startup

**Note:** This commit is rebased on (hence, requires) DRILL-6068 commit

This commit provides a way for distributions and users to automatically define the Drillbit JVM's Heap, Direct and CodeCache allocations. To do this, it leverages the DRILL-6068 commit, which provides for configuration setup.

The only new variable introduced is an environment variable - `DRILLBIT_MAX_PROC_MEM` that defines a Drillbit's max memory parameters. The logic defined by `auto-setup.sh` will check that if any (or all) of the JVM memory parameters have been declared; they, cumulatively, don't exceed the value specified by `DRILLBIT_MAX_PROC_MEM`.

```

NOTE: To disable this, simply unset (or don't define) the environment variable DRILLBIT_MAX_PROC_MEM

```

The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified. For e.g.

```

DRILLBIT_MAX_PROC_MEM=13G

DRILLBIT_MAX_PROC_MEM=8192m

DRILLBIT_MAX_PROC_MEM=4194304K

```

In addition, you can specify it as a percent of the total system memory prior to the Drillbit starting up:

`DRILLBIT_MAX_PROC_MEM=40%`

For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message:

```

2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB

2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations

Total Memory Requested : 19 GB

Check the following settings to possibly modify (or increase the Max Memory Permitted):

DRILLBIT_MAX_PROC_MEM=25%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`,

For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning:

```

2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB

2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory

DRILLBIT_MAX_PROC_MEM=50%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required:

```

2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB)

2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory.

```

For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741

Changes to auto configure messaging

Publishing final values prior to startup

Minor update for printing to console's err stream

close apache/drill#1082

    • -1
    • +201
    /distribution/src/resources/auto-setup.sh
    • -0
    • +10
    /distribution/src/resources/drill-env.sh
DRILL-6068: Support user/distrib-specific config checks during startup

1. Allows for distrib/user specific checks to be done

2. Place-holder files for distribution and user specific checks

3. Moved JVM Version Check to head of script

Separation of setups into 3 levels

1. Apache Drill (Default)

2. Distribition Specific

3. User-Defined

1 & 2 are mutually exclusive.

3 is additional checks that the user can specify.

    • -0
    • +29
    /distribution/src/resources/auto-setup.sh
    • -0
    • +29
    /distribution/src/resources/distrib-setup.sh
    • -0
    • +26
    /distribution/src/resources/drill-setup.sh
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

  1. … 209 more files in changeset.
DRILL-6081: Set query end time before writing final profile

close apache/drill#1088

DRILL-6025: Display execution time of a query in RUNNING state

This closes #1074

DRILL-6190: Fix handling of packets longer than legally allowed

closes #1133

DRILL-6080: Sort incorrectly limits batch size to 65535 records

closes #1090

* Sort incorrectly limits batch size to 65535 records rather than 65536.

* This PR also includes a few code cleanup items.

* Fix for overflow in offset vector in row set writer

* Performance tool update

* Replace "unsafe" methods with "set" methods

* Also fixes an indexing issue with nullable writers

* Removed debug & timing code

* Increase strictness for batch size

DRILL-6071: Limit batch size for flatten operator

closes #1091

DRILL-3993: Resolve conflicts

DRILL-6003: Fixed sporadic test failures caused by the creation of duplicate clusters in TestDynamicUDFSupport.

This closes #1023

DRILL-3993: Add unit tests for DRILL-4469 & DRILL-5768

DRILL-3993: Fix aggregate exchange rules for the cases when aggregate rel node contains several calls

DRILL-6833: Support for pushdown of rowkey based joins

closes #1532

DRILL-6191: Add acknowledgement sequence number and flags fields, details for flags

closes #1134

    • binary
    /exec/java-exec/src/test/resources/store/pcap/synscan.pcap
DRILL-6063: Set correct ThreadContext ClassLoader before using Hadoop Configuration class in DrillClient

This closes #1079

DRILL-3993: Add check for OOM in HashAgg

DRILL-5068: Create sys.profiles table

(Thanks to Hongze Zhang for the initial work)

Introduced a non-distributed (i.e. the foreman Drillbit executes a single fragment query) system table to list the completed profiles that are visible to it in the persistent store. This allows for querying the profiles on the following common metrics:

queryId

startTime

foreman

fragments

user

queue

planTime

queueTime

executeTime

totalTime

state

query

An additional system table - `sys.profiles_json` has also been introduced. This only has the following fields:

queryId

json

This allows for more detail metrics to be queried on the profile as a JSON document. Users are expected to use `sys.profiles` to identify profiles on the available metrics (such as long executeTime or fragments), and use the queryId to find the corresponding JSON document representation for further drill down into the profiles.

Updated TestMetadataProvider with new tally for current table and column counting tests

Security check to prevent unauthorized access to profile listing

Updated based on review comments

This closes #1077

DRILL-6054: don't try to split the filter when it is not AND

This closes #1078

DRILL-6079 : Fixed memory leak in Parquet Reader

close apache/drill#1087

DRILL-6118: Handle item star columns during project / filter push down and directory pruning

1. Added DrillFilterItemStarReWriterRule to re-write item star fields to regular field references.

2. Refactored DrillPushProjectIntoScanRule to handle item star fields, factored out helper classes and methods from PreUitl.class.

3. Fixed issue with dynamic star usage (after Calcite upgrade old usage of star was still present, replaced WILDCARD -> DYNAMIC_STAR for clarity).

4. Added unit tests to check project / filter push down and directory pruning with item star.

  1. … 12 more files in changeset.
DRILL-3993: Changes after code review.

DRILL-6049: Misc. hygiene and code cleanup changes

close apache/drill#1085

  1. … 109 more files in changeset.