drill

Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
edits

DRILL-6119: The OpenTSDB storage plugin is not included in the Drill distribution

closes #1102

minor edits

update docs for NaN infinity type support and bounds checking behavior change

DRILL-6089: Removed ordering trait from HashJoin in planner and verified the planner does not assume HashJoin preserves ordering.

closes #1117

DRILL-6106: Use valueOf method instead of constructor since valueOf has a higher performance by caching frequently requested values.

closes #1099

minor edit

    • -2
    • +2
    /_docs/developer-information/rest-api/010-submitting-queries-from-the-rest-api-when-impersonation-is-enabled-and-authentication-is-disabled.md
minor edit

    • -2
    • +2
    /_docs/developer-information/rest-api/010-submitting-queries-from-the-rest-api-when-impersonation-is-enabled-and-authentication-is-disabled.md
Doc updates per DRILL-5726

    • -2
    • +2
    /_docs/developer-information/rest-api/010-submitting-queries-from-the-rest-api-when-impersonation-is-enabled-and-authentication-is-disabled.md
DRILL-6102: Fix ConcurrentModificationException in the BaseAllocator's print method

closes #1100

DRILL-5868: Support SQL syntax highlighting of queries

Based on the commit for DRILL-5981 (PR #1043), this commit further leverages the Ace JavaScript library with customizations specific to Drill.

This commit introduces the following to the Query Editor (including the Edit Query tab within an existing profile to rerunning the query).

1. Syntax highlighting (This is supported for submitted query profiles

2. Autocomplete supported in editors

3. Specifying Drill specific keywords and functions in visible autocomplete

4. Key snippets (template SQLs) allowing for rapid writing of syntax:

i. Query System Tables

ii. CView, CTAS and CTempTAS

iii. Alter Session

iv. Explain and Select * queries

NOTE: The lists for #3 and #4 are not exhaustive. As more features are added to Drill, these lists can be expanded.

Updates based on review comments

1. Disabled warning message

2. Extended reserved keyword list

3. Fixed bugs

List:

Update 1: Bug fix for when Impersonation is enabled

Update 2: Remove the duplicate editor that might get injected

Update 3: Removed trailing whitespaces in Javascripts

close apache/drill#1084

Doc updates per DRILL-5726

    • -0
    • +67
    /_docs/developer-information/rest-api/010-submitting-queries-from-the-rest-api-when-impersonation-is-enabled-and-authentication-is-disabled.md
    • -2
    • +4
    /_docs/install/050-starting-drill-in-distributed-mode.md
    • -4
    • +6
    /_docs/install/060-starting-the-web-ui.md
DRILL-6100: Intermittent failure while reading Parquet file footer during planning phase

close apache/drill#1097

minor edit

edit JDBC SQuirreL on Windows doc per MD-1728

DRILL-6099: Push limit past flatten(project) without pushdown into scan

closes #1096

DRILL-3993: Fix typo in DrillRelBuilder Javadoc

DRILL-6090: While connecting to drill-bits using JDBC Driver through Zookeeper, a lot of "Curator-Framework-0" threads are created if connection to drill-bit is not successful(no drill-bits are up/reachable)

I am using Drill JDBC driver 1.12.0 to connect to MapR-DB. I am finding the available drill-bits using Zookeepers. When drill-bits are not up or not reachable, the connection is failed with exception: "Failure in connecting to Drill: oadd.org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client", which is expected, but number of threads created by ZKClusterCoordinator just keeps on increasing.

Steps to reproduce the issue

Setup a connection with a drill-bit using Apache Drill JDBC driver 1.12.0 through Zookeeper hosts(port 5181)

Now stop the drill-bit services or block the drill-bit IPs using iptable rules

Truncate catalina logs

Try to connect to the drill-bit/hit a code path that requires connection to drill-bits.

Take thread dump using kill -QUIT <java process id>

grep -c "Curator-Framework-0" catalina.out

Observe that the curator framework thread just keep on accumulating

RCA:

ZKClusterCoordinator creates curator threads in the constructor

ZKClusterCoordinator is instantiated by DrillClient.connect

DrillClient.connect is called in DrillConnectionImpl constructor

Fix:

Call DrillConnectionImpl .cleanup() from all the catch blocks in the DrillConnectionImpl constructor.

close apache/drill#1094

DRILL-6093 : Account for simple columns in project cpu costing

close apache/drill#1093

DRILL-6085: Fixed spontaneous vm exits on Travis.

close apache/drill#1095

DRILL-5993: Fixed copyEntry for var length nullable vectors.

DRILL-3993: Changes after review:

Moved checks for OOM into spillIfNeeded method to avoid excessive call of chooseAPartitionToFlush method.

Ignored unit tests until DRILL-6018 is fixed.

These failures appears because new Calcite tries to simplify expressions from the query when applying ReduceExpressionsRule and fails with NFE.

Similar problem, but with old version of Calcite was described in DRILL-6018, therefore these unit tests are marked as ignored until DRILL-6018 is fixed.

DRILL-6088: MainLoginPageModel errors out when http.auth.mechanisms is not configured

DRILL-6088: Update based on review feedback

close apache/drill#1092

DRILL-5741: Automatically manage memory allocations during startup

**Note:** This commit is rebased on (hence, requires) DRILL-6068 commit

This commit provides a way for distributions and users to automatically define the Drillbit JVM's Heap, Direct and CodeCache allocations. To do this, it leverages the DRILL-6068 commit, which provides for configuration setup.

The only new variable introduced is an environment variable - `DRILLBIT_MAX_PROC_MEM` that defines a Drillbit's max memory parameters. The logic defined by `auto-setup.sh` will check that if any (or all) of the JVM memory parameters have been declared; they, cumulatively, don't exceed the value specified by `DRILLBIT_MAX_PROC_MEM`.

```

NOTE: To disable this, simply unset (or don't define) the environment variable DRILLBIT_MAX_PROC_MEM

```

The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified. For e.g.

```

DRILLBIT_MAX_PROC_MEM=13G

DRILLBIT_MAX_PROC_MEM=8192m

DRILLBIT_MAX_PROC_MEM=4194304K

```

In addition, you can specify it as a percent of the total system memory prior to the Drillbit starting up:

`DRILLBIT_MAX_PROC_MEM=40%`

For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message:

```

2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB

2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations

Total Memory Requested : 19 GB

Check the following settings to possibly modify (or increase the Max Memory Permitted):

DRILLBIT_MAX_PROC_MEM=25%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`,

For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning:

```

2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB

2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory

DRILLBIT_MAX_PROC_MEM=50%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required:

```

2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB)

2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory.

```

For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741

Changes to auto configure messaging

Publishing final values prior to startup

Minor update for printing to console's err stream

close apache/drill#1082

    • -1
    • +201
    /distribution/src/resources/auto-setup.sh
    • -0
    • +10
    /distribution/src/resources/drill-env.sh
DRILL-6068: Support user/distrib-specific config checks during startup

1. Allows for distrib/user specific checks to be done

2. Place-holder files for distribution and user specific checks

3. Moved JVM Version Check to head of script

Separation of setups into 3 levels

1. Apache Drill (Default)

2. Distribition Specific

3. User-Defined

1 & 2 are mutually exclusive.

3 is additional checks that the user can specify.

    • -0
    • +29
    /distribution/src/resources/auto-setup.sh
    • -0
    • +29
    /distribution/src/resources/distrib-setup.sh
    • -0
    • +26
    /distribution/src/resources/drill-setup.sh
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

  1. … 209 more files in changeset.
DRILL-6081: Set query end time before writing final profile

close apache/drill#1088

DRILL-6025: Display execution time of a query in RUNNING state

This closes #1074

DRILL-6190: Fix handling of packets longer than legally allowed

closes #1133

DRILL-6080: Sort incorrectly limits batch size to 65535 records

closes #1090

* Sort incorrectly limits batch size to 65535 records rather than 65536.

* This PR also includes a few code cleanup items.

* Fix for overflow in offset vector in row set writer

* Performance tool update

* Replace "unsafe" methods with "set" methods

* Also fixes an indexing issue with nullable writers

* Removed debug & timing code

* Increase strictness for batch size