resources

Clone Tools
  • last updated 16 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6540: Updated Hadoop and HBase libraries to the latest versions

Hadoop: 3.2.1

HBase: 2.2.2

closes #1895

  1. … 68 more files in changeset.
DRILL-7401: Upgrade to SqlLine 1.9.0

closes #1875

  1. … 2 more files in changeset.
DRILL-7207: Update the copyright year in NOTICE.txt file

  1. … 1 more file in changeset.
DRILL-6989: Upgrade to SqlLine 1.7

closes #1717

  1. … 3 more files in changeset.
DRILL-6985: Fix sqlline.bat issues on Windows and add drill-embedded.bat

closes #1616

    • -0
    • +24
    ./drill-embedded.bat
  1. … 1 more file in changeset.
DRILL-6895: Warnings for license headers checks and wrong license headers in some files

- update license-maven-plugin plugin custom mappings

- update files licenses headers

- update from "/usr/bin/env bash" to "/bin/bash" shebang line in shell script files

  1. … 9 more files in changeset.
DRILL-6877: NPE when starting Drillbit

closes #1560

  1. … 1 more file in changeset.
DRILL-3933: Surround $QUERY variable in double-quotes to avoid asterisk expansion in sqlline script

closes #1538

DRILL-6039: Fixed drillbit.sh script to do graceful shutdown

closes #1536

  1. … 1 more file in changeset.
DRILL-6752: Surround Drill quotes with double quotes

1. Surround Drill quotes with double quotes.

2. Remove drill-sqlline-test.conf, use drill-sqlline.conf for tests instead.

closes #1475

  1. … 4 more files in changeset.
DRILL-3853: Upgrade to SqlLine 1.5.0 closes #1462

    • -0
    • +31
    ./drill-sqlline-override-example.conf
  1. … 13 more files in changeset.
DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM

closes #1446

  1. … 22 more files in changeset.
DRILL-6662: Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin

closes #1419

  1. … 1 more file in changeset.
DRILL-5671: Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster

closes #1467

  1. … 14 more files in changeset.
DRILL-6494: Drill Plugins Handler

- Storage Plugins Handler service is used op the Drill start-up stage and it updates storage plugins configs from

storage-plugins-override.conf file. If plugins configs are present in the persistence store - they are updated,

otherwise bootstrap plugins are updated and the result configs are loaded to persistence store. If the enabled

status is absent in the storage-plugins-override.conf file, the last plugin config enabled status persists.

- 'drill.exec.storage.action_on_plugins_override_file' Boot option is added. This is the action, which should be

performed on the storage-plugins-override.conf file after successful updating storage plugins configs.

Possible values are: "none" (default), "rename" and "remove".

- The "NULL" issue with updating Hive plugin config by REST is solved. But clients are still being instantiated for disabled

plugins - DRILL-6412.

- "org.honton.chas.hocon:jackson-dataformat-hocon" library is added for the proper deserializing HOCON conf file

- additional refactoring: "com.typesafe:config" and "org.apache.commons:commons-lang3" are placed into DependencyManagement

block with proper versions; correct properties for metrics in "drill-override-example.conf" are specified

closes #1345

    • -0
    • +65
    ./storage-plugins-override-example.conf
  1. … 33 more files in changeset.
DRILL-6485: Fix typo in drill-env.sh

closes #1315

DRILL-5700: Adding nohup support for Sqlline script

closes #1300

DRILL-143: CGroup Support for Drill-on-YARN

Initial patch

1. Minor fix up

2. Updated variable name for pid file

Revert changes to yarn-drillbit.sh

Based on discussion for PR #1239 (Ref: https://github.com/apache/drill/pull/1239)

closes #1239

DRILL-6320: Fixed license headers.

closes #1207

  1. … 2052 more files in changeset.
DRILL-143: Support CGROUPs resource management

Introduces the DRILLBIT_CGROUP option in drill-env.sh.

The startup script checks if the specified CGroup (ver 2) is available and tries to apply it to the launched Drillbit JVM.

This would benefit not just Drill-on-YARN usecases, but any setup that would like CGroups for enforcement of (cpu) resources management.

(Also introduced SYS_CGROUP_DIR to account for possible non default locations of CGroup).

e.g when Drillbit is configured to use `drillcpu` cgroup

```

[root@maprlabs ~]# /opt/mapr/drill/apache-drill-1.14.0-SNAPSHOT/bin/drillbit.sh restart

Stopping drillbit

..

Starting drillbit, logging to /var/log/drill/drillbit.out

WARN: Drillbit's CPU resource usage will be managed under the CGroup : drillcpu (up to 4.00 cores allowed)

```

e.g. Non-existent CGroup `droolcpu` is used

```

[root@kk127 ~]# /opt/mapr/drill/apache-drill-1.14.0-SNAPSHOT/bin/drillbit.sh restart

Stopping drillbit

..

Starting drillbit, logging to /var/log/drill/drillbit.out

ERROR: cgroup droolcpu does not found. Ensure that daemon is running and cgroup exists

```

closes #1200

DRILL-6270: Added general option --jvm, which can be used to set the remote debug jvm options.

The option is not needed for drillbit startup, you can set the jvm properties directly after the start command.

Usage:

bin/drill-embedded --jvm "[JVM options]"

bin/sqlline -u "jdbc:drill:zk=zkaddr:port" --jvm "[JVM options]"

bin/drillbit.sh start [JVM options]

sqlline.bat -u "jdbc:drill:zk=local" --jvm "[JVM options]"

closes #1210

DRILL-6103: lsb_release: command not found

close apache/drill#1191

Thanks to Sanel Zukan for providing a small patch that checks for /etc/fedora-release path. This is more common, than lsb_release command on Linux distros.

DRILL-6256: Remove references to java 7 from readme and other files

close apache/drill#1172

  1. … 4 more files in changeset.
DRILL-6271: Updated copyright range in NOTICE

closes #1188

  1. … 1 more file in changeset.
DRILL-6213: During restart drillbit should be killed forcefully if exceeds allowed timeout

closes #1155

DRILL-1491: Support for JDK 8

Changed jdk version from 7 to 8 in pom.xml travis and drill-config.sh

This closes #1143

  1. … 4 more files in changeset.
DRILL-6189: Security: passwords logging and file permisions

1. Overrided serialization methods for instances with passwords

2. Changed file permissions for configuration files

closes #1139

  1. … 5 more files in changeset.
Added usage for graceful_stop in drillbit.sh

closes #1135

DRILL-6088: MainLoginPageModel errors out when http.auth.mechanisms is not configured

DRILL-6088: Update based on review feedback

close apache/drill#1092

  1. … 3 more files in changeset.
DRILL-5741: Automatically manage memory allocations during startup

**Note:** This commit is rebased on (hence, requires) DRILL-6068 commit

This commit provides a way for distributions and users to automatically define the Drillbit JVM's Heap, Direct and CodeCache allocations. To do this, it leverages the DRILL-6068 commit, which provides for configuration setup.

The only new variable introduced is an environment variable - `DRILLBIT_MAX_PROC_MEM` that defines a Drillbit's max memory parameters. The logic defined by `auto-setup.sh` will check that if any (or all) of the JVM memory parameters have been declared; they, cumulatively, don't exceed the value specified by `DRILLBIT_MAX_PROC_MEM`.

```

NOTE: To disable this, simply unset (or don't define) the environment variable DRILLBIT_MAX_PROC_MEM

```

The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified. For e.g.

```

DRILLBIT_MAX_PROC_MEM=13G

DRILLBIT_MAX_PROC_MEM=8192m

DRILLBIT_MAX_PROC_MEM=4194304K

```

In addition, you can specify it as a percent of the total system memory prior to the Drillbit starting up:

`DRILLBIT_MAX_PROC_MEM=40%`

For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message:

```

2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB

2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations

Total Memory Requested : 19 GB

Check the following settings to possibly modify (or increase the Max Memory Permitted):

DRILLBIT_MAX_PROC_MEM=25%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`,

For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning:

```

2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB

2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory

DRILLBIT_MAX_PROC_MEM=50%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required:

```

2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB)

2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory.

```

For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741

Changes to auto configure messaging

Publishing final values prior to startup

Minor update for printing to console's err stream

close apache/drill#1082