distrib-setup.sh

Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6540: Updated Hadoop and HBase libraries to the latest versions

Hadoop: 3.2.1

HBase: 2.2.2

closes #1895

  1. … 82 more files in changeset.
DRILL-6895: Warnings for license headers checks and wrong license headers in some files

- update license-maven-plugin plugin custom mappings

- update files licenses headers

- update from "/usr/bin/env bash" to "/bin/bash" shebang line in shell script files

  1. … 21 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2066 more files in changeset.
DRILL-5741: Automatically manage memory allocations during startup

**Note:** This commit is rebased on (hence, requires) DRILL-6068 commit

This commit provides a way for distributions and users to automatically define the Drillbit JVM's Heap, Direct and CodeCache allocations. To do this, it leverages the DRILL-6068 commit, which provides for configuration setup.

The only new variable introduced is an environment variable - `DRILLBIT_MAX_PROC_MEM` that defines a Drillbit's max memory parameters. The logic defined by `auto-setup.sh` will check that if any (or all) of the JVM memory parameters have been declared; they, cumulatively, don't exceed the value specified by `DRILLBIT_MAX_PROC_MEM`.

```

NOTE: To disable this, simply unset (or don't define) the environment variable DRILLBIT_MAX_PROC_MEM

```

The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified. For e.g.

```

DRILLBIT_MAX_PROC_MEM=13G

DRILLBIT_MAX_PROC_MEM=8192m

DRILLBIT_MAX_PROC_MEM=4194304K

```

In addition, you can specify it as a percent of the total system memory prior to the Drillbit starting up:

`DRILLBIT_MAX_PROC_MEM=40%`

For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message:

```

2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB

2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations

Total Memory Requested : 19 GB

Check the following settings to possibly modify (or increase the Max Memory Permitted):

DRILLBIT_MAX_PROC_MEM=25%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`,

For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning:

```

2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB

2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory

DRILLBIT_MAX_PROC_MEM=50%

DRILL_HEAP=8G

DRILL_MAX_DIRECT_MEMORY=10G

DRILLBIT_CODE_CACHE_SIZE=1024m

*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB

```

In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required:

```

2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB)

2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory.

```

For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741

Changes to auto configure messaging

Publishing final values prior to startup

Minor update for printing to console's err stream

close apache/drill#1082

  1. … 2 more files in changeset.
DRILL-6068: Support user/distrib-specific config checks during startup

1. Allows for distrib/user specific checks to be done

2. Place-holder files for distribution and user specific checks

3. Moved JVM Version Check to head of script

Separation of setups into 3 levels

1. Apache Drill (Default)

2. Distribition Specific

3. User-Defined

1 & 2 are mutually exclusive.

3 is additional checks that the user can specify.

  1. … 4 more files in changeset.