Kunal Khatua

DRILL-5735: UI options grouping and filtering & Metrics hints Allow search/sort in the Options webUI Additional details, like the description is provided as well in a JScript lookup map. This helps reduce the need for the server to constantly recreate the entire page with the description details, as the client browser can fill in these details. Developers will be expected to update the description as old/new options are introduced or deprecated. System table options_val also carries the description, albeit a 'shorter' version.

closes #1279

DRILL-6470: Remove defunct repository

When build the Drill source code, Parquet jars from the repository hosted at http://repo.dremio.com/release/ is inaccessible.

These artifacts (parquet libraries) are now available within http://maven.corp.maprtech.com/nexus/content/groups/public/org/apache/parquet/

closes #1326

DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.

1. Added protobuf files (incl generated C++ and Java)

2. Updated JdbcSubScan's getOperatorType API

closes #1297

DRILL-6477: Drillbit crashes with OOME (Heap) for a large WebUI query

For queries submitted through the WebUI and retrieving a large result-set, the Drillbit often hangs or crashes due to the (foreman) Drillbit running out of Heap memory.

This is because the Web client translates the result set into a massive object in the heap-space and tries to send that back to the browser. This results in the VM thread actively trying to perform GC if the memory is not sufficient.

The workaround is to have the active webConnection of the query periodically timeout to allow for checking the consumed heap-space. A level of 0.85 (i.e. 85%) is set as default threshold, crossing which, a query submitted through the REST API is marked and failed.

In addition, a user exception is thrown, indicting the cause of the query failing, along with alternative suggestions for re-executing the query.

closes #1309

DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.

This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes

Set the query ID string and display in WebUI

closes #1301

DRILL-6450: Visualized plans for profiles querying JDBC sources is broken

When viewing a profile for a query against a JDBC source, the visualized plan is not rendered. This is because the generated SQL pushed down to the JDBC source has a line break injected just before the FROM clause.

The workaround is to strip away any injected newlines ('\\n') at least for the SQL defined in the text plan, so that the backend Javascript can render it correctly.

In addition, any single line comments are also removed, but any block comments (i.e. /* .. */ ) are retained as they might carry hints.

This closes #1295

DRILL-6443: Enable Search for both running AND completed queries

When running a query in Drill, the `/profiles` page will show the search (and pagination) capabilities only for the top most visible table (i.e. Running Queries ).

The Completed Queries table will show the search feature only when there are no running queries. This is because the backend uses a generalized freemarker macro to define the seach capabilities for the tables being rendered. With running queries, both, running and completed queries tables have the same element ID, resulting in the search capability only being applied to the first table.

This modifies the Freemarker macro to take an additional argument for distinguishing between Running and Completed Queries table.

DRILL-5990: Drill should not startup with JRE sourced java executable

Currently, the check for the Java compiler is made only during query execution.

This commit ensures that if the java executable used for instantiating the Drillbit is from a JRE (i.e. the Java compiler is not available), the Drillbit will fail with the exception reported in the logs:

```

Exception in thread "main" org.apache.drill.exec.exception.DrillbitStartupException: JDK Java compiler not available. Ensure Drill is running with the java executable from a JDK and not a JRE

at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:138)

at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:409)

at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:389)

at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:385)

```

DRILL-6423: Export query result as a CSV file

Added option for user specified delimiter

Also, we show query Id on results page as alink for popping out a new window

closes #1266

DRILL-5305: Query Profile must display Query ID

Introduced change to the Protobuf to inject the text-equivalent of the QueryID into the profile. This way, the profile's file name can be changed, but restored back based on this new field.

The Profile UI also shows the Query ID, though this is not inferred from this new field, for sake of backward compatibility with older profiles.

closes #1265

DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart

As a follow up to DRILL-6289, the following improvements have been done:

1. When loading the page for the first time, the WebUI enables the shutdown button without actually checking the state of the Drillbits.

The ideal behaviour should be to disable the button till the state is verified. [Done]

If a Drillbit is confirmed down (i.e. not in `/state` response), it is marked as OFFLINE and button is disabled.

2. When shutting down the current Drillbit, the WebUI no more has access to the cluster state.

The ideal behaviour here should be to fetch the state from any of the other Drillbits to update the status. [Done]

With the current Drillbit down, the other bits are requested for cluster state info and update accordingly.

3. When a new, previously unseen Drillbit comes up, the WebUI will never render it because the table is statically generated during the first page load.

The idea behaviour should be to append to the table on discovery of a new node. [Done]

The new Drillbit info is injected and a prompt appears to refresh the page to re-populate any missing info. This also works with feature (2) mentioned above.

The only Java code change was to have the state response carry the address and http-port as a tuple, instead of the user-port (which is never used).

Updates based on review

1. Added descriptive comments to functions

2. Handled possible race condition from multiple httpRequests for cluster state

3. Eliminated unused stateKey variable

4. Best practice of using local (let) instead of global (var) objects, and substituting currentRow variable instead of the jQuery search.

Additional changes based on review

1. Random selection of Drillbits to query for state when primary Drillbit is down (limited to 3; with a timeout of 3 sec)

2. Indicate when an Offline Drillbit is de-registered, versus just Offline as per Zookeeper

3. Hide shutdown column when Authentication is enabled, but the user is NOT an Admin.

When the column is visible, remote bits are disabled because

4. Metrics will be shown all the time (except HTTPS), because the information is available and a non-admin user would anyway not have actionable capabilities

5. Basic clean up on unused variables

Hide shutdown buttons for HTTPS scenarios

Fixed check for 'https'

Since, `location.protocol` returns a trailing ':' as well

Handle metrics lookup on local Drillbit for HTTPS

If the webpage is accessed via IP, the certificate is tied to the IP and not the FQDN, which is what is used for fetching the metrics.

close apache/drill#1241

DRILL-143: CGroup Support for Drill-on-YARN

Initial patch

1. Minor fix up

2. Updated variable name for pid file

Revert changes to yarn-drillbit.sh

Based on discussion for PR #1239 (Ref: https://github.com/apache/drill/pull/1239)

closes #1239

    • -15
    • +15
    /distribution/src/resources/drillbit.sh