Clone
 

kunal khatua <kkhatua@maprtech.com> in drill

DRILL-7222: Visualize estimated and actual row counts for a query

With statistics in place, it is useful to have the estimated rowcount along side the actual rowcount query profile's operator overview. A toggle button allows this with the estimated rows hidden by default

We can extract this from the Physical Plan section of the profile.

Added a toggle-ready table-column header

closes #1779

DRILL-7338: REST API calls to Drill fail due to insufficient heap memory

This PR allows for the 85% threshold to be customizable with a value of 0 meant for disabling.

closes #1837

DRILL-7201: Strange symbols in error window (Windows)

Looks like some unicode characters creeped in, causing the rendering to be messed up ONLY for Windows OS. This PR patches that issue and ensures that the Alert symbol also appears correctly.

DRILL-7202: Failed query shows warning that fragments made no progress

This bug arises because of the `toggleWarning()` javascript function that shows the warning if the number of slow major fragments matches the number of major fragments.

For failed queries, the number of major fragments = 0; and that matches the number of tags (which is also zero for a non-running query). Hence the warning shows.

DRILL-7160: e.q.max_rows QUERY-level option shown even if not set

The fix is to force setting to zero IFF autoLimit was intended to be set originally but is inapplicable; such as 'SHOW DATABASES'. If autolimit was not intended to be applied, setting the value to zero is not required.

WebUI is also patched to not show the zero value set in such scenarios.

Additional Logos for Powered-By Page

    • binary
    /images/logos/IORA_NUS.png
    • binary
    /images/logos/option3-io-logo.png
DRILL-7048: Implement JDBC Statement.setMaxRows() with System Option

This introduces support for JDBC's Statement.setMaxRows(int) API, which can help Drill execute a query much faster if it knows that not ALL the records in the resultset will be consumed upfront.

This Commit introduces the core changes to support the feature within Drill's execution engine

Protobuf Changes

1. RunQuery: Added "autolimit_rowcount"

2. QueryProfile: Added "autoLimit"

3. Regenerated Java and C++ client files

REST API support

1. Support for REST server to interpret a submitted query and also for rendering this information for an executed query

2. Updates to the Freemarker templates (for WebUI)

3. Safety check within Javascript (for WebUI)

JDBC API support

1. Introduces backend execution of 'ALTER SESSION' to apply the auto-limiting of resultset size

2. Added Unit Tests for PreparedStatement and Statement objects

3. Added getter setter methods to be skipped in testing for org.apache.drill.jdbc.test.Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight()

Updates based on review comments

Additional Updates

Test Cleanup

1. Revert Drill2489 hack

2. Formatting in *StatementTest

3. Removal f redundant `statement.close()`

4. Manage new Exception thrown when setting invalid maxRow values

Final updates

1. Test changes

2. Trim trailing spaces in auto-limit value (Javascript)

3. Before & After annotations to synchronize changes to system values for MaxRows(auto-limit)

Reorganized tests due to synchronized locking

Removed conflicting JsonCreator in QueryWrapper

Additional test cleanup

closes #1714

    • -6
    • +40
    /contrib/native/client/src/protobuf/User.pb.h
  1. … 20 more files in changeset.
DRILL-7061: Disable LIMIT Rows Option

Freemarker by default introduces a comma in numeric values greater than 999. This corrects that by removing the ',' in the default limit size.

However, since a Server-side implementation is in progress (DRILL-6960 and DRILL-7048), it is best to disable this for now. The latest commit in this will hide those capabilities in the WebUI until the server-side feature goes in.

X

closes #1689

DRILL-7056: Drill fails with NPE when starting in distributed mode & 31010 port is used closes #1656

DRILL-7036: Improve UI for alert and error messages closes #1644 This PR standardizes error and alert messages to a cleaner interface by leveraging Bootstraps UX elements for publishing the messages in a presentable format. Exceptions reported back to the browser and rendered in a neat tabular format (using Panels) All errors can be redirected to errorMessage.ftl which will render it in a neat format. Alerts are replaced with modals. Interactions (pages) affected by Alert modals 1. Missing Query submission 2. profile Query Rerun 3. invalid Profile Listing Fetch 4. invalid Option Value for update 5. Missing username/password submission

The errorMessage.ftl has been moved to root dir, and unused `error.ftl` was removed

    • -0
    • +104
    /exec/java-exec/src/main/resources/rest/alertModals.ftl
    • -0
    • +37
    /exec/java-exec/src/main/resources/rest/errorMessage.ftl
DRILL-6971: Labelled Query State with color coding

closes #1611

DRILL-6050: Provide a limit to number of rows fetched for a query in UI

Currently, the WebServer side needs to process the entire set of results and stream it back to the WebClient.

Since the WebUI does paginate results, we can load a larger set for pagination on the browser client and relieve pressure off the WebServer to host all the data (most of which will never be streamed to the browser).

e.g. Fetching all rows from a 1Billion records table is impractical and can be capped at (say) 1K. Currently, the user has to explicitly specify LIMIT in the submitted query.

An option is provided in the field to allow for this entry, and can be set to selected by default for the Web UI.

The submitted query indicates that an auto-limiting wrapper was applied.

[Update #1] Updated as per comments

1. Limit Wrapping Unchecked by default

2. Full List configuration of results

[Update #2] Minor update

[Update #3] Followup

closes #1593

DRILL-6941: Incorrect EARLY_LIMIT0_OPT_KEY description

Description was accidentally duplicated from `planner.parser.quoting_identifiers`

closes #1597

DRILL-6942: Provide ability to sort list of profiles on Drill Web UI

This provides an option to order the list of query profiles based on any of the displayed fields, including total duration. This way, a user can easily identify long running queries.

In addition, the number of profiles listed per page for both, completed and running list of queries, has been made configurable with the parameter: `drill.exec.http.profiles_per_page` (default is 10,25,50,100)

closes #1594

DRILL-6939: Indicate when a query is submitted and is in progress

On query submission, a modal popup blocks further interaction until the result of the query are available.

Since the Query ID is not available at this point, the only way to cancel a running query is to navigate to it via the `/profiles` tab. For this, the modal allows a button to pop out to this tab without closing the current window which is waiting for the result-set.

[Update] Shared running query modal introduced

closes #1592

    • -0
    • +44
    /exec/java-exec/src/main/resources/rest/runningQuery.ftl
    • binary
    /exec/java-exec/src/main/resources/rest/static/img/loader.gif
DRILL-6933: Fix ctrl+enter when Impersonation is disabled

Without impersonation, the key combination did not work because a conditional block of the FTLs skipped the Javascript functions required for submitting with the key combination.

This commit fixes that by unifying the functions that are common to both approaches of submitting a query (with and without impersoination)

[Update] Changes made to isOnlyImpersonationEnabled method and onlyImpersonationEnabled variable

[Update] userName is a global var in the script

closes #1591

DRILL-6921: Add Clear button for /options filter

This commit adds the following search enhancements:

1. Addition of a clear ('x') button in the search field

2. If a filter term has been entered in the search box, on clicking the Update or Reset-to-Default button of any option, the reloaded page will re-apply the last entered filter term in the search box.

3. If the search box is empty, on clicking the Update or Reset-to-Default button of any option, the reloaded page will automatically filter out everything except the updated/reset option to show the changed value.

4. Customization of the quick search terms. (using defaults in drill-module.conf)

closes #1588

DRILL-6879: Show warnings for potential performance issues

1. Introduced warning for non-progressive fragments. Based on a threshold (`drill.exec.http.profile.warning.progress.threshold`), if all fragments have not made progress within that time, a warning is issued. The default is 5 minutes (300 sec)

2. Introduced a warning if any of the buffered operators spill to disk.

3. Introduced a warning for operators where the longest running fragment runs beyond a minimum threshold (drill.exec.http.profile.warning.time.skew.min), and runs atleast 2 times longer than the average (drill.exec.http.profile.warning.time.skew.ratio.process). The clock symbol with a tooltip indicates the extent of the skew. For wait times, the ratio is defined by `drill.exec.http.profile.warning.time.skew.ratio.wait`

3. Introduced a warning for operators where the average wait time of a scan operator exceeds its processing time, for a minimum threshold (drill.exec.http.profile.warning.scan.wait.min). The turtle symbol with a tooltip indicates which scan operator spent more time waiting than processing.

4. TableBuilder Refactored

a. Using attribute map instead of String arguments, eg. for 'title'

b. Removed APIs that pass a hyperlink since that is never used.

closes #1572

    • binary
    /exec/java-exec/src/main/resources/rest/static/img/turtle.png
DRILL-6883: Force reload of options after resetting parameter

If an update is initiated, the webpage loaded changes. Attempting a reset tries to reload this page, but it does not show the updated values from the back end.

This patch forces the reload of updated values by redirecting to /options page

Use AJAX call for Update and Reset

Eliminates the need for window.location=<redirectURL>

is Number check

Added comments to explain extraction

close apache/drill#1563

DRILL-6668: In Web UI, highlight options that are not default values

This commit introduces a new button on the options page that allows a user to reset an option to its system default value.

To simplify things, a tooltip is shown when the mouse hovers over the button. If the option value is already default, the button is disabled.

Currently, the Update button redirects to /option/optionName . This change reuses what we already are using to set the default (using AJAX) and auto-refreshing

Switch [Default] label to [Reset]

Patch To Pass StatusResourcesTest

closes #1543

DRILL-6837: Missing reference for dynamically created Javascript library

A PR for DRILL-6084 broke the auto-complete library required for Edit Query page, with the removal of the static javascript library.

https://github.com/apache/drill/pull/1491/files#diff-2bb3c04f1a9eead1faf57212cce5d2a9

The dynamically loaded script is not defined in the Freemarker template page.

The fix is trivial.

DRILL-6611: Add Ctrl+Enter support for query submission

1. BugFix on parent commit: Ensure query submission is done with user name when impersonation is enabled.

2. Support non-Mac browsers

3. Support keyboard submission for profile pages with Edit Query tab.

DRILL-6333: Fixed Quotation marks

Initial step to making the source-code ready for Javadoc generation

This closes #1229

DRILL-143: Support CGROUPs resource management

Introduces the DRILLBIT_CGROUP option in drill-env.sh.

The startup script checks if the specified CGroup (ver 2) is available and tries to apply it to the launched Drillbit JVM.

This would benefit not just Drill-on-YARN usecases, but any setup that would like CGroups for enforcement of (cpu) resources management.

(Also introduced SYS_CGROUP_DIR to account for possible non default locations of CGroup).

e.g when Drillbit is configured to use `drillcpu` cgroup

```

[root@maprlabs ~]# /opt/mapr/drill/apache-drill-1.14.0-SNAPSHOT/bin/drillbit.sh restart

Stopping drillbit

..

Starting drillbit, logging to /var/log/drill/drillbit.out

WARN: Drillbit's CPU resource usage will be managed under the CGroup : drillcpu (up to 4.00 cores allowed)

```

e.g. Non-existent CGroup `droolcpu` is used

```

[root@kk127 ~]# /opt/mapr/drill/apache-drill-1.14.0-SNAPSHOT/bin/drillbit.sh restart

Stopping drillbit

..

Starting drillbit, logging to /var/log/drill/drillbit.out

ERROR: cgroup droolcpu does not found. Ensure that daemon is running and cgroup exists

```

closes #1200

    • -0
    • +10
    /distribution/src/resources/drill-env.sh
    • -0
    • +39
    /distribution/src/resources/drillbit.sh
DRILL-6103: lsb_release: command not found

close apache/drill#1191

Thanks to Sanel Zukan for providing a small patch that checks for /etc/fedora-release path. This is more common, than lsb_release command on Linux distros.

DRILL-6303: Provide a button to copy the Drillbit's JStack shown in /threads

Provides a button in the web UI to copy the thread stack to the user's clipboard.

closes #1199

DRILL-6279: Indicate operators that spilled in-memory data to disk on Web UI

As part of this coomit added Bootstrap's Glyphicons.

closes #1197

DRILL-6289: Cluster view should show more relevant information

Protobuf change to carry HTTP port info

Allow CORS for access to remote Drillbit metrics

Cross-origin resource sharing (CORS) is required to ensure that the WebServer is able serve REST calls for status pages.

Materialize relevant metrics

1. Heap memory (incl usage)

2. Heap memory (incl usage)

3. Average System Load (last 1 min)

4. Option to view from other nodes (pop out)

5. Added Glyphicons

Update System Table and related tests

1. Updated System Table to show HTTP port

2. Updated unit tests

Skip updating remote bit info when HTTPS (SSL) or Authentication is enabled.

Default CpuGaugeSet is public; Added Gauges

* CPU Utiization by Drill

* Uptime

Show ALL Buttons, but do HTTPS Check

Reduce power button to icon

Allowing CORS for /status/metrics only

Accounting for situations when JVM does not report Process CPU Load

i.e. returned value is negative.

See https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad()

Addressed shutdown security conditions

Added C++ Client Protobuf

Added steps for Protobuf generation to protocol/readme.txt

This closes #1203

    • -0
    • +1
    /protocol/src/main/protobuf/Coordination.proto
DRILL-6224: Publish metrics gauge values correctly

The `metrics.ftl` page had gauges incorrectly set to near zero values. The commit for metrics.ftl fixes that, and also provides an estimate of the current direct memory actively in use (based on the `drill.allocator.root.used` value reported by the Drillbit)

closes #1160

DRILL-6197: Skip duplicate entry for OperatorStats

org.apache.drill.exec.ops.FragmentStats should skip injecting the org.apache.drill.exec.ops.OperatorStats instance for these operators:

org.apache.drill.exec.proto.beans.CoreOperatorType.SCREEN

org.apache.drill.exec.proto.beans.CoreOperatorType.SINGLE_SENDER

org.apache.drill.exec.proto.beans.CoreOperatorType.BROADCAST_SENDER

org.apache.drill.exec.proto.beans.CoreOperatorType.HASH_PARTITION_SENDER

closes #1141