Clone
 

kunal khatua <kkhatua@users.noreply.github.com> in drill

DRILL-7110: Skip writing profile when an ALTER SESSION is executed (#1703)

Allows (by default) for `ALTER SESSION SET <option>=<value>` queries to NOT be writen to the profile store. This would avoid the risk of potentially adding up to a lot of profiles being written unnecessarily, since those changes are also reflected on the queries that follow.

DRILL-6886: Cancel Query Button for running queries

The Freemarker template missed a check for a query's Running state to display the [Cancel] button.

The fix is trivial and only needs the check.

DRILL-6848: Duration panel in a query profile's WebUI does not open

DRILL-5571 ( PR #1531 ) accidentally introduced a bug that permanently keeps the "Duration" panel closed instead of opening it. This trivial patch fixes that.

DRILL-2035: Add ability to cancel multiple queries

Currently Drill UI allows canceling one query at a time.

This commit (on lines of DRILL-5571 / PR #1531) allows for cancelling multiple `running` queries.

DRILL-5571: Cancel running query from its Web UI

Currently, a running query can be cancelled by navigating to the Edit Query Tab of the profile page, making it difficult to access. This commit allows the user to cancel.

In addition, the Duration segment is also made visible by default rather than collapsed.

DRILL-6715: Update descriptions for System Options table

With introduction of DRILL-5735 , the descriptions for about half the system options still remain missing. This commit collects descriptions review by @bbevens

1. Update options for HashAgg/Join (@Ben-Zvi )

2. Update options for Parquet Reader/Writer (@sachouche )

3. Update options for Planners (@HanumathRao , @vdiravka , @KazydubB )

4. Update options for BatchSizing (@bitblender )

5. Update options for Planner Optimizations (@arina-ielchiieva )

6. Update options for Security & Kafka (Krystal Nguyen)

7. Update options for Misc entries (@arina-ielchiieva , @vvysotskyi )

In additional, there is a patch for `org.apache.drill.exec.compile.ClassTransformer.scalar_replacement` , which appears to have replaced `exec.compile.scalar_replacement`. References to the latter have been removed to avoid confusion.

Additional changes include moving the `ClassTransformer` validator to `ExecConstants.java`

Adding support for internal options' descriptions

Removed mention of {{Will be removed in 1.15.0}}. (Refer DRILL-6527)

DRILL-6084: Show Drill functions in WebUI for autocomplete

Building on top of DRILL-3988 and leveraging DRILL-5868, this allows support for Drill functions to be now available in the WebUI.

If users wants UDFs to show up, they should place the UDF jars in the `$DRILL_HOME/jars/3rdparty` directory so that this can be loaded during the Drillbit's startup.

Concept of internal Drill functions are introduced. With this, internal Drill functions like `ConvertToNullableXYZ` has been marked as internal.

The WebUI will not show these functions. However, they are still visible in `sys.functions` table with an additional column indicating that it is an internal function.

Tests have been added as a part of this commit to verify the internal functions concept.

DRILL-3988: Expose Drill built-in functions & UDFs in a system table (#1483)

This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill.

An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill.

```

0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source;

+-----------------------------------------+----------------+

| source | functionCount |

+-----------------------------------------+----------------+

| built-in | 2704 |

| simple-drill-function-1.0-SNAPSHOT.jar | 12 |

| drill-url-tools-1.0.jar | 1 |

+-----------------------------------------+----------------+

3 rows selected (0.209 seconds)

```

The system table exposes information as shown. The UDF is initialized, making the `returnType` available.

The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a UDF that has overloaded arguments (see `signature`).

The `url_parse(VARCHAR-REQUIRED)` function is another example of an initialized UDF.

Rest are built-in functions that meet the query's filter criteria.

```

0: jdbc:drill:schema=sys> select * from sys.functions where name like 'random' or name like '%url%';

+-------------+----------------------------------+-------------+-----------------------------------------+

| name | signature | returnType | source |

+-------------+----------------------------------+-------------+-----------------------------------------+

| parse_url | VARCHAR-REQUIRED | LATE | built-in |

| random | | FLOAT8 | built-in |

| random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | FLOAT8 | simple-drill-function-1.0-SNAPSHOT.jar |

| url_decode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_encode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_parse | VARCHAR-REQUIRED | LATE | drill-url-tools-1.0.jar |

+-------------+----------------------------------+-------------+-----------------------------------------+

6 rows selected (0.619 seconds)

```

DRILL-6702: Disable CPU Reporting for non-HotSpot JDKs

When running Drill on the IBM JDK (J9), the webUI throws an ClassCastException :

Caused by: java.lang.ClassCastException: com.ibm.lang.management.ExtendedOperatingSystem incompatible with com.sun.management.OperatingSystemMXBean

This PR simply disables that, since Drill should ideally be recompiled with these alternative JDKs.

DRILL-6684: Swap sys.options and sys.options_val tables

The current `sys.options` table has a verbose layout, because of which `sys.options_internal` was introduced. The latter also supports descriptions, which means it makes sense to have that table as the new `sys.options`.

This PR deprecates the old format, so that any dependencies continue to make use of it as long as required.

DRILL-6588: Make Sys tables of nullable datatypes

This is to address the generic problem of NULL values being projected as a string because of all datatypes being non-nullable.

This patch only applies to tables backed by the PojoReader (in our case, System tables). Added NonNullable annotations wherever application in any of the System tables, along with a unit test that verifies both nullable and non-nullable datatypes exist in the system tables.

closes #1371

DRILL-6591: Show Exception for failed queries submitted in WebUI

* DRILL-6591: Show Exception for failed queries submitted in WebUI

When query fails on Web UI result page no error is shown, only "No result found."

This was because DRILL-6477 (PR #1309) switched to `WebUserConnection.await(long timeoutInMillis)` . Unlike the original `WebUserConnection.await()`, this method did not throw any UserException generated by a query failure. The fix was to use WebUserConnection.getError() method to know about failure of the query and throw UserRemoteException with that.

closes #1379

DRILL-6583: Add space between pagination links in Profiles (WebUI) list

Inject a small space to improve readability of the pagination links (CSS)

closes #1369

DRILL-6515: Render a link between the Unnest operator and it's source

* DRILL-6515: Render a link between the Unnest operator and it's source

1. Unnest operator is expected to specify srcOp=##-## to help identify the source of its input.

2. Implicit Source Map is leveraged to capture implicit data flow pipelines between Unnest and Lateral. (Can be expanded to other operators too).

3. In addition, code refactored for more readability.

* Minor comment updates

* Minor Update

Converted implicitSrc to a local variable

closes #1327

DRILL-6467: Percentage usage of memory is reported as zero by the WebUI

The memory reported as a percentage is incorrectly reported as 0%, irrespective of the actual usage.

closes #1305

DRILL-6432: Show Button to print visualized query plan (#1278)

* DRILL-6432: Show Button to print visualized query plan

Having the capability to only print the visualized query plan means the user can specifyi the zoom level (in the printer settings) when printing large complex plans that might span multiple pages.

* Update based on review comment

Padded button by 15px to avoid sticking to pane

closes #1278

DRILL-5242: The UI breaks when trying to render profiles having unknown metrics

Skip any metrics whose metric ID is unknown, This prevents any ArrayIndexOutOfBoundsException from being thrown and breaking the UI rendering.