drill

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6804: Simplify usage of OperatorPhase in HashAgg.

DRILL-6810: Disable NULL_IF_NULL NullHandling for functions with ComplexWriter closes #1509

DRILL-6084: Show Drill functions in WebUI for autocomplete

Building on top of DRILL-3988 and leveraging DRILL-5868, this allows support for Drill functions to be now available in the WebUI.

If users wants UDFs to show up, they should place the UDF jars in the `$DRILL_HOME/jars/3rdparty` directory so that this can be loaded during the Drillbit's startup.

Concept of internal Drill functions are introduced. With this, internal Drill functions like `ConvertToNullableXYZ` has been marked as internal.

The WebUI will not show these functions. However, they are still visible in `sys.functions` table with an additional column indicating that it is an internal function.

Tests have been added as a part of this commit to verify the internal functions concept.

DRILL-1328: Support table statistics

  1. … 38 more files in changeset.
DRILL-6793: FragmentExecutor cannot send its final state for the case when RootExec root wasn't initialized

closes #1506

DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf reference count bugs & tune the execution flow & support left deep tree

closes #1504

  1. … 14 more files in changeset.
DRILL-6381: Address code review comments (part 3).

DRILL-6381: Add missing joinControl logic for INTERSECT_DISTINCT.

- Modified HashJoin's probe phase to process INTERSECT_DISTINCT.

- NOTE: For build phase, the functionality will be same as for SemiJoin when it is added later.

DRILL-6381: Address code review comment for intersect_distinct.

DRILL-6381: Rebase on latest master and fix compilation issues.

DRILL-6381: Generate protobuf files for C++ native client.

DRILL-6381: Use shaded Guava classes. Add more comments and Javadoc.

  1. … 20 more files in changeset.
DRILL-6795: Upgrade Janino compiler from 2.7.6 to 3.0.10

closes #1503

DRILL-540: Allow querying hive views in Drill

1. Added DrillHiveViewTable which allows construction of DrillViewTable based

on Hive metadata

2. Added initialization of DrillHiveViewTable in HiveSchemaFactory

3. Extracted conversion of Hive data types from DrillHiveTable

to HiveToRelDataTypeConverter

4. Removed throwing of UnsupportedOperationException from HiveStoragePlugin

5. Added TestHiveViewsSupport and authorization tests

6. Added closeSilently() method to AutoCloseables

closes #1559

DRILL-6791: Scan projection framework

The "schema projection" mechanism:

* Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *) projection.

* Handles null columns (for projection a column "x" that does not exist in the base table.)

* Handles constant columns as used for file metadata (AKA "implicit" columns).

* Handle schema persistence: the need to reuse the same vectors across different scanners

* Provides a framework for consuming externally-supplied metadata

* Since we don't yet have a way to provide "real" metadata, obtains metadata hints from

previous batches and from the projection list (a.b implies that "a" is a map, c[0]

implies that "c" is an array, etc.)

* Handles merging the set of data source columns and null columns to create the final output batch.

* Running tests found a failure due to an uninialized "bits" vector. Added code to explicitly fill

the bits vectors with zeros in the "result set loader."

  1. … 19 more files in changeset.
DRILL-3988: Expose Drill built-in functions & UDFs in a system table (#1483)

This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill.

An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill.

```

0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source;

+-----------------------------------------+----------------+

| source | functionCount |

+-----------------------------------------+----------------+

| built-in | 2704 |

| simple-drill-function-1.0-SNAPSHOT.jar | 12 |

| drill-url-tools-1.0.jar | 1 |

+-----------------------------------------+----------------+

3 rows selected (0.209 seconds)

```

The system table exposes information as shown. The UDF is initialized, making the `returnType` available.

The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a UDF that has overloaded arguments (see `signature`).

The `url_parse(VARCHAR-REQUIRED)` function is another example of an initialized UDF.

Rest are built-in functions that meet the query's filter criteria.

```

0: jdbc:drill:schema=sys> select * from sys.functions where name like 'random' or name like '%url%';

+-------------+----------------------------------+-------------+-----------------------------------------+

| name | signature | returnType | source |

+-------------+----------------------------------+-------------+-----------------------------------------+

| parse_url | VARCHAR-REQUIRED | LATE | built-in |

| random | | FLOAT8 | built-in |

| random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | FLOAT8 | simple-drill-function-1.0-SNAPSHOT.jar |

| url_decode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_encode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_parse | VARCHAR-REQUIRED | LATE | drill-url-tools-1.0.jar |

+-------------+----------------------------------+-------------+-----------------------------------------+

6 rows selected (0.619 seconds)

```

DRILL-6797: Fix UntypedNull handling for complex types

DRILL-6381: Address review comments (part 2): fix formatting issues and add javadoc.

  1. … 15 more files in changeset.
DRILL-6788: Intermittent unit test failure TestDrillbitResilience.failsWhenParsing: Query state should be FAILED (and not COMPLETED) closes #1499

DRILL-6798: Planner changes to support semi-join.

  1. … 5 more files in changeset.
DRILL-6775: The schema for empty output is not shown in Drill Web UI

Removed an excess check for the result emptiness that prevented retrieval of the column names.

close apache/drill#1498

DRILL-6731: use thread pool to run the runtime filter aggregating work closes #1459

DRILL-6785: DataClient is using RootAllocator in the bootstrap instead of dataPool

closes #1502

DRILL-6410: Fixed memory leak in flat Parquet reader

DRILL-6783: CAST string literal as INTERVAL MONTH/YEAR works inconsistently when selecting from a table with multiple rows

close apache/drill#1496

DRILL-6776: Removed external jQuery dependency

DRILL-6773: The renamed schema with aliases is not shown for queries on empty directories

closes #1492

  1. … 3 more files in changeset.
DRILL-6770: JsonTableGroupScan should use new MapRDB 6.1.0 APIs

closes #1489

DRILL-6768: Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when option is enabled closes #1494

  1. … 10 more files in changeset.
DRILL-786: Allow CROSS JOIN syntax

1. Removed throw statement in UnsupportedOperatorsVisitor

2. Extended UnsupportedRelOperatorException's message

closes #1488

DRILL-6764: Query fails with IOB when Unnest has reference to deep nested field like (t.c_orders.o_lineitems).

closes #1487

DRILL-6765: Exclude unused shaded guava classes from drill-jdbc-all jar

closes #1486

DRILL-6759: Make columns array name for csv data case insensitive

closes #1485

DRILL-6766: Lateral Unnest query : IllegalStateException - rowId in right batch of lateral is smaller than rowId in left batch being processed Note: Issue was in StreamingAgg where if output from one or multiple input batch was splitting into multiple output batch, then remaining input records were discarded after producing first output batch closes #1490

DRILL-6381: Address code review comments.