drill

Clone Tools
  • last updated 13 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6791: Scan projection framework

The "schema projection" mechanism:

* Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *) projection.

* Handles null columns (for projection a column "x" that does not exist in the base table.)

* Handles constant columns as used for file metadata (AKA "implicit" columns).

* Handle schema persistence: the need to reuse the same vectors across different scanners

* Provides a framework for consuming externally-supplied metadata

* Since we don't yet have a way to provide "real" metadata, obtains metadata hints from

previous batches and from the projection list (a.b implies that "a" is a map, c[0]

implies that "c" is an array, etc.)

* Handles merging the set of data source columns and null columns to create the final output batch.

* Running tests found a failure due to an uninialized "bits" vector. Added code to explicitly fill

the bits vectors with zeros in the "result set loader."

  1. … 19 more files in changeset.
DRILL-3988: Expose Drill built-in functions & UDFs in a system table (#1483)

This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill.

An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill.

```

0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source;

+-----------------------------------------+----------------+

| source | functionCount |

+-----------------------------------------+----------------+

| built-in | 2704 |

| simple-drill-function-1.0-SNAPSHOT.jar | 12 |

| drill-url-tools-1.0.jar | 1 |

+-----------------------------------------+----------------+

3 rows selected (0.209 seconds)

```

The system table exposes information as shown. The UDF is initialized, making the `returnType` available.

The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a UDF that has overloaded arguments (see `signature`).

The `url_parse(VARCHAR-REQUIRED)` function is another example of an initialized UDF.

Rest are built-in functions that meet the query's filter criteria.

```

0: jdbc:drill:schema=sys> select * from sys.functions where name like 'random' or name like '%url%';

+-------------+----------------------------------+-------------+-----------------------------------------+

| name | signature | returnType | source |

+-------------+----------------------------------+-------------+-----------------------------------------+

| parse_url | VARCHAR-REQUIRED | LATE | built-in |

| random | | FLOAT8 | built-in |

| random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | FLOAT8 | simple-drill-function-1.0-SNAPSHOT.jar |

| url_decode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_encode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_parse | VARCHAR-REQUIRED | LATE | drill-url-tools-1.0.jar |

+-------------+----------------------------------+-------------+-----------------------------------------+

6 rows selected (0.619 seconds)

```

DRILL-6797: Fix UntypedNull handling for complex types

DRILL-6381: Address review comments (part 2): fix formatting issues and add javadoc.

  1. … 15 more files in changeset.
DRILL-6788: Intermittent unit test failure TestDrillbitResilience.failsWhenParsing: Query state should be FAILED (and not COMPLETED) closes #1499

DRILL-6798: Planner changes to support semi-join.

  1. … 5 more files in changeset.
DRILL-6775: The schema for empty output is not shown in Drill Web UI

Removed an excess check for the result emptiness that prevented retrieval of the column names.

close apache/drill#1498

DRILL-6731: use thread pool to run the runtime filter aggregating work closes #1459

DRILL-6785: DataClient is using RootAllocator in the bootstrap instead of dataPool

closes #1502

DRILL-6410: Fixed memory leak in flat Parquet reader

DRILL-6783: CAST string literal as INTERVAL MONTH/YEAR works inconsistently when selecting from a table with multiple rows

close apache/drill#1496

DRILL-6776: Removed external jQuery dependency

DRILL-6773: The renamed schema with aliases is not shown for queries on empty directories

closes #1492

  1. … 3 more files in changeset.
DRILL-6770: JsonTableGroupScan should use new MapRDB 6.1.0 APIs

closes #1489

DRILL-6768: Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when option is enabled closes #1494

  1. … 10 more files in changeset.
DRILL-786: Allow CROSS JOIN syntax

1. Removed throw statement in UnsupportedOperatorsVisitor

2. Extended UnsupportedRelOperatorException's message

closes #1488

DRILL-6764: Query fails with IOB when Unnest has reference to deep nested field like (t.c_orders.o_lineitems).

closes #1487

DRILL-6765: Exclude unused shaded guava classes from drill-jdbc-all jar

closes #1486

DRILL-6759: Make columns array name for csv data case insensitive

closes #1485

DRILL-6766: Lateral Unnest query : IllegalStateException - rowId in right batch of lateral is smaller than rowId in left batch being processed Note: Issue was in StreamingAgg where if output from one or multiple input batch was splitting into multiple output batch, then remaining input records were discarded after producing first output batch closes #1490

DRILL-6381: Address code review comments.

add ssl link to doc

edit SP config basics to update case sensitivity info, fix link to explain command

DRILL-6762: Fix dynamic UDFs versioning issue

1. Added UndefinedVersionDelegatingStore to serve as versioned wrapper for those stores that do not support versioning.

2. Aligned remote and local function registries version type. Type will be represented as int since ZK version is returned as int.

3. Added NOT_AVAILABLE and UNDEFINED versions to DataChangeVersion holder to indicate proper registry state.

4. Added additional trace logging.

5. Minor refactoring and clean up.

closes #1484

  1. … 7 more files in changeset.
DRILL-6731: Resolving race conditions in RuntimeFilterSink Add condition variable to avoid starvation of producer thread while acquiring queue lock

DRILL-6763: Codegen optimization of SQL functions with constant values(#1481)

closes #1481

  1. … 5 more files in changeset.
DRILL-6755: Avoid building Hash Table for inner/left join when probe side is empty

- Preparations and cleanup for DRILL-6755

clsoes #1480

DRILL-6761: Updated table of contents on the REST-API page

DRILL-6824: Handle schema changes in MapRDBJsonRecordReader closes #1518

DRILL-6753: Fix show files command to return result the same way as before

1. Add ACCESS_TIME column.

2. Re-write show files command to return result using ShowFilesCommandResult to maintain backward compatibility.

closes #1477