Clone Tools
  • last updated 22 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7337: Add vararg UDFs support

  1. … 37 more files in changeset.
DRILL-7317: Close ClassLoaders used for udf jars uploading when closing FunctionImplementationRegistry

- Fix issue with caching DrillMergeProjectRule and FunctionImplementationRegistry when different drillbits are started within the same JVM

  1. … 2 more files in changeset.
DRILL-3988: Expose Drill built-in functions & UDFs in a system table (#1483)

This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill.

An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill.

```

0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source;

+-----------------------------------------+----------------+

| source | functionCount |

+-----------------------------------------+----------------+

| built-in | 2704 |

| simple-drill-function-1.0-SNAPSHOT.jar | 12 |

| drill-url-tools-1.0.jar | 1 |

+-----------------------------------------+----------------+

3 rows selected (0.209 seconds)

```

The system table exposes information as shown. The UDF is initialized, making the `returnType` available.

The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a UDF that has overloaded arguments (see `signature`).

The `url_parse(VARCHAR-REQUIRED)` function is another example of an initialized UDF.

Rest are built-in functions that meet the query's filter criteria.

```

0: jdbc:drill:schema=sys> select * from sys.functions where name like 'random' or name like '%url%';

+-------------+----------------------------------+-------------+-----------------------------------------+

| name | signature | returnType | source |

+-------------+----------------------------------+-------------+-----------------------------------------+

| parse_url | VARCHAR-REQUIRED | LATE | built-in |

| random | | FLOAT8 | built-in |

| random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | FLOAT8 | simple-drill-function-1.0-SNAPSHOT.jar |

| url_decode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_encode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_parse | VARCHAR-REQUIRED | LATE | drill-url-tools-1.0.jar |

+-------------+----------------------------------+-------------+-----------------------------------------+

6 rows selected (0.619 seconds)

```

  1. … 8 more files in changeset.
DRILL-6762: Fix dynamic UDFs versioning issue

1. Added UndefinedVersionDelegatingStore to serve as versioned wrapper for those stores that do not support versioning.

2. Aligned remote and local function registries version type. Type will be represented as int since ZK version is returned as int.

3. Added NOT_AVAILABLE and UNDEFINED versions to DataChangeVersion holder to indicate proper registry state.

4. Added additional trace logging.

5. Minor refactoring and clean up.

closes #1484

  1. … 19 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

  1. … 982 more files in changeset.
DRILL-6492: Ensure schema / workspace case insensitivity in Drill

1. StoragePluginsRegistryImpl was updated:

a. for backward compatibility at init to convert all existing storage plugins names to lower case, in case of duplicates, to log warning and skip the duplicate.

b. to wrap persistent plugins registry into case insensitive store wrapper (CaseInsensitivePersistentStore) to ensure all given keys are converted into lower case when performing insert, update, delete, search operations.

c. to load system storage plugins dynamically by @SystemStorage annotation.

2. StoragePlugins class was updated to stored storage plugins configs by name in case insensitive map.

3. SchemaUtilities.searchSchemaTree method was updated to convert all schema names into lower case to ensure that are they are matched case insensitively (all schemas are stored in Drill in lower case).

4. FileSystemConfig was updated to store workspaces by name in case insensitive hash map.

5. All plugins schema factories are now extend AbstractSchemaFactory to ensure that given schema name is converted to lower case.

6. New method areTableNamesAreCaseInsensitive was added to AbstractSchema to indicate if schema tables names are case insensitive. By default, false. Schema implementation is responsible for table names case insensitive search in case it supports one. Currently, information_schema, sys and hive do so.

7. System storage plugins (information_schema, sys) were refactored to ensure their schema, table names are case insensitive, also the annotation @SystemPlugin and additional constructor were added to allow dynamically load system plugins at storage plugin registry during init phase.

8. MetadataProvider was updated to concert all schema filter conditions into lower case to ensure schema would be matched case insensitively.

9. ShowSchemasHandler, ShowTablesHandler, DescribeTableHandler were updated to ensure schema / tables names (this depends if schema supports case insensitive table names) would be found case insensitively.

git closes #1439

  1. … 55 more files in changeset.
DRILL-6656: Disallow extra semicolons and multiple statements on the same line.

closes #1415

  1. … 144 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2063 more files in changeset.
DRILL-6053: Avoid excessive locking in LocalPersistentStore

closes #1163

  1. … 17 more files in changeset.
DRILL-5601: Rollup of external sort fixes an improvements

- DRILL-5513: Managed External Sort : OOM error during the merge phase

- DRILL-5519: Sort fails to spill and results in an OOM

- DRILL-5522: OOM during the merge and spill process of the managed external sort

- DRILL-5594: Excessive buffer reallocations during merge phase of external sort

- DRILL-5597: Incorrect "bits" vector allocation in nullable vectors allocateNew()

- DRILL-5602: Repeated List Vector fails to initialize the offset vector

- DRILL-5617: Spill file name collisions when spill file is on a shared file system

0 DRILL-5445: bug in repeated map vector deserialization

- Workaround for DRILL-5656: Streaming Agg Batch forces sort to retain in-memory batches past NONE

- Fixes for the "record batch sizer" to handle for UNION, MAP, LIST types

Fixes a longstanding bug in the deserialization of a repeated map

vector read from a spill file. A few minor code cleanups also.

All of the bugs have to do with handling low-memory conditions, and with

correctly estimating the sizes of vectors, even when those vectors come

from the spill file or from an exchange. Hence, the changes for all of

the above issues are interrelated.

Also includes some fixes for tests:

* Certain tests require the ability to enforce the output size of the

memory merge/sort. Restored this option.

* Resolve issue with TestDrillbitResilience

* A particular test injects a fault during in-memory sort, but used a

single-batch input (which does not need the merge phase.)

Rather than introduce a new config property (the earlier solution),

altered the test to use input that returns more than one batch.

Two fixes forBasicPhysicalOpUnitTest — a test that uses JMockit to create a “fake”

fragment context.

1. No drillbit endpoint is available, so the SpillSet change that adds

the node to the spill path failed. Solution was to omit the node path

segment in such tests.

2. The Dynamic UDF registry is null causing a crash. This has nothing

to do with sort. Perhaps some pre-existing error? Anyway, added a check

for this condition.

close #860

  1. … 63 more files in changeset.
DRILL-4963: Fix issues with dynamically loaded overloaded functions

close #701

  1. … 24 more files in changeset.
DRILL-5043: Function that returns a unique id per session/connection similar to MySQL's CONNECTION_ID() #685

  1. … 27 more files in changeset.
DRILL-5085: Update description for dynamic UDFs directories in drill-env.sh and drill-module.conf

closes #672

  1. … 6 more files in changeset.
DRILL-4987: Use ImpersonationUtil to get process user’s groups in RemoteFunctionRegistry

closes #642

  1. … 1 more file in changeset.
DRILL-4726: Dynamic UDF Support

1) Configuration / parsing / options / protos

2) Zookeeper integration

3) Registration / unregistration / lazy-init

4) Unit tests

This closes #574

    • -0
    • +54
    ./FunctionHolder.java
    • -0
    • +377
    ./FunctionRegistryHolder.java
    • -0
    • +329
    ./LocalFunctionRegistry.java
    • -0
    • +269
    ./RemoteFunctionRegistry.java
  1. … 67 more files in changeset.