Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-5844: Incorrect values of TABLE_TYPE returned from method DatabaseMetaData.getTables of JDBC API

closes #1904

  1. … 5 more files in changeset.
DRILL-7357: Expose Drill Metastore data through information_schema

1. Add additional columns to TABLES and COLUMNS tables.

2. Add PARTITIONS table.

3. General refactoring to adjust information_schema data retrieval from multiple sources.

closes #1860

  1. … 33 more files in changeset.
DRILL-6946: Implement java.sql.Connection setSchema and getSchema methods in DrillConnectionImpl closes #1596

  1. … 9 more files in changeset.
DRILL-6084: Show Drill functions in WebUI for autocomplete

Building on top of DRILL-3988 and leveraging DRILL-5868, this allows support for Drill functions to be now available in the WebUI.

If users wants UDFs to show up, they should place the UDF jars in the `$DRILL_HOME/jars/3rdparty` directory so that this can be loaded during the Drillbit's startup.

Concept of internal Drill functions are introduced. With this, internal Drill functions like `ConvertToNullableXYZ` has been marked as internal.

The WebUI will not show these functions. However, they are still visible in `sys.functions` table with an additional column indicating that it is an internal function.

Tests have been added as a part of this commit to verify the internal functions concept.

  1. … 13 more files in changeset.
DRILL-3988: Expose Drill built-in functions & UDFs in a system table (#1483)

This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill.

An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill.

```

0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source;

+-----------------------------------------+----------------+

| source | functionCount |

+-----------------------------------------+----------------+

| built-in | 2704 |

| simple-drill-function-1.0-SNAPSHOT.jar | 12 |

| drill-url-tools-1.0.jar | 1 |

+-----------------------------------------+----------------+

3 rows selected (0.209 seconds)

```

The system table exposes information as shown. The UDF is initialized, making the `returnType` available.

The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a UDF that has overloaded arguments (see `signature`).

The `url_parse(VARCHAR-REQUIRED)` function is another example of an initialized UDF.

Rest are built-in functions that meet the query's filter criteria.

```

0: jdbc:drill:schema=sys> select * from sys.functions where name like 'random' or name like '%url%';

+-------------+----------------------------------+-------------+-----------------------------------------+

| name | signature | returnType | source |

+-------------+----------------------------------+-------------+-----------------------------------------+

| parse_url | VARCHAR-REQUIRED | LATE | built-in |

| random | | FLOAT8 | built-in |

| random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | FLOAT8 | simple-drill-function-1.0-SNAPSHOT.jar |

| url_decode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_encode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_parse | VARCHAR-REQUIRED | LATE | drill-url-tools-1.0.jar |

+-------------+----------------------------------+-------------+-----------------------------------------+

6 rows selected (0.619 seconds)

```

  1. … 9 more files in changeset.
DRILL-6753: Fix show files command to return result the same way as before

1. Add ACCESS_TIME column.

2. Re-write show files command to return result using ShowFilesCommandResult to maintain backward compatibility.

closes #1477

  1. … 7 more files in changeset.
DRILL-6684: Swap sys.options and sys.options_val tables

The current `sys.options` table has a verbose layout, because of which `sys.options_internal` was introduced. The latter also supports descriptions, which means it makes sense to have that table as the new `sys.options`.

This PR deprecates the old format, so that any dependencies continue to make use of it as long as required.

  1. … 7 more files in changeset.
DRILL-5735: UI options grouping and filtering & Metrics hints Allow search/sort in the Options webUI Additional details, like the description is provided as well in a JScript lookup map. This helps reduce the need for the server to constantly recreate the entire page with the description details, as the client browser can fill in these details. Developers will be expected to update the description as old/new options are introduced or deprecated. System table options_val also carries the description, albeit a 'shorter' version.

closes #1279

  1. … 14 more files in changeset.
DRILL-6492: Ensure schema / workspace case insensitivity in Drill

1. StoragePluginsRegistryImpl was updated:

a. for backward compatibility at init to convert all existing storage plugins names to lower case, in case of duplicates, to log warning and skip the duplicate.

b. to wrap persistent plugins registry into case insensitive store wrapper (CaseInsensitivePersistentStore) to ensure all given keys are converted into lower case when performing insert, update, delete, search operations.

c. to load system storage plugins dynamically by @SystemStorage annotation.

2. StoragePlugins class was updated to stored storage plugins configs by name in case insensitive map.

3. SchemaUtilities.searchSchemaTree method was updated to convert all schema names into lower case to ensure that are they are matched case insensitively (all schemas are stored in Drill in lower case).

4. FileSystemConfig was updated to store workspaces by name in case insensitive hash map.

5. All plugins schema factories are now extend AbstractSchemaFactory to ensure that given schema name is converted to lower case.

6. New method areTableNamesAreCaseInsensitive was added to AbstractSchema to indicate if schema tables names are case insensitive. By default, false. Schema implementation is responsible for table names case insensitive search in case it supports one. Currently, information_schema, sys and hive do so.

7. System storage plugins (information_schema, sys) were refactored to ensure their schema, table names are case insensitive, also the annotation @SystemPlugin and additional constructor were added to allow dynamically load system plugins at storage plugin registry during init phase.

8. MetadataProvider was updated to concert all schema filter conditions into lower case to ensure schema would be matched case insensitively.

9. ShowSchemasHandler, ShowTablesHandler, DescribeTableHandler were updated to ensure schema / tables names (this depends if schema supports case insensitive table names) would be found case insensitively.

git closes #1439

  1. … 55 more files in changeset.
DRILL-6680: Expose show files command into INFORMATION_SCHEMA

  1. … 21 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2065 more files in changeset.
DRILL-6289: Cluster view should show more relevant information

Protobuf change to carry HTTP port info

Allow CORS for access to remote Drillbit metrics

Cross-origin resource sharing (CORS) is required to ensure that the WebServer is able serve REST calls for status pages.

Materialize relevant metrics

1. Heap memory (incl usage)

2. Heap memory (incl usage)

3. Average System Load (last 1 min)

4. Option to view from other nodes (pop out)

5. Added Glyphicons

Update System Table and related tests

1. Updated System Table to show HTTP port

2. Updated unit tests

Skip updating remote bit info when HTTPS (SSL) or Authentication is enabled.

Default CpuGaugeSet is public; Added Gauges

* CPU Utiization by Drill

* Uptime

Show ALL Buttons, but do HTTPS Check

Reduce power button to icon

Allowing CORS for /status/metrics only

Accounting for situations when JVM does not report Process CPU Load

i.e. returned value is negative.

See https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad()

Addressed shutdown security conditions

Added C++ Client Protobuf

Added steps for Protobuf generation to protocol/readme.txt

This closes #1203

  1. … 16 more files in changeset.
DRILL-5068: Create sys.profiles table

(Thanks to Hongze Zhang for the initial work)

Introduced a non-distributed (i.e. the foreman Drillbit executes a single fragment query) system table to list the completed profiles that are visible to it in the persistent store. This allows for querying the profiles on the following common metrics:

queryId

startTime

foreman

fragments

user

queue

planTime

queueTime

executeTime

totalTime

state

query

An additional system table - `sys.profiles_json` has also been introduced. This only has the following fields:

queryId

json

This allows for more detail metrics to be queried on the profile as a JSON document. Users are expected to use `sys.profiles` to identify profiles on the available metrics (such as long executeTime or fragments), and use the queryId to find the corresponding JSON document representation for further drill down into the profiles.

Updated TestMetadataProvider with new tally for current table and column counting tests

Security check to prevent unauthorized access to profile listing

Updated based on review comments

This closes #1077

  1. … 8 more files in changeset.
DRILL-6036: Create sys.connections table

Introduced a distributed (i.e. each Drillbit executes a fragment) system table to list all the active client connections to all the Drillbits in a cluster. This is achieved by having the UserServer maintain a static (class-level) map of these connections, which is updated as and when connections are established and closed.

The following details are provided by the table:

user

targetUser

client

drillbit

established

duration

queries

isAuthenticated

isEncrypted

usingSSL

session

Security check to prevent unauthorized access to connection listing

Match is done between the inbound impersonated user and the available list of connections' targetUser property

Removed listing of targetUser

Removed getQueryContext from FragmentContext

FragmentContext provides isUserAuthenticationEnabled()

This closes #1076

  1. … 5 more files in changeset.
DRILL-5783, DRILL-5841, DRILL-5894: Rationalize test temp directories

This change includes:

DRILL-5783:

- A unit test is created for the priority queue in the TopN operator.

- The code generation classes passed around a completely unused function registry reference in some places so it is removed.

- The priority queue had unused parameters for some of its methods so it is removed.

DRILL-5841:

- Created standardized temp directory classes DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. And updated all unit tests to use them.

DRILL-5894:

- Removed the dfs_test storage plugin for tests and replaced it with the already existing dfs storage plugin.

Misc:

- General code cleanup.

- Removed unnecessary use of String.format in the tests.

This closes #984

  1. … 364 more files in changeset.
DRILL-5752 this change includes:

1. Increased test parallelism and fixed associated bugs

2. Added test categories and categorized tests appropriately

- Don't exclude anything by default

- Increase test timeout

- Fixed flakey test

closes #940

  1. … 265 more files in changeset.
DRILL-4286: Graceful shutdown of drillbit

closes #921

  1. … 26 more files in changeset.
DRILL-5723: Added System Internal Options That can be Modified at Runtime Changes include:

1. Addition of internal options.

2. Refactoring of OptionManagers and OptionValidators.

3. Fixed ambiguity in the meaning of an option type, and changed its name to accessibleScopes.

4. Updated javadocs in the Option System classes.

5. Added RestClientFixture for testing the Rest API.

6. Fixed flakey test in TestExceptionInjection caused by race condition.

7. Fixed various tests which started zookeeper but failed to shut it down at the end of tests.

8. Added port hunting to the Drill Webserver for testing

9. Fixed various flaky tests

10. Fix compile issue

closes #923

  1. … 85 more files in changeset.
DRILL-5547: Linking config options with system option manager

closes #868

  1. … 38 more files in changeset.
DRILL-5301: Server metadata API

Add a Server metadata API to the User protocol, to query server support

of various SQL features.

Add support to the client (DrillClient) to query this information.

Add support to the JDBC driver to query this information, if the server supports

the new API, or fallback to the previous behaviour (rely on Avatica defaults) otherwise.

close #764

    • -0
    • +48
    ./TestServerMetaProvider.java
  1. … 35 more files in changeset.
DRILL-4925: Add tableType filter to GetTables metadata query

- Adding tableType filter to GetTablesReq query (needed for JDBC and ODBC

drivers).

- Fix table type returned by sys and INFORMATION_SCHEMA tables

- Also fixes some protobuf typos to related classes.

this closes #612

Change-Id: If95246a312f6c6d64a88872936f516308874c2d2

  1. … 18 more files in changeset.
DRILL-4728: Add support for new metadata fetch APIs

+ Protobuf messages

- GetCatalogsReq -> GetCatalogsResp

- GetSchemasReq -> GetSchemasResp

- GetTablesReq -> GetTablesResp

- GetColumnsReq -> GetColumnsResp

+ Java Drill client changes

+ Server side changes to handle the metadata API calls

- Provide a self contained `Runnable` implementation for each metadata API

that process the requests and sends the response to client

- In `UserWorker` override the `handle` method that takes the `ResponseSender` and

send the response from the `handle` method instead of returning it.

- Add a method for each new API to UserWorker to submit the metadata work.

- Add a method `addNewWork(Runnable runnable)` to `WorkerBee` to submit a generic

`Runnable` to `ExecutorService`.

- Move out couple of methods from `QueryContext` into a separate interface

`SchemaConfigInfoProvider` to enable instantiating Schema trees without the

full `QueryContext`

+ New protobuf messages increased the `jdbc-all.jar` size. Up the limit to 21MB.

this closes #527

    • -0
    • +308
    ./TestMetadataProvider.java
  1. … 31 more files in changeset.
DRILL-4604: Generate warning on Web UI if drillbits version mismatch is detected

close apache/drill#482

  1. … 9 more files in changeset.