Clone Tools
  • last updated 12 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7761: Drill fails with OOM for the case of large filter conditions

  1. … 1 more file in changeset.
DRILL-7744: Move Filters from HTTP Storage Plugin to Drill Core

    • -0
    • +384
    ./base/filter/ConstantHolder.java
    • -0
    • +234
    ./base/filter/ExprNode.java
    • -0
    • +149
    ./base/filter/FilterPushDownListener.java
    • -0
    • +323
    ./base/filter/FilterPushDownStrategy.java
    • -0
    • +322
    ./base/filter/FilterPushDownUtils.java
    • -0
    • +98
    ./base/filter/RelOp.java
  1. … 8 more files in changeset.
DRILL-7725: Updates to the EVF2 framework

* Supports internal implicit columns

* Better support for standard conversions

* Handle several reader corner cases

* Simplified file reader

closes #2073

    • -12
    • +6
    ./easy/text/reader/CompliantTextBatchReader.java
  1. … 77 more files in changeset.
DRILL-7724: Refactor metadata controller batch

Also changed for (;;) infinite loops to

while (true) as preferred by IntelliJ.

    • -1
    • +1
    ./easy/json/parser/DummyValueParser.java
    • -2
    • +2
    ./easy/json/parser/JsonValueParser.java
  1. … 20 more files in changeset.
DRILL-7717: Support Mongo extended types in V2 JSON loader

Adds support for Mongo extended types to the JSON loader.

Refactors the JSON code to make such extensions easier.

Added support for provided schema with extended types.

    • -0
    • +107
    ./easy/json/extended/BaseExtendedValueParser.java
    • -0
    • +282
    ./easy/json/extended/ExtendedTypeFieldFactory.java
    • -0
    • +51
    ./easy/json/extended/ExtendedTypeNames.java
    • -0
    • +150
    ./easy/json/extended/MongoBinaryValueParser.java
    • -0
    • +89
    ./easy/json/extended/MongoDateValueParser.java
    • -0
    • +48
    ./easy/json/extended/SimpleExtendedValueParser.java
    • -0
    • +142
    ./easy/json/extended/package-info.java
    • -97
    • +0
    ./easy/json/loader/AbstractArrayListener.java
    • -81
    • +0
    ./easy/json/loader/AbstractValueListener.java
    • -0
    • +261
    ./easy/json/loader/BaseFieldFactory.java
    • -69
    • +0
    ./easy/json/loader/BigIntListener.java
    • -62
    • +0
    ./easy/json/loader/BooleanListener.java
    • -66
    • +0
    ./easy/json/loader/DoubleListener.java
    • -0
    • +66
    ./easy/json/loader/EmptyArrayFieldParser.java
    • -0
    • +135
    ./easy/json/loader/FieldDefn.java
  1. … 83 more files in changeset.
DRILL-7711: Add data path, parameter filter pushdown to HTTP plugin

Adds an option to specify the path to data so the plugin will

ignore REST message "overhead" except the actual data.

Allows specifying HTTP URL parameters as filter push-downs from

SQL.

* Revised scan cost model to allow a scan more freedom

to define cost. Needed to ensure that filter push-down

is actually accepted by Calcite.

* Reduced size of HTTP scan plan by including on the one

needed connection config rather than all of them.

* Revised URL building so that the proxy sees the full

URL after adding parameters, etc.

* Many code refinements.

* Added more details to README

    • -0
    • +7
    ./easy/json/loader/JsonLoaderImpl.java
    • -13
    • +33
    ./easy/json/parser/JsonStructureParser.java
    • -25
    • +114
    ./easy/json/parser/RootParser.java
  1. … 29 more files in changeset.
DRILL-7713: Upgrade misc libraries which outdated versions have reported vulnerabilities

1. Jackson

2. Retrofit

3. Commons-beanutils

4. Xalan

5. Xerdes

6. Commons-codec

7. Snakeyaml

8. Metadata-extractor

9. Protostuff

  1. … 13 more files in changeset.
DRILL-7703: Support for 3+D arrays in EVF JSON loader

Revises the EVF-based JSON loader to support nested

repeated lists.

    • -8
    • +9
    ./easy/json/loader/JsonLoaderImpl.java
    • -0
    • +2
    ./easy/json/loader/JsonLoaderOptions.java
    • -22
    • +45
    ./easy/json/loader/RepeatedListValueListener.java
    • -125
    • +245
    ./easy/json/loader/TupleListener.java
    • -20
    • +25
    ./easy/json/parser/JsonStructureParser.java
  1. … 7 more files in changeset.
DRILL-7701: EVF V2 Scan Framework

Revises the scan framework to use the revised schema resolution

introduced in DRILL-7696.

  1. … 57 more files in changeset.
DRILL-7437: Storage Plugin for Generic HTTP REST API

  1. … 35 more files in changeset.
DRILL-6168: Revise format plugin table functions

Allows table functions to inherit properties from a

defined format plugin.

Also DRILL-7612: enforces immutability for all format plugins.

    • -3
    • +7
    ./dfs/FormatPluginOptionExtractor.java
    • -53
    • +202
    ./dfs/FormatPluginOptionsDescriptor.java
    • -17
    • +22
    ./dfs/NamedFormatPluginConfig.java
    • -59
    • +94
    ./dfs/WorkspaceSchemaFactory.java
    • -26
    • +30
    ./easy/json/JSONFormatPlugin.java
    • -25
    • +43
    ./easy/text/TextFormatPlugin.java
    • -23
    • +33
    ./httpd/HttpdLogFormatConfig.java
  1. … 32 more files in changeset.
DRILL-7696: EVF v2 scan schema resolution

Provides the mechanism to resolve the scan schema from a

projection list, provided schema, early reader schema and

actual reader schema.

  1. … 75 more files in changeset.
DRILL-7683: Add "message parsing" to new JSON loader

Adds the ability to parse "extra" JSON around the data payload,

as often needed for a REST API.

closes #2045

    • -17
    • +88
    ./easy/json/loader/JsonLoaderImpl.java
    • -0
    • +9
    ./easy/json/loader/JsonLoaderOptions.java
    • -0
    • +14
    ./easy/json/parser/JsonStructureOptions.java
    • -0
    • +43
    ./easy/json/parser/MessageParser.java
    • -0
    • +141
    ./easy/json/parser/SimpleMessageParser.java
  1. … 5 more files in changeset.
DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

1. Upgraded Iceberg version and removed unneeded code for In / NotIn Expressions.

2. Updated Metastore Read / Modify interfaces to support required metadata types:

a. introduced abstract Read / Modify classes with boilerplate code;

b. added delete operation with filter and metadata type;

c. added metadata type validator which checks supported metadata types for each component;

d. made purge operation terminal;

e. made necessary changes in REAME.md files.

3. Added / updated unit tests.

closes #2042

  1. … 41 more files in changeset.
DRILL-7643: Fix issues with using columns with the same name as a reserved keyword

closes #2028

    • -0
    • +1
    ./easy/json/StatisticsCollectorImpl.java
  1. … 6 more files in changeset.
DRILL-7640: EVF-based JSON Loader

Builds on the JSON structure parser and several other PRs

to provide an enhanced, robust mechanism to read JSON data

into value vectors via the EVF. This is not the JSON reader,

rather it is the "V2" version of the JsonProcessor which

does the actual JSON parsing/loading work.

closes #2023

    • -0
    • +97
    ./easy/json/loader/AbstractArrayListener.java
    • -0
    • +81
    ./easy/json/loader/AbstractValueListener.java
    • -0
    • +69
    ./easy/json/loader/BigIntListener.java
    • -0
    • +62
    ./easy/json/loader/BooleanListener.java
    • -0
    • +66
    ./easy/json/loader/DoubleListener.java
    • -0
    • +72
    ./easy/json/loader/JsonLoader.java
    • -0
    • +345
    ./easy/json/loader/JsonLoaderImpl.java
    • -0
    • +44
    ./easy/json/loader/JsonLoaderOptions.java
    • -0
    • +82
    ./easy/json/loader/ListListener.java
    • -0
    • +173
    ./easy/json/loader/RepeatedListValueListener.java
    • -0
    • +107
    ./easy/json/loader/ScalarListener.java
    • -0
    • +143
    ./easy/json/loader/StructuredValueListener.java
    • -0
    • +405
    ./easy/json/loader/TupleListener.java
    • -0
    • +215
    ./easy/json/loader/UnknownFieldListener.java
    • -0
    • +61
    ./easy/json/loader/VarCharListener.java
  1. … 27 more files in changeset.
DRILL-7330: Implement metadata usage for all format plugins

    • -17
    • +22
    ./dfs/easy/EasyFormatPlugin.java
    • -43
    • +104
    ./dfs/easy/EasyGroupScan.java
    • -34
    • +0
    ./dfs/easy/SimpleFileTableMetadataProviderBuilder.java
    • -6
    • +28
    ./parquet/AbstractParquetGroupScan.java
    • -1
    • +1
    ./parquet/AbstractParquetScanBatchCreator.java
    • -27
    • +103
    ./parquet/BaseParquetMetadataProvider.java
    • -0
    • +249
    ./parquet/FilePushDownFilter.java
    • -55
    • +0
    ./parquet/ParquetFileTableMetadataProviderBuilder.java
    • -247
    • +0
    ./parquet/ParquetPushDownFilter.java
    • -486
    • +0
    ./parquet/ParquetTableMetadataProviderImpl.java
  1. … 48 more files in changeset.
DRILL-7631: Updates to the Json Structure Parser

Revised how "look-ahead" works. Added support for unknown

types.

closes #2016

    • -61
    • +28
    ./easy/json/parser/ArrayListener.java
    • -9
    • +54
    ./easy/json/parser/ArrayParser.java
    • -2
    • +0
    ./easy/json/parser/DummyValueParser.java
    • -0
    • +116
    ./easy/json/parser/JsonValueParser.java
    • -53
    • +57
    ./easy/json/parser/ObjectListener.java
    • -3
    • +57
    ./easy/json/parser/ObjectParser.java
    • -0
    • +16
    ./easy/json/parser/TokenIterator.java
    • -0
    • +85
    ./easy/json/parser/ValueDef.java
    • -0
    • +99
    ./easy/json/parser/ValueDefFactory.java
    • -222
    • +0
    ./easy/json/parser/ValueFactory.java
    • -24
    • +13
    ./easy/json/parser/ValueListener.java
    • -17
    • +38
    ./easy/json/parser/ValueParser.java
  1. … 11 more files in changeset.
DRILL-7706: Implement Drill RDBMS Metastore

1. Fix issue with undeterministic execution of batch update / delete statements, now they will be executed in the same order as they were added.

2. Abstracted Metastore common test classes to be used by different Metastore implementations.

3. Added drill-metastore-override-example.conf with example of Drill Metastore configuration.

4. Replaced list of metadata types which are required to be passed during read / write operations with set to avoid possible duplicates.

5. Add RDBMS Metastore implementation, README.md and unit tests.

  1. … 77 more files in changeset.
DRILL-7620: Fix plugin mutability issues

A recent commit made the plugin registry more strict about

the rule that, once a plugin is registered, it must be

immutable. A flaw enforcing that rule in the UI put the

registry in an inconsistent state.

Also

* Registry-specific errors

* Push more operations from UI layer into registry

* Clean up semantics of "resolve" for plugins

* Add more unit tests

* Better handling of "bad" plugins

* Force plugin names to lower case

* Fix comparison bugs in some format plugins

    • -74
    • +359
    ./StoragePluginRegistryImpl.java
  1. … 87 more files in changeset.
DRILL-7617: Disabled plugins not showing in Web UI

Fixes a misunderstanding in prior PR: getConfigs()

should return both enabled and disabled configs.

  1. … 2 more files in changeset.
DRILL-7601: Shift column conversion to reader from scan framework

Allows the column writers to be generic, moves scan-specific

conversions into each reader where needed, implemented in

a reader-specific way.

Adds a revised way of handling projections in the result set

loader that is not coupled with conversion, as the prior

design was.

Updates the CSV, Avro, Log and HDF5 readers.

closes #1993

    • -0
    • +257
    ./avro/ColumnConverterFactory.java
    • -115
    • +0
    ./avro/ColumnConvertersUtil.java
    • -32
    • +27
    ./dfs/easy/EasyFormatPlugin.java
    • -7
    • +3
    ./easy/text/reader/BaseFieldOutput.java
    • -20
    • +107
    ./easy/text/reader/CompliantTextBatchReader.java
    • -0
    • +44
    ./easy/text/reader/ConstrainedFieldOutput.java
    • -7
    • +11
    ./easy/text/reader/FieldVarCharOutput.java
    • -53
    • +32
    ./easy/text/reader/TextParsingSettings.java
  1. … 206 more files in changeset.
DRILL-7477: Allow passing table function parameters into ANALYZE statement

- Fix logical dir pruning when table function is used

closes #2005

    • -34
    • +41
    ./dfs/WorkspaceSchemaFactory.java
  1. … 19 more files in changeset.
DRILL-7565: ANALYZE TABLE ... REFRESH METADATA does not work for empty Parquet files

- Fixed ConvertMetadataAggregateToDirectScanRule rule to distinguish array columns correctly and proceed using other parquet metadata if such columns are found.

- Added new implicit column which signalizes whether the empty result is obtained during collecting metadata and helps to distinguish real data results from metadata results.

- Updated scan to return row with metadata if the above implicit column is present.

- Added unit tests for checking the correctness of both optional and required columns from empty files.

closes #1985

    • -1
    • +1
    ./parquet/BaseParquetMetadataProvider.java
  1. … 12 more files in changeset.
DRILL-7590: Refactor plugin registry

Major cleanup of the plugin registry to split it into components

in preparation for a proper plugin API.

Better coordinates the named and ephemeral plugin caches.

Cleans up the registry API. Sharpens rules for modifying

plugin configs.

closes #1988

    • -0
    • +302
    ./ClassicConnectorLocator.java
    • -0
    • +100
    ./ConnectorHandle.java
    • -0
    • +157
    ./ConnectorLocator.java
    • -0
    • +91
    ./DrillSchemaFactory.java
    • -0
    • +67
    ./DrillbitPluginRegistryContext.java
    • -0
    • +60
    ./PluginBootstrapLoader.java
  1. … 149 more files in changeset.
DRILL-7634: Rollup of code cleanup changes

Collection of code cleanup changes. The most significant

is to create constants for function names.

closes #2020

    • -17
    • +20
    ./ischema/InfoSchemaFilterBuilder.java
    • -18
    • +17
    ./schedule/AssignmentCreator.java
  1. … 121 more files in changeset.
DRILL-7576: Fail fast for operator errors

Converts operators to fail with a UserException rather than using

the STOP iterator status. The result is clearer error messages

and simpler code.

closes #1975

    • -0
    • +60
    ./StatisticsRecordWriter.java
    • -12
    • +15
    ./easy/json/JsonStatisticsRecordWriter.java
  1. … 64 more files in changeset.
DRILL-7572: JSON structure parser

Provides a structure-driven parser for JSON. Provides an intermediate

level beteen the Jackson JSON parser and the "shims" that write to

Drill value vectors via the EVF.

close #1971

    • -0
    • +39
    ./easy/json/parser/AbstractElementParser.java
    • -0
    • +151
    ./easy/json/parser/ArrayListener.java
    • -0
    • +68
    ./easy/json/parser/ArrayParser.java
    • -0
    • +80
    ./easy/json/parser/DummyValueParser.java
    • -0
    • +39
    ./easy/json/parser/ElementParser.java
    • -0
    • +76
    ./easy/json/parser/ErrorFactory.java
    • -0
    • +54
    ./easy/json/parser/JsonStructureOptions.java
    • -0
    • +218
    ./easy/json/parser/JsonStructureParser.java
    • -0
    • +25
    ./easy/json/parser/JsonType.java
    • -0
    • +131
    ./easy/json/parser/ObjectListener.java
    • -0
    • +167
    ./easy/json/parser/ObjectParser.java
    • -0
    • +121
    ./easy/json/parser/RootParser.java
    • -0
    • +149
    ./easy/json/parser/TokenIterator.java
    • -0
    • +222
    ./easy/json/parser/ValueFactory.java
    • -0
    • +152
    ./easy/json/parser/ValueListener.java
  1. … 6 more files in changeset.
DRILL-5733: Unable to SELECT from parquet file with Hadoop 2.7.4

closes #1969

    • -12
    • +15
    ./parquet/ParquetTableMetadataProviderImpl.java
DRILL-7549: Fix validation error when querying absent sub folder in embedded mode

closes #1963

    • -102
    • +167
    ./sys/store/LocalPersistentStore.java
  1. … 2 more files in changeset.