Clone Tools
  • last updated 25 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6929: Exclude maprfs jar for default profile closes #1586

    • -180
    • +0
    ./planner/sql/logical/ConvertHiveMapRDBJsonScanToDrillMapRDBJsonScan.java
    • -10
    • +24
    ./store/hive/HiveStoragePlugin.java
  1. … 6 more files in changeset.
DRILL-6850: Force setting DRILL_LOGICAL Convention for DrillRelFactories and DrillFilterRel

- Fix workspace case insensitivity for JDBC storage plugin

    • -11
    • +10
    ./store/hive/schema/HiveSchemaFactory.java
  1. … 13 more files in changeset.
DRILL-4456: Add Hive translate UDF

closes #1527

    • -15
    • +30
    ./expr/fn/HiveFunctionRegistry.java
  1. … 1 more file in changeset.
DRILL-6744: Support varchar and decimal push down

1. Added enableStringsSignedMinMax parquet format plugin config and store.parquet.reader.strings_signed_min_max session option to control reading binary statistics for files generated by prior versions of Parquet 1.10.0.

2. Added ParquetReaderConfig to store configuration needed during reading parquet statistics or files.

3. Provided mechanism to enable varchar / decimal filter push down.

4. Added VersionUtil to compare Drill versions in string representation.

5. Added appropriate unit tests.

closes #1537

    • -3
    • +7
    ./planner/sql/logical/ConvertHiveParquetScanToDrillParquetScan.java
    • -8
    • +9
    ./store/hive/HiveDrillNativeParquetRowGroupScan.java
    • -8
    • +11
    ./store/hive/HiveDrillNativeParquetScan.java
  1. … 39 more files in changeset.
DRILL-6793: FragmentExecutor cannot send its final state for the case when RootExec root wasn't initialized

closes #1506

  1. … 5 more files in changeset.
DRILL-540: Allow querying hive views in Drill

1. Added DrillHiveViewTable which allows construction of DrillViewTable based

on Hive metadata

2. Added initialization of DrillHiveViewTable in HiveSchemaFactory

3. Extracted conversion of Hive data types from DrillHiveTable

to HiveToRelDataTypeConverter

4. Removed throwing of UnsupportedOperationException from HiveStoragePlugin

5. Added TestHiveViewsSupport and authorization tests

6. Added closeSilently() method to AutoCloseables

closes #1559

    • -0
    • +158
    ./planner/types/HiveToRelDataTypeConverter.java
    • -13
    • +22
    ./store/hive/ColumnListsCache.java
    • -11
    • +5
    ./store/hive/HiveStoragePlugin.java
    • -151
    • +15
    ./store/hive/schema/DrillHiveTable.java
    • -12
    • +77
    ./store/hive/schema/DrillHiveViewTable.java
    • -23
    • +23
    ./store/hive/schema/HiveSchemaFactory.java
  1. … 10 more files in changeset.
DRILL-6724: Dump operator context to logs when error occurs during query execution

closes #1455

  1. … 102 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

    • -1
    • +1
    ./planner/sql/HivePartitionDescriptor.java
    • -1
    • +1
    ./planner/sql/HivePartitionLocation.java
    • -6
    • +6
    ./store/hive/DrillHiveMetaStoreClient.java
    • -1
    • +1
    ./store/hive/HiveAuthorizationHelper.java
    • -1
    • +1
    ./store/hive/HiveDrillNativeParquetRowGroupScan.java
    • -1
    • +1
    ./store/hive/HiveDrillNativeParquetScan.java
    • -1
    • +1
    ./store/hive/HiveDrillNativeParquetScanBatchCreator.java
    • -7
    • +7
    ./store/hive/HiveMetadataProvider.java
  1. … 970 more files in changeset.
DRILL-6492: Ensure schema / workspace case insensitivity in Drill

1. StoragePluginsRegistryImpl was updated:

a. for backward compatibility at init to convert all existing storage plugins names to lower case, in case of duplicates, to log warning and skip the duplicate.

b. to wrap persistent plugins registry into case insensitive store wrapper (CaseInsensitivePersistentStore) to ensure all given keys are converted into lower case when performing insert, update, delete, search operations.

c. to load system storage plugins dynamically by @SystemStorage annotation.

2. StoragePlugins class was updated to stored storage plugins configs by name in case insensitive map.

3. SchemaUtilities.searchSchemaTree method was updated to convert all schema names into lower case to ensure that are they are matched case insensitively (all schemas are stored in Drill in lower case).

4. FileSystemConfig was updated to store workspaces by name in case insensitive hash map.

5. All plugins schema factories are now extend AbstractSchemaFactory to ensure that given schema name is converted to lower case.

6. New method areTableNamesAreCaseInsensitive was added to AbstractSchema to indicate if schema tables names are case insensitive. By default, false. Schema implementation is responsible for table names case insensitive search in case it supports one. Currently, information_schema, sys and hive do so.

7. System storage plugins (information_schema, sys) were refactored to ensure their schema, table names are case insensitive, also the annotation @SystemPlugin and additional constructor were added to allow dynamically load system plugins at storage plugin registry during init phase.

8. MetadataProvider was updated to concert all schema filter conditions into lower case to ensure schema would be matched case insensitively.

9. ShowSchemasHandler, ShowTablesHandler, DescribeTableHandler were updated to ensure schema / tables names (this depends if schema supports case insensitive table names) would be found case insensitively.

git closes #1439

    • -13
    • +16
    ./store/hive/schema/HiveSchemaFactory.java
  1. … 54 more files in changeset.
DRILL-6656: Disallow extra semicolons and multiple statements on the same line.

closes #1415

  1. … 143 more files in changeset.
DRILL-6614: Allow usage of MapRDBFormatPlugin for HiveStoragePlugin

    • -8
    • +2
    ./planner/sql/logical/ConvertHiveMapRDBJsonScanToDrillMapRDBJsonScan.java
    • -0
    • +21
    ./store/hive/HiveStoragePlugin.java
  1. … 5 more files in changeset.
DRILL-6575: Add store.hive.conf.properties option to allow set Hive properties at session level

closes #1365

    • -3
    • +3
    ./planner/sql/logical/ConvertHiveMapRDBJsonScanToDrillMapRDBJsonScan.java
    • -10
    • +11
    ./planner/sql/logical/ConvertHiveParquetScanToDrillParquetScan.java
    • -3
    • +14
    ./store/hive/HiveDrillNativeParquetRowGroupScan.java
    • -4
    • +16
    ./store/hive/HiveDrillNativeParquetScan.java
    • -16
    • +41
    ./store/hive/HiveStoragePlugin.java
    • -9
    • +12
    ./store/hive/HiveStoragePluginConfig.java
  1. … 13 more files in changeset.
DRILL-6494: Drill Plugins Handler

- Storage Plugins Handler service is used op the Drill start-up stage and it updates storage plugins configs from

storage-plugins-override.conf file. If plugins configs are present in the persistence store - they are updated,

otherwise bootstrap plugins are updated and the result configs are loaded to persistence store. If the enabled

status is absent in the storage-plugins-override.conf file, the last plugin config enabled status persists.

- 'drill.exec.storage.action_on_plugins_override_file' Boot option is added. This is the action, which should be

performed on the storage-plugins-override.conf file after successful updating storage plugins configs.

Possible values are: "none" (default), "rename" and "remove".

- The "NULL" issue with updating Hive plugin config by REST is solved. But clients are still being instantiated for disabled

plugins - DRILL-6412.

- "org.honton.chas.hocon:jackson-dataformat-hocon" library is added for the proper deserializing HOCON conf file

- additional refactoring: "com.typesafe:config" and "org.apache.commons:commons-lang3" are placed into DependencyManagement

block with proper versions; correct properties for metrics in "drill-override-example.conf" are specified

closes #1345

    • -7
    • +4
    ./store/hive/schema/HiveSchemaFactory.java
  1. … 34 more files in changeset.
DRILL-6557: Use size in bytes during Hive statistics calculation if present

1. Check size in bytes presence in stats before fetching input splits and use it if present.

2. Add log trace suggesting to use ANALYZE command before running queries if statistics is unavailable and Drill had to fetch all input splits.

3. Minor refactoring / cleanup in HiveMetadataProvider class.

closes #1357

    • -109
    • +123
    ./store/hive/HiveMetadataProvider.java
DRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table

closes #1314

    • -0
    • +186
    ./planner/sql/logical/ConvertHiveMapRDBJsonScanToDrillMapRDBJsonScan.java
    • -111
    • +5
    ./planner/sql/logical/ConvertHiveParquetScanToDrillParquetScan.java
    • -1
    • +1
    ./store/hive/HiveMetadataProvider.java
    • -5
    • +11
    ./store/hive/HiveStoragePlugin.java
    • -24
    • +125
    ./store/hive/HiveUtilities.java
  1. … 11 more files in changeset.
DRILL-6473: Update MapR Hive

close apache/drill#1307

  1. … 2 more files in changeset.
DRILL-6421: Refactor DecimalUtility and CoreDecimalUtility classes

closes #1267

  1. … 13 more files in changeset.
DRILL-6386: Remove unused imports and star imports.

    • -1
    • +0
    ./planner/sql/HivePartitionLocation.java
    • -1
    • +0
    ./store/hive/HiveAuthorizationHelper.java
    • -1
    • +0
    ./store/hive/schema/DrillHiveTable.java
  1. … 224 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

    • -1
    • +1
    ./expr/fn/impl/hive/DrillDeferredObject.java
    • -1
    • +1
    ./planner/sql/HivePartitionLocation.java
    • -1
    • +1
    ./planner/sql/HiveUDFOperatorWithoutInference.java
    • -15
    • +16
    ./store/hive/ColumnListsCache.java
    • -1
    • +1
    ./store/hive/HiveAuthorizationHelper.java
    • -1
    • +1
    ./store/hive/HiveStoragePluginConfig.java
    • -15
    • +16
    ./store/hive/HiveTableWithColumnCache.java
  1. … 2052 more files in changeset.
DRILL-6094: Decimal data type enhancements

Add ExprVisitors for VARDECIMAL

Modify writers/readers to support VARDECIMAL

- Added usage of VarDecimal for parquet, hive, maprdb, jdbc;

- Added options to store decimals as int32 and int64 or fixed_len_byte_array or binary;

Add UDFs for VARDECIMAL data type

- modify type inference rules

- remove UDFs for obsolete DECIMAL types

Enable DECIMAL data type by default

Add unit tests for DECIMAL data type

Fix mapping for NLJ when literal with non-primitive type is used in join conditions

Refresh protobuf C++ source files

Changes in C++ files

Add support for decimal logical type in Avro.

Add support for date, time and timestamp logical types.

Update Avro version to 1.8.2.

    • -98
    • +14
    ./store/hive/HiveFieldConverter.java
  1. … 198 more files in changeset.
DRILL-6375 : Support for ANY_VALUE aggregate function

closes #1256

  1. … 36 more files in changeset.
DRILL-6331: Revisit Hive Drill native parquet implementation to be exposed to Drill optimizations (filter / limit push down, count to direct scan)

1. Factored out common logic for Drill parquet reader and Hive Drill native parquet readers: AbstractParquetGroupScan, AbstractParquetRowGroupScan, AbstractParquetScanBatchCreator.

2. Rules that worked previously only with ParquetGroupScan, now can be applied for any class that extends AbstractParquetGroupScan: DrillFilterItemStarReWriterRule, ParquetPruneScanRule, PruneScanRule.

3. Hive populated partition values based on information returned from Hive metastore. Drill populates partition values based on path difference between selection root and actual file path.

Before ColumnExplorer populated partition values based on Drill approach. Since now ColumnExplorer populates values for parquet files from Hive tables,

`populateImplicitColumns` method logic was changed to populated partition columns only based on given partition values.

4. Refactored ParquetPartitionDescriptor to be responsible for populating partition values rather than storing this logic in parquet group scan class.

5. Metadata class was moved to separate metadata package (org.apache.drill.exec.store.parquet.metadata). Factored out several inner classed to improve code readability.

6. Collected all Drill native parquet reader unit tests into one class TestHiveDrillNativeParquetReader, also added new tests to cover new functionality.

7. Reduced excessive logging when parquet files metadata is read

closes #1214

    • -17
    • +40
    ./planner/sql/logical/ConvertHiveParquetScanToDrillParquetScan.java
    • -0
    • +131
    ./store/hive/HiveDrillNativeParquetRowGroupScan.java
    • -47
    • +157
    ./store/hive/HiveDrillNativeParquetScan.java
    • -0
    • +79
    ./store/hive/HiveDrillNativeParquetScanBatchCreator.java
    • -55
    • +0
    ./store/hive/HiveDrillNativeParquetSubScan.java
    • -208
    • +0
    ./store/hive/HiveDrillNativeScanBatchCreator.java
    • -1
    • +4
    ./store/hive/HiveMetadataProvider.java
    • -0
    • +95
    ./store/hive/HivePartitionHolder.java
  1. … 57 more files in changeset.
DRILL-6381: (Part 3) Planner and Execution implementation to support Secondary Indexes

  1. Index Planning Rules and Plan generators

    - DbScanToIndexScanRule: Top level physical planning rule that drives index planning for several relational algebra patterns.

- DbScanSortRemovalRule: Physical planning rule for index planning for Sort-based operations.

    - Plan Generators: Covering, Non-Covering and Intersect physical plan generators.

    - Support planning with functional indexes such as CAST functions.

    - Enhance PlannerSettings with several configuration options for indexes.

  2. Index Selection and Statistics

    - An IndexSelector that support cost-based index selection of covering and non-covering indexes using statistics and collation properties.

    - Costing of index intersection for comparison with single-index plans.

  3. Planning and execution operators

    - Support RangePartitioning physical operator during query planning and execution.

    - Support RowKeyJoin physical operator during query planning and execution.

    - HashTable and HashJoin changes to support RowKeyJoin and Index Intersection.

    - Enhance Materializer to keep track of subscan association with a particular rowkey join.

  4. Index Planning utilities

    - Utility classes to perform RexNode analysis, including conversion to and from SchemaPath.

    - Utility class to analyze filter condition and an input collation to determine output collation.

    - Helper classes to maintain index contexts for logical and physical planning phase.

    - IndexPlanUtils utility class for various helper methods.

  5. Miscellaneous

    - Separate physical rel for DirectScan.

    - Modify LimitExchangeTranspose rule to handle SingleMergeExchange.

- MD-3880: Return correct status from RangePartitionRecordBatch setupNewSchema

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashJoinPOP.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillMergeProjectRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushProjectIntoScanRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScanRel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTrait.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PrelUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java

exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetPushDownFilter.java

exec/java-exec/src/main/resources/drill-module.conf

logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java

Resolve merge comflicts and compilation issues.

    • -1
    • +1
    ./planner/sql/logical/ConvertHiveMapRDBJsonScanToDrillMapRDBJsonScan.java
  1. … 93 more files in changeset.
DRILL-6145: Implement Hive MapR-DB JSON handler

closes #1158

  1. … 3 more files in changeset.
DRILL-6204: Pass tables columns without partition columns to empty Hive reader

closes #1146

    • -5
    • +5
    ./store/hive/HiveDrillNativeScanBatchCreator.java
DRILL-6195: Quering Hive non-partitioned transactional tables via Drill

closes #1140

DRILL-6164: Heap memory leak during parquet scan and OOM

closes #1122

    • -1
    • +2
    ./store/hive/HiveDrillNativeScanBatchCreator.java
  1. … 14 more files in changeset.
DRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin

closes #1282

    • -14
    • +4
    ./store/hive/HiveStoragePlugin.java
  1. … 11 more files in changeset.
DRILL-6130: Fix NPE during physical plan submission for various storage plugins

1. Fixed ser / de issues for Hive, Kafka, Hbase plugins.

2. Added physical plan submission unit test for all storage plugins in contrib module.

3. Refactoring.

closes #1108

    • -7
    • +7
    ./planner/sql/HivePartitionDescriptor.java
    • -7
    • +7
    ./planner/sql/logical/ConvertHiveParquetScanToDrillParquetScan.java
    • -11
    • +11
    ./store/hive/HiveDrillNativeParquetScan.java
    • -2
    • +2
    ./store/hive/HiveDrillNativeParquetSubScan.java
  1. … 21 more files in changeset.
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

    • -3
    • +4
    ./store/hive/HiveDrillNativeScanBatchCreator.java
    • -3
    • +3
    ./store/hive/HiveScanBatchCreator.java
  1. … 220 more files in changeset.