Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7254: Read Hive union w/o nulls

    • -6
    • +4
    ./drill/exec/store/hive/HiveUtilities.java
    • -0
    • +44
    ./drill/exec/store/hive/writers/complex/HiveUnionWriter.java
  1. … 18 more files in changeset.
DRILL-7252: Read Hive map using Dict<K,V> vector

    • -2
    • +2
    ./drill/exec/store/hive/HiveUtilities.java
    • -0
    • +61
    ./drill/exec/store/hive/writers/complex/HiveMapWriter.java
  1. … 13 more files in changeset.
DRILL-7337: Add vararg UDFs support

    • -1
    • +1
    ./drill/exec/expr/fn/HiveFuncHolder.java
    • -4
    • +8
    ./drill/exec/expr/fn/HiveFunctionRegistry.java
  1. … 35 more files in changeset.
DRILL-7314: Use TupleMetadata instead of concrete implementation

1. Add ser / de implementation for TupleMetadata interface based on types.

2. Replace TupleSchema usage where possible.

3. Move patcher classes into commons.

4. Upgrade some dependencies and general refactoring.

  1. … 39 more files in changeset.
DRILL-7315: Revise precision and scale order in the method arguments

  1. … 28 more files in changeset.
DRILL-7307: casthigh for decimal type can lead to the issues with VarDecimalHolder

- Fixed code-gen for VarDecimal type

- Fixed code-gen issue with nullable holders for simple cast functions

with passed constants as arguments.

- Code-gen now honnoring DataType.Optional type defined by UDF for

NULL-IF-NULL functions.

    • -1
    • +1
    ./drill/exec/expr/fn/HiveFuncHolder.java
  1. … 9 more files in changeset.
DRILL-7313: Use Hive schema for MaprDB native reader when field was empty

- Added all_text_mode option for hive maprDB Json

- Improved logic to convert Hive's schema into Drill's one

- Added unit tests for schema conversion

    • -0
    • +100
    ./drill/exec/store/hive/HiveUtilities.java
  1. … 26 more files in changeset.
DRILL-7271: Refactor Metadata interfaces and classes to contain all needed information for the File based Metastore

  1. … 119 more files in changeset.
DRILL-7253: Read Hive struct w/o nulls

    • -3
    • +1
    ./drill/exec/store/hive/HiveUtilities.java
  1. … 15 more files in changeset.
DRILL-7268: Read Hive array with parquet native reader

1. Fixed preserving of group originalType for projected schema

in DrillParquetReader

2. Added reading of LIST logical type to DrillParquetGroupConverter.

Intermediate noop converter used to skip writing for next nested

repeated field after recognition of parent field as LIST. For this

skipRepeated 'true' passed to child converter's constructor.

close apache/drill#1805

    • -2
    • +1
    ./drill/exec/store/hive/HiveUtilities.java
  1. … 6 more files in changeset.
DRILL-7251: Read Hive array w/o nulls

1. HiveFieldConverter replaced by Hive writers for primitives

2. Created HiveValueWriterFactory and HiveListWriter to implement arrays support

4. Readers generation replaced by HiveDefaultRecordReader and HiveTextRecordReader

5. Few reader initializers replaced by one

6. Added method to repeated vardecimal writer

7. Minor fix for array column in View

    • -241
    • +0
    ./drill/exec/store/hive/HiveFieldConverter.java
    • -1
    • +1
    ./drill/exec/store/hive/HiveUtilities.java
    • -0
    • +125
    ./drill/exec/store/hive/readers/HiveTextRecordReader.java
    • -0
    • +127
    ./drill/exec/store/hive/readers/ReadersInitializer.java
    • -0
    • +38
    ./drill/exec/store/hive/writers/HiveValueWriter.java
  1. … 39 more files in changeset.
DRILL-7196: Queries are still runnable on disabled plugins

- Storage client is not created anymore for disabled plugins

- GET "/storage/{name}.json" endpoint now working with

plugin configuration directly, without client instantination.

It have increased UI responsitivity.

- Hbase and mongo base test classes refactored to honor enabled

plugin attribute

- Fixed path contructor for mongo test datasets:

Now it is cross-platform

- Fixed test json files format which using plugin definitions

- Code cleanup

  1. … 106 more files in changeset.
DRILL-7062: Initial implementation of run-time rowgroup pruning closes #1738

  1. … 23 more files in changeset.
DRILL-7115: Improve Hive schema show tables performance

1. To make SHOW TABLES for Hive schema work much faster, additional Drill

feature of showing only accesible tables when Storage-Based authorization

is enabled was sacrificed. Now the behaviour matches to Hive/Beeline, all

tables will be shown despite of accessibility. For details about previous

show tables results, check description of DRILL-540.

2. In HiveDatabaseSchema implemented faster getTableNamesAndTypes() method

and removed bulk related code.

3. Deprecated bulk related options and removed bulk code from AbstractSchema,

DrillHiveMetastoreClient.

4. For 8000 Hive tables query returned in 1.8 seconds, for combination of

4000 tables and 8000 views query returned in 2.3 seconds. Note, that

after first query table names will be cached and next queries will perform

in less than 1 sec.

5. Refactored WorkspaceSchemaFactory's getTableNamesAndTypes()

method to reuse existing getViews() method.

6. DrillHiveMetastoreClient was refactored. Classes were unnested and enclosed

within client package with restricted visibility. Also was updated cache

values type to avoid unnecessarry List to Set back and forth conversions.

Client creation methods moved to separate class. So the new package

exposes only factory and client class.

closes #1706

    • -586
    • +0
    ./drill/exec/store/hive/DrillHiveMetaStoreClient.java
    • -0
    • +16
    ./drill/exec/store/hive/HiveUtilities.java
    • -0
    • +106
    ./drill/exec/store/hive/client/DrillHiveMetaStoreClient.java
    • -0
    • +110
    ./drill/exec/store/hive/client/DrillHiveMetaStoreClientWithAuthorization.java
    • -0
    • +157
    ./drill/exec/store/hive/client/HiveAuthorizationHelper.java
    • -0
    • +172
    ./drill/exec/store/hive/client/HiveMetadataCache.java
    • -0
    • +111
    ./drill/exec/store/hive/client/TableEntryCacheLoader.java
    • -0
    • +72
    ./drill/exec/store/hive/client/TableName.java
    • -0
    • +94
    ./drill/exec/store/hive/client/TableNameCacheLoader.java
  1. … 6 more files in changeset.
DRILL-7089: Implement caching for TableMetadataProvider at query level and adapt statistics to use Drill metastore API

closes #1728

  1. … 49 more files in changeset.
DRILL-6852: Adapt current Parquet Metadata cache implementation to use Drill Metastore API

Co-authored-by: Volodymyr Vysotskyi <vvovyk@gmail.com>

Co-authored-by: Vitalii Diravka <vitalii@apache.org>

close apache/drill#1646

    • -0
    • +104
    ./drill/exec/store/hive/HiveParquetTableMetadataProvider.java
  1. … 65 more files in changeset.
DRILL-6927: Avoid double conversion from impala timestamp when hive native parquet reader is used closes #1655

  1. … 1 more file in changeset.
DRILL-5603: Replace String file paths to Hadoop Path - replaced all String path representation with org.apache.hadoop.fs.Path - added PathSerDe.Se JSON serializer - refactoring of DFSPartitionLocation code by leveraging existing listPartitionValues() functionality

closes #1657

    • -5
    • +6
    ./drill/exec/store/hive/HivePartitionHolder.java
  1. … 78 more files in changeset.
DRILL-7019: Add check for redundant imports

close apache/drill#1629

  1. … 23 more files in changeset.
DRILL-6944: UnsupportedOperationException thrown for view over MapR-DB binary table

1. Added persistence of MAP key and value types in Drill views (affects .view.drill file) for avoiding cast problems in future.

2. Preserved backward compatibility of older view files by treating untyped maps as ANY.

closes #1602

  1. … 5 more files in changeset.
DRILL-6929: Exclude maprfs jar for default profile closes #1586

    • -10
    • +24
    ./drill/exec/store/hive/HiveStoragePlugin.java
  1. … 6 more files in changeset.
DRILL-6850: Force setting DRILL_LOGICAL Convention for DrillRelFactories and DrillFilterRel

- Fix workspace case insensitivity for JDBC storage plugin

    • -11
    • +10
    ./drill/exec/store/hive/schema/HiveSchemaFactory.java
  1. … 13 more files in changeset.
DRILL-4456: Add Hive translate UDF

closes #1527

    • -15
    • +30
    ./drill/exec/expr/fn/HiveFunctionRegistry.java
  1. … 1 more file in changeset.
DRILL-6744: Support varchar and decimal push down

1. Added enableStringsSignedMinMax parquet format plugin config and store.parquet.reader.strings_signed_min_max session option to control reading binary statistics for files generated by prior versions of Parquet 1.10.0.

2. Added ParquetReaderConfig to store configuration needed during reading parquet statistics or files.

3. Provided mechanism to enable varchar / decimal filter push down.

4. Added VersionUtil to compare Drill versions in string representation.

5. Added appropriate unit tests.

closes #1537

  1. … 39 more files in changeset.
DRILL-6793: FragmentExecutor cannot send its final state for the case when RootExec root wasn't initialized

closes #1506

  1. … 5 more files in changeset.
DRILL-540: Allow querying hive views in Drill

1. Added DrillHiveViewTable which allows construction of DrillViewTable based

on Hive metadata

2. Added initialization of DrillHiveViewTable in HiveSchemaFactory

3. Extracted conversion of Hive data types from DrillHiveTable

to HiveToRelDataTypeConverter

4. Removed throwing of UnsupportedOperationException from HiveStoragePlugin

5. Added TestHiveViewsSupport and authorization tests

6. Added closeSilently() method to AutoCloseables

closes #1559

    • -0
    • +158
    ./drill/exec/planner/types/HiveToRelDataTypeConverter.java
    • -13
    • +22
    ./drill/exec/store/hive/ColumnListsCache.java
    • -11
    • +5
    ./drill/exec/store/hive/HiveStoragePlugin.java
    • -151
    • +15
    ./drill/exec/store/hive/schema/DrillHiveTable.java
    • -23
    • +23
    ./drill/exec/store/hive/schema/HiveSchemaFactory.java
  1. … 10 more files in changeset.
DRILL-6724: Dump operator context to logs when error occurs during query execution

closes #1455

  1. … 102 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

    • -2
    • +2
    ./drill/exec/expr/fn/HiveFunctionRegistry.java
    • -3
    • +3
    ./drill/exec/store/hive/ColumnListsCache.java
    • -1
    • +1
    ./drill/exec/store/hive/HiveFieldConverter.java
    • -1
    • +1
    ./drill/exec/store/hive/HiveReadEntry.java
    • -1
    • +1
    ./drill/exec/store/hive/HiveStoragePlugin.java
    • -3
    • +3
    ./drill/exec/store/hive/HiveSubScan.java
    • -1
    • +1
    ./drill/exec/store/hive/HiveTableWrapper.java
  1. … 970 more files in changeset.
DRILL-6492: Ensure schema / workspace case insensitivity in Drill

1. StoragePluginsRegistryImpl was updated:

a. for backward compatibility at init to convert all existing storage plugins names to lower case, in case of duplicates, to log warning and skip the duplicate.

b. to wrap persistent plugins registry into case insensitive store wrapper (CaseInsensitivePersistentStore) to ensure all given keys are converted into lower case when performing insert, update, delete, search operations.

c. to load system storage plugins dynamically by @SystemStorage annotation.

2. StoragePlugins class was updated to stored storage plugins configs by name in case insensitive map.

3. SchemaUtilities.searchSchemaTree method was updated to convert all schema names into lower case to ensure that are they are matched case insensitively (all schemas are stored in Drill in lower case).

4. FileSystemConfig was updated to store workspaces by name in case insensitive hash map.

5. All plugins schema factories are now extend AbstractSchemaFactory to ensure that given schema name is converted to lower case.

6. New method areTableNamesAreCaseInsensitive was added to AbstractSchema to indicate if schema tables names are case insensitive. By default, false. Schema implementation is responsible for table names case insensitive search in case it supports one. Currently, information_schema, sys and hive do so.

7. System storage plugins (information_schema, sys) were refactored to ensure their schema, table names are case insensitive, also the annotation @SystemPlugin and additional constructor were added to allow dynamically load system plugins at storage plugin registry during init phase.

8. MetadataProvider was updated to concert all schema filter conditions into lower case to ensure schema would be matched case insensitively.

9. ShowSchemasHandler, ShowTablesHandler, DescribeTableHandler were updated to ensure schema / tables names (this depends if schema supports case insensitive table names) would be found case insensitively.

git closes #1439

    • -13
    • +16
    ./drill/exec/store/hive/schema/HiveSchemaFactory.java
  1. … 54 more files in changeset.
DRILL-6656: Disallow extra semicolons and multiple statements on the same line.

closes #1415

    • -1
    • +1
    ./drill/exec/store/hive/HiveUtilities.java
  1. … 143 more files in changeset.