Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

1. Upgraded Iceberg version and removed unneeded code for In / NotIn Expressions.

2. Updated Metastore Read / Modify interfaces to support required metadata types:

a. introduced abstract Read / Modify classes with boilerplate code;

b. added delete operation with filter and metadata type;

c. added metadata type validator which checks supported metadata types for each component;

d. made purge operation terminal;

e. made necessary changes in REAME.md files.

3. Added / updated unit tests.

closes #2042

    • -0
    • +77
    ./MetastoreColumn.java
    • -63
    • +67
    ./components/tables/BasicTablesRequests.java
    • -60
    • +63
    ./components/tables/TableMetadataUnit.java
    • -0
    • +46
    ./components/tables/TablesMetadataTypeValidator.java
    • -24
    • +26
    ./expressions/FilterExpression.java
    • -11
    • +13
    ./expressions/ListPredicate.java
    • -19
    • +21
    ./expressions/SimplePredicate.java
    • -0
    • +54
    ./operate/AbstractModify.java
    • -0
    • +71
    ./operate/AbstractRead.java
    • -0
    • +88
    ./operate/Delete.java
    • -0
    • +60
    ./operate/MetadataTypeValidator.java
  1. … 27 more files in changeset.
DRILL-7706: Implement Drill RDBMS Metastore

1. Fix issue with undeterministic execution of batch update / delete statements, now they will be executed in the same order as they were added.

2. Abstracted Metastore common test classes to be used by different Metastore implementations.

3. Added drill-metastore-override-example.conf with example of Drill Metastore configuration.

4. Replaced list of metadata types which are required to be passed during read / write operations with set to avoid possible duplicates.

5. Add RDBMS Metastore implementation, README.md and unit tests.

    • -8
    • +11
    ./components/tables/BasicTablesRequests.java
    • -0
    • +2
    ./components/tables/TableMetadataUnit.java
    • -4
    • +4
    ./components/tables/TablesMetadataTypeValidator.java
    • -1
    • +6
    ./config/MetastoreConfigConstants.java
  1. … 66 more files in changeset.
DRILL-7527: DROP METADATA doesn't work with table name starting with '/' inside workspace

closes #1958

  1. … 10 more files in changeset.
DRILL-7509: Incorrect TupleSchema is created for DICT column when querying Parquet files

  1. … 15 more files in changeset.
DRILL-7481: Fix raw type warnings in Iceberg Metastore and related classes

closes #1924

  1. … 4 more files in changeset.
DRILL-7479: Partial fixes for metadata parameterized type issues

See DRILL-7479 and DRILL-7480 for an explanation. Adds generic

type parameters where needed to avoid the need to supporess

warnings. However, type parameters are probably not needed

at all and should be removed in the future for reasons explained

in DRILL-7480.

closes #1923

    • -0
    • +3
    ./components/tables/BasicTablesTransformer.java
    • -0
    • +2
    ./components/tables/TableMetadataUnit.java
    • -7
    • +6
    ./metadata/NonInterestingColumnsMetadata.java
    • -1
    • +1
    ./statistics/CollectableColumnStatisticsKind.java
    • -12
    • +18
    ./statistics/ColumnStatistics.java
    • -14
    • +17
    ./statistics/ColumnStatisticsKind.java
  1. … 25 more files in changeset.
DRILL-7471: DESCRIBE TABLE command fails with ClassCastException when Metastore is enabled

    • -1
    • +19
    ./statistics/ColumnStatisticsKind.java
    • -0
    • +1
    ./statistics/ExactStatisticsConstants.java
  1. … 7 more files in changeset.
DRILL-7359: Add support for DICT type in RowSet Framework

closes #1870

  1. … 82 more files in changeset.
DRILL-7357: Expose Drill Metastore data through information_schema

1. Add additional columns to TABLES and COLUMNS tables.

2. Add PARTITIONS table.

3. General refactoring to adjust information_schema data retrieval from multiple sources.

closes #1860

  1. … 32 more files in changeset.
DRILL-7331: Drill Iceberg Metastore metadata expiration

closes #1831

  1. … 14 more files in changeset.
DRILL-7313: Use Hive schema for MaprDB native reader when field was empty

- Added all_text_mode option for hive maprDB Json

- Improved logic to convert Hive's schema into Drill's one

- Added unit tests for schema conversion

  1. … 27 more files in changeset.
DRILL-7310: Move schema-related classes from exec module to be able to use them in metastore module

closes #1816

  1. … 102 more files in changeset.
DRILL-7273: Introduce operators for handling metadata

closes #1886

    • -0
    • +179
    ./components/tables/BasicTablesRequests.java
    • -0
    • +29
    ./statistics/BaseStatisticsKind.java
  1. … 142 more files in changeset.
DRILL-7271: Refactor Metadata interfaces and classes to contain all needed information for the File based Metastore

    • -36
    • +0
    ./CollectableColumnStatisticsKind.java
    • -36
    • +0
    ./CollectableTableStatisticsKind.java
    • -74
    • +0
    ./NonInterestingColumnsMetadata.java
    • -0
    • +148
    ./metadata/BaseMetadata.java
    • -0
    • +148
    ./metadata/BaseTableMetadata.java
  1. … 105 more files in changeset.
DRILL-7272: Drill Metastore Read / Write API and Drill Iceberg Metastore implementation

1. Drill Metastore Read / Write API.

2. Drill Iceberg Metastore implementation in iceberg-metastore module.

3. Patches Guava Preconditions class for Apache Iceberg.

4. General refactoring.

5. Unit tests.

6. Documentation.

    • -0
    • +43
    ./MetastoreFieldDefinition.java
    • -0
    • +107
    ./MetastoreRegistry.java
    • -0
    • +657
    ./components/tables/BasicTablesRequests.java
    • -0
    • +147
    ./components/tables/BasicTablesTransformer.java
    • -0
    • +118
    ./components/tables/MetastoreTableInfo.java
    • -0
    • +561
    ./components/tables/TableMetadataUnit.java
    • -0
    • +35
    ./components/tables/Tables.java
    • -0
    • +27
    ./components/views/ViewMetadataUnit.java
    • -0
    • +27
    ./components/views/Views.java
    • -0
    • +57
    ./config/MetastoreConfigConstants.java
    • -0
    • +46
    ./config/MetastoreConfigFileInfo.java
    • -0
    • +33
    ./exceptions/MetastoreException.java
    • -0
    • +90
    ./expressions/DoubleExpressionPredicate.java
    • -0
    • +146
    ./expressions/FilterExpression.java
  1. … 83 more files in changeset.
DRILL-7199: Optimize population of metadata for non-interesting columns closes #1771

    • -0
    • +74
    ./NonInterestingColumnsMetadata.java
  1. … 7 more files in changeset.
DRILL-7187: Improve selectivity estimation of BETWEEN predicates and arbitrary combination of range predicates.

Address review comments.

Modify unit test expected rowcount after rebasing.

close apache/drill#1772

  1. … 5 more files in changeset.
DRILL-7098: File Metadata Metastore Plugin closes #1754

    • -0
    • +36
    ./CollectableColumnStatisticsKind.java
    • -0
    • +36
    ./CollectableTableStatisticsKind.java
    • -0
    • +82
    ./ColumnStatistics.java
    • -0
    • +83
    ./ColumnStatisticsImpl.java
    • -0
    • +167
    ./ColumnStatisticsKind.java
    • -0
    • +33
    ./LocationProvider.java
    • -0
    • +135
    ./PartitionMetadata.java
    • -0
    • +118
    ./RowGroupMetadata.java
    • -0
    • +46
    ./StatisticsKind.java
    • -0
    • +35
    ./TableMetadata.java
    • -0
    • +122
    ./TableStatisticsKind.java
  1. … 47 more files in changeset.
DRILL-7096: Develop vector for canonical Map<K,V>

- Added new type DICT;

- Created value vectors for the type for single and repeated modes;

- Implemented corresponding FieldReaders and FieldWriters;

- Made changes in EvaluationVisitor to be able to read values from the map by key;

- Made changes to DrillParquetGroupConverter to be able to read Parquet's MAP type;

- Added an option `store.parquet.reader.enable_map_support` to disable reading MAP type as DICT from Parquet files;

- Updated AvroRecordReader to use new DICT type for Avro's MAP;

- Added support of the new type to ParquetRecordWriter.

  1. … 108 more files in changeset.
DRILL-7148: Use improved join cardinality and ndv estimation with statistics

closes #1744

  1. … 11 more files in changeset.