Clone
 

arina ielchiieva <arina.yelchiyeva@gmail.com> in drill

DRILL-7368: Fix Iceberg Metastore failure when filter column contains nulls

DRILL-7367: Remove Server details from response headers

closes #1851

DRILL-7168: Implement ALTER SCHEMA ADD / REMOVE commands

DRILL-7360: Refactor WatchService in Drillbit class and fix concurrency issues

DRILL-7339: Iceberg commit upgrade and Metastore tests categorization

1. Upgraded Iceberg commit to fix issue with deletes in transaction

2. Categorize Metastore tests

closes #1842

DRILL-4517: Support reading empty Parquet files

1. Modified flat and complex parquet readers to output schema only when requested number of records to read is 0. In this case readers are not initialized to improve performance.

2. Allowed reading requested number of rows instead of all rows in the row group (DRILL-6528).

3. Fixed issue with nulls number determination in the row group (fixed IsPredicate#isAllNulls method).

4. Allowed reading empty parquet files via adding empty / fake row group.

5. General refactoring and unit tests.

6. Parquet tests categorization.

closes #1839

    • binary
    /contrib/storage-hive/core/src/test/resources/empty.parquet
  1. … 34 more files in changeset.
DRILL-7335: Fix error when reading csv file with headers only

closes #1834

DRILL-7334: Update Iceberg Metastore Parquet write mode

closes #1832

    • -9
    • +10
    /metastore/iceberg-metastore/README.md
DRILL-7332: Allow parsing empty schema

closes #1828

DRILL-7331: Drill Iceberg Metastore metadata expiration

closes #1831

    • -7
    • +18
    /metastore/iceberg-metastore/README.md
DRILL-7314: Use TupleMetadata instead of concrete implementation

1. Add ser / de implementation for TupleMetadata interface based on types.

2. Replace TupleSchema usage where possible.

3. Move patcher classes into commons.

4. Upgrade some dependencies and general refactoring.

  1. … 26 more files in changeset.
DRILL-7272: Drill Metastore Read / Write API and Drill Iceberg Metastore implementation

1. Drill Metastore Read / Write API.

2. Drill Iceberg Metastore implementation in iceberg-metastore module.

3. Patches Guava Preconditions class for Apache Iceberg.

4. General refactoring.

5. Unit tests.

6. Documentation.

    • -0
    • +3
    /distribution/src/assemble/component.xml
    • -0
    • +178
    /metastore/iceberg-metastore/README.md
    • -0
    • +188
    /metastore/iceberg-metastore/pom.xml
  1. … 83 more files in changeset.
DRILL-7236: SqlLine 1.8 upgrade

closes #1804

DRILL-7186: Add back storage.json REST endpoint (regression after DRILL-6562)

DRILL-7159: Fix typeString method to return correct name for MAP (aka STRUCT) closes #1741

DRILL-7157: Wrap SchemaParsingException into UserException when creating schema

closes #1740

DRILL-7142: Add space after > in SqlLine prompt

closes #1721

DRILL-7138: Implement command to describe schema for table

closes #1719

DRILL-7124: Fix logger for ShowFilesHandler

closes #1705

DRILL-6965: Implement schema table function parameter

1. Added common schema table function parameter with can be used as single unit or with format plugin table function parameters.

2. Allowed creating schema without columns, in case if user needs only to indicate table properties.

3. Added unit tests.

closes #1777

  1. … 16 more files in changeset.
DRILL-7111: Fix table function execution for directories

closes #1700

DRILL-7106: Fix Intellij warning for FieldSchemaNegotiator

closes #1698

DRILL-7095: Expose table schema (TupleMetadata) to physical operator (EasySubScan)

1. Add system / session option store.table.use_schema_file to control if file schema can be used during query execution. False by default.

2. Added methods in StoragePlugin interface which allow to create Group Scan with provided table schema.

3. EasyGroupScan and EasySubScan now contain table schema, also they are able to serialize / deserialize it along with other scan properties.

4. DrillTable which is the main entry point for schema provisioning, has method to store schema and later uses it to create physical scan.

5. WorkspaceSchema when returning Drill table instance will get table schema from table root if available and if store.table.use_schema_file is set to true.

This PR is the next step for Schema Provisioning project which currently exposes schema only for text reader.

closes #1696

DRILL-7092: Rename map to struct in schema definition 1. Renamed map to struct in schema parser. 2. Updated sqlTypeOf function to return STRUCT instead of MAP, drillTypeOf function will return MAP as before until internal renaming is done. 3. Add is_struct alias to already existing is_map function. Function should be revisited once Drill supports true maps. 4. Updated unit tests.

closes #1688

DRILL-7073: CREATE SCHEMA command / TupleSchema / ColumnMetadata improvements

1. Add format, default, column properties logic.

2. Changed schema JSON after serialization.

3. Added appropriate unit tests.

closes #1684

DRILL-6989: Upgrade to SqlLine 1.7

closes #1717

    • -1
    • +1
    /distribution/src/resources/sqlline.bat
DRILL-6954: Move commons libs used in UDFs module to the dependency management closes #1604

DRILL-6903: SchemaBuilder code improvements

1. ColumnBuilder: setPrecisionAndScale method

2. SchemaContainer: addColumn method parameter AbstractColumnMetadata was changed to ColumnMetadata

3. MapBuilder / RepeatedListBuilder / UnionBuilder: added constructors without parent, made buildColumn method public

4. TupleMetadata: added toMetadataList method

5. Other refactoring

  1. … 12 more files in changeset.
DRILL-6946: Implement java.sql.Connection setSchema and getSchema methods in DrillConnectionImpl closes #1596

    • -1
    • +86
    /contrib/native/client/src/protobuf/User.pb.h
    • -0
    • +2
    /protocol/src/main/protobuf/User.proto
DRILL-6931: File listing: fix issue for S3 directory objects and improve performance for recursive listing closes #1590