Clone
 

arina ielchiieva <arina.yelchiyeva@gmail.com> in drill

DRILL-7713: Upgrade misc libraries which outdated versions have reported vulnerabilities

1. Jackson

2. Retrofit

3. Commons-beanutils

4. Xalan

5. Xerdes

6. Commons-codec

7. Snakeyaml

8. Metadata-extractor

9. Protostuff

Update Maven version to 3.6.3

DRILL-7704: Update Maven to 3.6.3

DRILL-7665: Add UNION to schema parser

DRILL-7672: Make metadata type required when reading from / writing into Drill Metastore

1. Upgraded Iceberg version and removed unneeded code for In / NotIn Expressions.

2. Updated Metastore Read / Modify interfaces to support required metadata types:

a. introduced abstract Read / Modify classes with boilerplate code;

b. added delete operation with filter and metadata type;

c. added metadata type validator which checks supported metadata types for each component;

d. made purge operation terminal;

e. made necessary changes in REAME.md files.

3. Added / updated unit tests.

closes #2042

  1. … 27 more files in changeset.
DRILL-7639: Replace DBCP2 with HikariCP in RDBMS (JDBC) plugin

DRILL-7630: Add additional types into SchemaParser for Parquet

closes #2019

DRILL-7627: Update MySql version for JdbcStoragePlugin tests and cache ~/.embedmysql for GitHub Actions workflow run

closes #2014

DRILL-7706: Implement Drill RDBMS Metastore

1. Fix issue with undeterministic execution of batch update / delete statements, now they will be executed in the same order as they were added.

2. Abstracted Metastore common test classes to be used by different Metastore implementations.

3. Added drill-metastore-override-example.conf with example of Drill Metastore configuration.

4. Replaced list of metadata types which are required to be passed during read / write operations with set to avoid possible duplicates.

5. Add RDBMS Metastore implementation, README.md and unit tests.

    • -0
    • +6
    /distribution/src/assemble/component.xml
    • -0
    • +64
    /distribution/src/main/resources/drill-metastore-override-example.conf
  1. … 63 more files in changeset.
DRILL-7587: Fix ValuesPrule distribution trait type

closes #2002

DRILL-7549: Fix validation error when querying absent sub folder in embedded mode

closes #1963

DRILL-7544: Upgrade Iceberg version to support Parquet 1.11.0

1. Upgraded Iceberg to the commit that supports Parquet 1.11.0.

2. Removed workaround in ExpirationHandler and used built-in logic from Iceberg library.

3. Updated description about expiration logic in README.md.

    • -5
    • +5
    /metastore/iceberg-metastore/README.md
DRILL-7530: Fix class names in loggers

1. Fix incorrect class names for loggers.

2. Minor code cleanup.

closes #1957

  1. … 41 more files in changeset.
DRILL-7467: Jdbc plugin enhancements and fixes

1. Added logic to close data source when plugin is closed.

2. Added disabled jdbc plugin template to the bootstrap storage plugins.

3. Added new jdbc storage plugin configuration property sourceParameters which would allow setting data source parameters described in BasicDataSource Configuration Parameters.

4. Upgraded commons-dbcp2 version and added it to the dependency management section in common pom.xml.

closes #1956

  1. … 3 more files in changeset.
DRILL-7518: Support INT_64 for nullable INT64 in Parquet

closes #1952

DRILL-7361: Support MAP (DICT) type in schema file

closes #1967

DRILL-7472: Fix ser / de for sys and information_schema schemas queries

closes #1925

DRILL-7481: Fix raw type warnings in Iceberg Metastore and related classes

closes #1924

DRILL-7454: Convert Avro to EVF

1. Replaced old format implementation with EVF.

2. Updated, added and improved performance for Avro tests.

3. Code refactoring.

closes #1951

  1. … 18 more files in changeset.
DRILL-7388: Kafka improvements

1. Upgraded Kafka libraries to 2.3.1 (DRILL-6739).

2. Added new options to support the same features as native JSON reader:

a. store.kafka.reader.skip_invalid_records, default: false (DRILL-6723);

b. store.kafka.reader.allow_nan_inf, default: true;

c. store.kafka.reader.allow_escape_any_char, default: false.

3. Fixed issue when Kafka topic contains only one message (DRILL-7388).

4. Replaced Gson parser with Jackson to parse JSON in the same manner as Drill native Json reader.

5. Performance improvements: Kafka consumers will be closed async, fixed issue with resource leak (DRILL-7290), moved to debug unnecessary info logging.

6. Updated bootstrap-storage-plugins.json to reflect actual Kafka connection properties.

7. Added unit tests.

8. Refactoring and code clean up.

closes #1901

  1. … 19 more files in changeset.
DRILL-7347: Upgrade Apache Iceberg to released version

DRILL-1709: Add desc alias for describe command

closes #1881

DRILL-7418: MetadataDirectGroupScan improvements

1. Replaced files listing with selection root information to reduce query plan size in MetadataDirectGroupScan.

2. Fixed MetadataDirectGroupScan ser / de issues.

3. Added PlanMatcher to QueryBuilder for more convenient plan matching.

4. Re-written TestConvertCountToDirectScan to use ClusterTest.

5. Refactoring and code clean up.

DRILL-5674: Support ZIP compression

1. Added ZipCodec implementation which can read / write single file.

2. Revisited Drill plugin formats to ensure 'openPossiblyCompressedStream' method is used in those which support compression.

3. Added unit tests.

4. General refactoring.

  1. … 3 more files in changeset.
DRILL-7401: Upgrade to SqlLine 1.9.0

closes #1875

    • -7
    • +6
    /distribution/src/resources/sqlline.bat
DRILL-6096: Provide mechanism to configure text writer configuration

1. Usage of format plugin configuration allows to specify line and field delimiters, quotes and escape characters.

2. Usage of system / session options allows to specify if writer should add headers, force quotes.

closes #1873

  1. … 5 more files in changeset.
DRILL-5983: Add missing nullable Parquet readers for INT and UINT logical types

closes #1866

DRILL-7357: Expose Drill Metastore data through information_schema

1. Add additional columns to TABLES and COLUMNS tables.

2. Add PARTITIONS table.

3. General refactoring to adjust information_schema data retrieval from multiple sources.

closes #1860

  1. … 19 more files in changeset.
DRILL-7368: Fix Iceberg Metastore failure when filter column contains nulls

DRILL-7367: Remove Server details from response headers

closes #1851