Clone
 

igor guzenko <ihor.huzenko.igs@gmail.com> in drill

DRILL-7254: Read Hive union w/o nulls

  1. … 6 more files in changeset.
DRILL-7387: Failed to get value by int key from map nested into struct

DRILL-7380: Query of a field inside of an array of structs returns null

1. Fixed parquet reader projection for Logical lists (DrillParquetReader.java)

2. Fixed projection pushdown for RexFieldAccess (ProjectFieldsVisitor.java)

3. DrillParquetReader.getProjection(...) splitted into few methods

4. Added javadocs for PathSegment and SchemaPath

DRILL-7252: Read Hive map using Dict<K,V> vector

DRILL-7326: Support repeated lists for CTAS parquet format

closes #1844

DRILL-7253: Read Hive struct w/o nulls

  1. … 3 more files in changeset.
DRILL-7268: Read Hive array with parquet native reader

1. Fixed preserving of group originalType for projected schema

in DrillParquetReader

2. Added reading of LIST logical type to DrillParquetGroupConverter.

Intermediate noop converter used to skip writing for next nested

repeated field after recognition of parent field as LIST. For this

skipRepeated 'true' passed to child converter's constructor.

close apache/drill#1805

    • binary
    /exec/java-exec/src/test/resources/parquet2/hive_arrays_p.parquet
DRILL-7251: Read Hive array w/o nulls

1. HiveFieldConverter replaced by Hive writers for primitives

2. Created HiveValueWriterFactory and HiveListWriter to implement arrays support

4. Readers generation replaced by HiveDefaultRecordReader and HiveTextRecordReader

5. Few reader initializers replaced by one

6. Added method to repeated vardecimal writer

7. Minor fix for array column in View

  1. … 39 more files in changeset.
DRILL-7115: Improve Hive schema show tables performance

1. To make SHOW TABLES for Hive schema work much faster, additional Drill

feature of showing only accesible tables when Storage-Based authorization

is enabled was sacrificed. Now the behaviour matches to Hive/Beeline, all

tables will be shown despite of accessibility. For details about previous

show tables results, check description of DRILL-540.

2. In HiveDatabaseSchema implemented faster getTableNamesAndTypes() method

and removed bulk related code.

3. Deprecated bulk related options and removed bulk code from AbstractSchema,

DrillHiveMetastoreClient.

4. For 8000 Hive tables query returned in 1.8 seconds, for combination of

4000 tables and 8000 views query returned in 2.3 seconds. Note, that

after first query table names will be cached and next queries will perform

in less than 1 sec.

5. Refactored WorkspaceSchemaFactory's getTableNamesAndTypes()

method to reuse existing getViews() method.

6. DrillHiveMetastoreClient was refactored. Classes were unnested and enclosed

within client package with restricted visibility. Also was updated cache

values type to avoid unnecessarry List to Set back and forth conversions.

Client creation methods moved to separate class. So the new package

exposes only factory and client class.

closes #1706

  1. … 6 more files in changeset.
DRILL-6977: Improve Hive tests configuration

1. HiveTestBase data initialization moved to static block

to be initialized once for all derivatives.

2. Extracted Hive driver and storage plugin management from HiveTestDataGenerator

to HiveTestFixture class. This increased cohesion of generator and

added loose coupling between hive test configuration and data generation

tasks.

3. Replaced usage of Guava ImmutableLists with TestBaseViewSupport

helper methods by using standard JDK collections.

closes #1613

DRILL-6944: UnsupportedOperationException thrown for view over MapR-DB binary table

1. Added persistence of MAP key and value types in Drill views (affects .view.drill file) for avoiding cast problems in future.

2. Preserved backward compatibility of older view files by treating untyped maps as ANY.

closes #1602

DRILL-6936: TestGracefulShutdown.gracefulShutdownThreadShouldBeInitializedBeforeClosingDrillbit fails if loopback address is set in hosts closes #1589

DRILL-6912: NPE when other drillbit is already running

closes #1577

DRILL-6862: Update Calcite to 1.18.0

1. Moved Calcite dependency from profile hadoop-default to general dependency managment

2. Updated Calcite version to 1.18.0-drill-r0 and Avatica version to 1.13.0

3. Hook.REL_BUILDER_SIMPLIFY moved to static block, cause now it can't be removed (fixes DRILL-6830)

4. Removed WrappedAccessor, since it was workaround fixed in CALCITE-1408

5. Fixed setting of multiple options in TestBuilder

6. Timstampadd type inference aligned with CALCITE-2699

7. Dependency update caused 417 kB increase of jdb-all jar size, so the maxsize limit was

increased from 39.5 to 40 MB

8. Added test into TestDrillParquetReader to ensure that DRILL-6856 was

fixed by Calcite update

close apache/drill#1631

DRILL-540: Allow querying hive views in Drill

1. Added DrillHiveViewTable which allows construction of DrillViewTable based

on Hive metadata

2. Added initialization of DrillHiveViewTable in HiveSchemaFactory

3. Extracted conversion of Hive data types from DrillHiveTable

to HiveToRelDataTypeConverter

4. Removed throwing of UnsupportedOperationException from HiveStoragePlugin

5. Added TestHiveViewsSupport and authorization tests

6. Added closeSilently() method to AutoCloseables

closes #1559

DRILL-6776: Removed external jQuery dependency

DRILL-786: Allow CROSS JOIN syntax

1. Removed throw statement in UnsupportedOperatorsVisitor

2. Extended UnsupportedRelOperatorException's message

closes #1488

DRILL-6765: Exclude unused shaded guava classes from drill-jdbc-all jar

closes #1486