Clone Tools
  • last updated 27 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7376: Drill ignores Hive schema for MaprDB tables when group scan has star column

  1. … 3 more files in changeset.
DRILL-7314: Use TupleMetadata instead of concrete implementation

1. Add ser / de implementation for TupleMetadata interface based on types.

2. Replace TupleSchema usage where possible.

3. Move patcher classes into commons.

4. Upgrade some dependencies and general refactoring.

    • -14
    • +13
    ./db/json/JsonTableGroupScan.java
    • -3
    • +2
    ./db/json/RestrictedJsonTableGroupScan.java
  1. … 36 more files in changeset.
DRILL-7316: Move classes from org.apache.drill.metastore into org.apache.drill.exec.metastore package in java-exec module

    • -1
    • +1
    ./db/json/RestrictedJsonTableGroupScan.java
  1. … 29 more files in changeset.
DRILL-7313: Use Hive schema for MaprDB native reader when field was empty

- Added all_text_mode option for hive maprDB Json

- Improved logic to convert Hive's schema into Drill's one

- Added unit tests for schema conversion

    • -2
    • +2
    ./db/MapRDBRestrictedScanBatchCreator.java
    • -14
    • +21
    ./db/binary/BinaryTableGroupScan.java
    • -20
    • +29
    ./db/json/JsonTableGroupScan.java
    • -2
    • +4
    ./db/json/RestrictedJsonRecordReader.java
    • -7
    • +13
    ./db/json/RestrictedJsonTableGroupScan.java
  1. … 14 more files in changeset.
DRILL-7150: Fix timezone conversion for timestamp from MaprDB after the transition from PDT to PST closes #1729

    • -2
    • +9
    ./db/json/CompareFunctionsProcessor.java
  1. … 1 more file in changeset.
DRILL-7118: Filter not getting pushed down on MapR-DB tables.

closes #1708

    • -7
    • +7
    ./db/json/CompareFunctionsProcessor.java
DRILL-7113: Fix creation of filter conditions for IS NULL and IS NOT NULL for MapR-DB format plugin

close apache/drill#1704

  1. … 1 more file in changeset.
DRILL-7200: Update Calcite to 1.19.0 / 1.20.0

    • -0
    • +10
    ./db/json/JsonConditionBuilder.java
  1. … 46 more files in changeset.
DRILL-5603: Replace String file paths to Hadoop Path - replaced all String path representation with org.apache.hadoop.fs.Path - added PathSerDe.Se JSON serializer - refactoring of DFSPartitionLocation code by leveraging existing listPartitionValues() functionality

closes #1657

  1. … 82 more files in changeset.
DRILL-6969: Fix inconsistency of reading MaprDB JSON tables using hive plugin when native reader is enabled

closes #1610

    • -11
    • +46
    ./db/json/CompareFunctionsProcessor.java
    • -13
    • +62
    ./db/json/MaprDBJsonRecordReader.java
  1. … 6 more files in changeset.
DRILL-6691: Unify checkstyle-config.xml files.

closes #1550

    • -3
    • +3
    ./db/json/JsonTableRangePartitionFunction.java
    • -1
    • +0
    ./db/json/ProjectionPassthroughVectorWriter.java
  1. … 15 more files in changeset.
DRILL-6744: Support varchar and decimal push down

1. Added enableStringsSignedMinMax parquet format plugin config and store.parquet.reader.strings_signed_min_max session option to control reading binary statistics for files generated by prior versions of Parquet 1.10.0.

2. Added ParquetReaderConfig to store configuration needed during reading parquet statistics or files.

3. Provided mechanism to enable varchar / decimal filter push down.

4. Added VersionUtil to compare Drill versions in string representation.

5. Added appropriate unit tests.

closes #1537

  1. … 41 more files in changeset.
Fixed imports for DRILL-6381

    • -1
    • +1
    ./db/json/ProjectionPassthroughVectorWriter.java
    • -1
    • +1
    ./db/json/RestrictedJsonRecordReader.java
  1. … 5 more files in changeset.
DRILL-1328: Support table statistics

  1. … 51 more files in changeset.
DRILL-6793: FragmentExecutor cannot send its final state for the case when RootExec root wasn't initialized

closes #1506

  1. … 5 more files in changeset.
DRILL-6381: Address code review comments (part 3).

DRILL-6381: Add missing joinControl logic for INTERSECT_DISTINCT.

- Modified HashJoin's probe phase to process INTERSECT_DISTINCT.

- NOTE: For build phase, the functionality will be same as for SemiJoin when it is added later.

DRILL-6381: Address code review comment for intersect_distinct.

DRILL-6381: Rebase on latest master and fix compilation issues.

DRILL-6381: Generate protobuf files for C++ native client.

DRILL-6381: Use shaded Guava classes. Add more comments and Javadoc.

    • -19
    • +16
    ./db/MapRDBPushProjectIntoScan.java
    • -2
    • +2
    ./db/MapRDBRestrictedScanBatchCreator.java
    • -2
    • +2
    ./db/json/JsonTableRangePartitionFunction.java
    • -3
    • +3
    ./db/json/RestrictedJsonTableGroupScan.java
  1. … 26 more files in changeset.
DRILL-6381: Address review comments (part 2): fix formatting issues and add javadoc.

    • -2
    • +2
    ./db/json/JsonTableRangePartitionFunction.java
    • -4
    • +4
    ./db/json/RestrictedJsonTableGroupScan.java
  1. … 24 more files in changeset.
DRILL-6773: The renamed schema with aliases is not shown for queries on empty directories

closes #1492

  1. … 17 more files in changeset.
DRILL-6381: Address code review comments.

    • -28
    • +0
    ./db/RestrictedMapRDBSubScanSpec.java
    • -1
    • +1
    ./db/json/JsonTableRangePartitionFunction.java
  1. … 10 more files in changeset.
DRILL-6824: Handle schema changes in MapRDBJsonRecordReader closes #1518

    • -26
    • +101
    ./db/json/MaprDBJsonRecordReader.java
  1. … 3 more files in changeset.
DRILL-6724: Dump operator context to logs when error occurs during query execution

closes #1455

    • -1
    • +16
    ./db/json/MaprDBJsonRecordReader.java
  1. … 102 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

    • -1
    • +1
    ./db/binary/MaprDBCompareFunctionsProcessor.java
    • -2
    • +2
    ./db/json/CompareFunctionsProcessor.java
  1. … 971 more files in changeset.
DRILL-6639: Exception happens while displaying operator profiles for some queries

  1. … 17 more files in changeset.
DRILL-6639: Exception happens while displaying operator profiles for some queries

closes #1404

  1. … 17 more files in changeset.
DRILL-6614: Allow usage of MapRDBFormatPlugin for HiveStoragePlugin

  1. … 6 more files in changeset.
DRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table

closes #1314

  1. … 11 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

    • -1
    • +1
    ./db/json/CompareFunctionsProcessor.java
    • -1
    • +1
    ./streams/StreamsFormatPluginConfig.java
  1. … 2052 more files in changeset.
DRILL-6094: Decimal data type enhancements

Add ExprVisitors for VARDECIMAL

Modify writers/readers to support VARDECIMAL

- Added usage of VarDecimal for parquet, hive, maprdb, jdbc;

- Added options to store decimals as int32 and int64 or fixed_len_byte_array or binary;

Add UDFs for VARDECIMAL data type

- modify type inference rules

- remove UDFs for obsolete DECIMAL types

Enable DECIMAL data type by default

Add unit tests for DECIMAL data type

Fix mapping for NLJ when literal with non-primitive type is used in join conditions

Refresh protobuf C++ source files

Changes in C++ files

Add support for decimal logical type in Avro.

Add support for date, time and timestamp logical types.

Update Avro version to 1.8.2.

    • -1
    • +10
    ./db/json/CompareFunctionsProcessor.java
    • -1
    • +10
    ./db/json/MaprDBJsonRecordReader.java
  1. … 200 more files in changeset.
DRILL-6381: (Part 3) Planner and Execution implementation to support Secondary Indexes

  1. Index Planning Rules and Plan generators

    - DbScanToIndexScanRule: Top level physical planning rule that drives index planning for several relational algebra patterns.

- DbScanSortRemovalRule: Physical planning rule for index planning for Sort-based operations.

    - Plan Generators: Covering, Non-Covering and Intersect physical plan generators.

    - Support planning with functional indexes such as CAST functions.

    - Enhance PlannerSettings with several configuration options for indexes.

  2. Index Selection and Statistics

    - An IndexSelector that support cost-based index selection of covering and non-covering indexes using statistics and collation properties.

    - Costing of index intersection for comparison with single-index plans.

  3. Planning and execution operators

    - Support RangePartitioning physical operator during query planning and execution.

    - Support RowKeyJoin physical operator during query planning and execution.

    - HashTable and HashJoin changes to support RowKeyJoin and Index Intersection.

    - Enhance Materializer to keep track of subscan association with a particular rowkey join.

  4. Index Planning utilities

    - Utility classes to perform RexNode analysis, including conversion to and from SchemaPath.

    - Utility class to analyze filter condition and an input collation to determine output collation.

    - Helper classes to maintain index contexts for logical and physical planning phase.

    - IndexPlanUtils utility class for various helper methods.

  5. Miscellaneous

    - Separate physical rel for DirectScan.

    - Modify LimitExchangeTranspose rule to handle SingleMergeExchange.

- MD-3880: Return correct status from RangePartitionRecordBatch setupNewSchema

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashJoinPOP.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillMergeProjectRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushProjectIntoScanRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScanRel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTrait.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PrelUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java

exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetPushDownFilter.java

exec/java-exec/src/main/resources/drill-module.conf

logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java

Resolve merge comflicts and compilation issues.

  1. … 91 more files in changeset.
DRILL-6381: (Part 4) Enhance MapR-DB plugin to support querying secondary indexes

  1. Implementation of the index descriptor for MapR-DB.

2. MapR-DB specific costing for covering and non-covering indexes.

3. Discovery componenent to discover the indexes available for a MapR-DB table including CAST functional indexes.

4. Utility functions to build a canonical index descriptor.

5. Statistics: fetch and initialize statistcs from MapR-DB for a query condition. Maintain a query-scoped cache for the statistics. Utility functions to compute selectivity.

6. Range Partitioning: partitioning function that takes into account the tablet map to find out where a particular rowkey belongs.

7. Restricted Scan: support doing restricted (i.e skip) scan through lookups on the rowkey. Added a group-scan and record reader for this.

8. MD-3726: Simple Order by queries (without limit) when an index is used are showing regression.

9. MD-3995: Do not pushdown limit 0 past project with CONVERT_FROMJSON

10. MD-4259 : Account for limit during hashcode computation

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBFormatMatcher.java

contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBPushProjectIntoScan.java

contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/JsonTableGroupScan.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/index/rules/DbScanSortRemovalRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SortPrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/TopNPrel.java

Fix additional compilation issues.

    • -3
    • +195
    ./db/json/JsonTableGroupScan.java
    • -0
    • +237
    ./db/json/JsonTableRangePartitionFunction.java
    • -0
    • +184
    ./db/json/RestrictedJsonTableGroupScan.java
  1. … 23 more files in changeset.