Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7096: Develop vector for canonical Map<K,V>

- Added new type DICT;

- Created value vectors for the type for single and repeated modes;

- Implemented corresponding FieldReaders and FieldWriters;

- Made changes in EvaluationVisitor to be able to read values from the map by key;

- Made changes to DrillParquetGroupConverter to be able to read Parquet's MAP type;

- Added an option `store.parquet.reader.enable_map_support` to disable reading MAP type as DICT from Parquet files;

- Updated AvroRecordReader to use new DICT type for Avro's MAP;

- Added support of the new type to ParquetRecordWriter.

  1. … 108 more files in changeset.
DRILL-7019: Add check for redundant imports

close apache/drill#1629

  1. … 23 more files in changeset.
DRILL-6844: Query with ORDER BY DESC on indexed column does not pick secondary index.

  1. … 2 more files in changeset.
DRILL-6381: Address code review comments (part 3).

DRILL-6381: Add missing joinControl logic for INTERSECT_DISTINCT.

- Modified HashJoin's probe phase to process INTERSECT_DISTINCT.

- NOTE: For build phase, the functionality will be same as for SemiJoin when it is added later.

DRILL-6381: Address code review comment for intersect_distinct.

DRILL-6381: Rebase on latest master and fix compilation issues.

DRILL-6381: Generate protobuf files for C++ native client.

DRILL-6381: Use shaded Guava classes. Add more comments and Javadoc.

    • -0
    • +4
    ./InvalidIndexDefinitionException.java
    • -9
    • +9
    ./generators/IndexIntersectPlanGenerator.java
  1. … 33 more files in changeset.
DRILL-6381: Address review comments (part 2): fix formatting issues and add javadoc.

  1. … 23 more files in changeset.
DRILL-6381: Address code review comments.

    • -5
    • +17
    ./generators/IndexIntersectPlanGenerator.java
    • -10
    • +11
    ./generators/NonCoveringIndexPlanGenerator.java
  1. … 7 more files in changeset.
DRILL-6381: (Part 5) Update Javadoc for a few interfaces.

  1. … 1 more file in changeset.
DRILL-6381: (Part 3) Planner and Execution implementation to support Secondary Indexes

  1. Index Planning Rules and Plan generators

    - DbScanToIndexScanRule: Top level physical planning rule that drives index planning for several relational algebra patterns.

- DbScanSortRemovalRule: Physical planning rule for index planning for Sort-based operations.

    - Plan Generators: Covering, Non-Covering and Intersect physical plan generators.

    - Support planning with functional indexes such as CAST functions.

    - Enhance PlannerSettings with several configuration options for indexes.

  2. Index Selection and Statistics

    - An IndexSelector that support cost-based index selection of covering and non-covering indexes using statistics and collation properties.

    - Costing of index intersection for comparison with single-index plans.

  3. Planning and execution operators

    - Support RangePartitioning physical operator during query planning and execution.

    - Support RowKeyJoin physical operator during query planning and execution.

    - HashTable and HashJoin changes to support RowKeyJoin and Index Intersection.

    - Enhance Materializer to keep track of subscan association with a particular rowkey join.

  4. Index Planning utilities

    - Utility classes to perform RexNode analysis, including conversion to and from SchemaPath.

    - Utility class to analyze filter condition and an input collation to determine output collation.

    - Helper classes to maintain index contexts for logical and physical planning phase.

    - IndexPlanUtils utility class for various helper methods.

  5. Miscellaneous

    - Separate physical rel for DirectScan.

    - Modify LimitExchangeTranspose rule to handle SingleMergeExchange.

- MD-3880: Return correct status from RangePartitionRecordBatch setupNewSchema

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashJoinPOP.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillMergeProjectRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushProjectIntoScanRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScanRel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTrait.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PrelUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java

exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetPushDownFilter.java

exec/java-exec/src/main/resources/drill-module.conf

logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java

Resolve merge comflicts and compilation issues.

    • -0
    • +199
    ./FindFiltersForCollation.java
    • -0
    • +215
    ./FunctionalIndexHelper.java
    • -0
    • +250
    ./IndexConditionInfo.java
    • -0
    • +75
    ./IndexDiscoverFactory.java
    • -0
    • +178
    ./IndexLogicalPlanCallContext.java
    • -0
    • +193
    ./IndexPhysicalPlanCallContext.java
    • -0
    • +855
    ./IndexPlanUtils.java
    • -0
    • +766
    ./IndexSelector.java
    • -0
    • +147
    ./PathInExpr.java
    • -0
    • +116
    ./RexSeparator.java
    • -0
    • +165
    ./RexToExpression.java
    • -0
    • +300
    ./SimpleRexRemap.java
  1. … 79 more files in changeset.
DRILL-6381: (Part 4) Enhance MapR-DB plugin to support querying secondary indexes

  1. Implementation of the index descriptor for MapR-DB.

2. MapR-DB specific costing for covering and non-covering indexes.

3. Discovery componenent to discover the indexes available for a MapR-DB table including CAST functional indexes.

4. Utility functions to build a canonical index descriptor.

5. Statistics: fetch and initialize statistcs from MapR-DB for a query condition. Maintain a query-scoped cache for the statistics. Utility functions to compute selectivity.

6. Range Partitioning: partitioning function that takes into account the tablet map to find out where a particular rowkey belongs.

7. Restricted Scan: support doing restricted (i.e skip) scan through lookups on the rowkey. Added a group-scan and record reader for this.

8. MD-3726: Simple Order by queries (without limit) when an index is used are showing regression.

9. MD-3995: Do not pushdown limit 0 past project with CONVERT_FROMJSON

10. MD-4259 : Account for limit during hashcode computation

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBFormatMatcher.java

contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBPushProjectIntoScan.java

contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/JsonTableGroupScan.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/index/rules/DbScanSortRemovalRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SortPrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/TopNPrel.java

Fix additional compilation issues.

    • -17
    • +32
    ./generators/AbstractIndexPlanGenerator.java
    • -8
    • +8
    ./generators/CoveringPlanNoFilterGenerator.java
    • -1
    • +1
    ./generators/NonCoveringIndexPlanGenerator.java
    • -20
    • +33
    ./rules/DbScanSortRemovalRule.java
  1. … 23 more files in changeset.
DRILL-6381: (Part 1) Secondary Index framework

  1. Secondary Index planning interfaces and abstract classes like DBGroupScan, DbSubScan, IndexDecriptor etc.

  2. Statistics and Cost model interfaces/classes: PluginCost, Statistics, StatisticsPayload, AbstractIndexStatistics

  3. ScanBatch and RecordReader to support repeatable scan

  4. Secondary Index execution related interfaces: RangePartitionSender, RowKeyJoin, PartitionFunction

5. MD-3979: Query using cast index plan fails with NPE

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java

protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java

protocol/src/main/java/org/apache/drill/exec/proto/beans/CoreOperatorType.java

protocol/src/main/protobuf/UserBitShared.proto

    • -0
    • +96
    ./AbstractIndexCollection.java
    • -0
    • +74
    ./AbstractIndexDescriptor.java
    • -0
    • +51
    ./AbstractIndexStatistics.java
    • -0
    • +37
    ./CollationContext.java
    • -0
    • +75
    ./DrillIndexCollection.java
    • -0
    • +278
    ./DrillIndexDefinition.java
    • -0
    • +110
    ./DrillIndexDescriptor.java
    • -0
    • +85
    ./FunctionalIndexInfo.java
    • -0
    • +76
    ./IndexCallContext.java
    • -0
    • +99
    ./IndexCollection.java
    • -0
    • +105
    ./IndexDefinition.java
    • -0
    • +68
    ./IndexDescriptor.java
    • -0
    • +23
    ./IndexDiscover.java
    • -0
    • +110
    ./IndexDiscoverBase.java
    • -0
    • +37
    ./IndexDiscoverable.java
  1. … 30 more files in changeset.