Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7607: support dynamic credit based flow control

closes #2000

  1. … 31 more files in changeset.
DRILL-7634: Rollup of code cleanup changes

Collection of code cleanup changes. The most significant

is to create constants for function names.

closes #2020

    • -4
    • +6
    ./HardAffinityFragmentParallelizer.java
    • -4
    • +6
    ./SoftAffinityFragmentParallelizer.java
    • -25
    • +26
    ./contrib/SplittingParallelizer.java
  1. … 118 more files in changeset.
DRILL-7154: TPCH query 4, 17 and 18 take longer with sf 1000 when Statistics are disabled.

closes #1737

DRILL-7146: Query failing with NPE when ZK queue is enabled.

  1. … 1 more file in changeset.
DRILL-7068: Support memory adjustment framework for resource management with Queues. closes #1677

    • -0
    • +66
    ./DefaultQueryParallelizer.java
    • -0
    • +160
    ./MemoryCalculator.java
    • -0
    • +63
    ./QueryParallelizer.java
    • -0
    • +173
    ./QueueQueryParallelizer.java
    • -9
    • +19
    ./contrib/SplittingParallelizer.java
  1. … 27 more files in changeset.
DRILL-6952: Host compliant text reader on the row set framework

The result set loader allows controlling batch sizes. The new scan framework

built on top of that framework handles projection, implicit columns, null

columns and more. This commit converts the "new" ("compliant") text reader

to use the new framework. Options select the use of the V2 ("new") or V3

(row-set based) versions. Unit tests demonstrate V3 functionality.

closes #1683

    • -8
    • +8
    ./HardAffinityFragmentParallelizer.java
  1. … 57 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

    • -2
    • +2
    ./HardAffinityFragmentParallelizer.java
    • -3
    • +3
    ./SoftAffinityFragmentParallelizer.java
    • -1
    • +1
    ./contrib/ExchangeRemoverMaterializer.java
  1. … 974 more files in changeset.
DRILL-6671: Multi level lateral unnest join is throwing an exception during materializing the plan.

closes #1426

  1. … 4 more files in changeset.
DRILL-6385: Support JPPD feature

  1. … 61 more files in changeset.
DRILL-6389: Fixed building javadocs - Added documentation about how to build javadocs - Fixed some of the javadoc warnings

closes #1276

  1. … 63 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

    • -3
    • +3
    ./HardAffinityFragmentParallelizer.java
    • -2
    • +2
    ./SoftAffinityFragmentParallelizer.java
    • -1
    • +1
    ./contrib/ExchangeRemoverMaterializer.java
  1. … 2052 more files in changeset.
DRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning

  1. … 26 more files in changeset.
DRILL-6381: (Part 3) Planner and Execution implementation to support Secondary Indexes

  1. Index Planning Rules and Plan generators

    - DbScanToIndexScanRule: Top level physical planning rule that drives index planning for several relational algebra patterns.

- DbScanSortRemovalRule: Physical planning rule for index planning for Sort-based operations.

    - Plan Generators: Covering, Non-Covering and Intersect physical plan generators.

    - Support planning with functional indexes such as CAST functions.

    - Enhance PlannerSettings with several configuration options for indexes.

  2. Index Selection and Statistics

    - An IndexSelector that support cost-based index selection of covering and non-covering indexes using statistics and collation properties.

    - Costing of index intersection for comparison with single-index plans.

  3. Planning and execution operators

    - Support RangePartitioning physical operator during query planning and execution.

    - Support RowKeyJoin physical operator during query planning and execution.

    - HashTable and HashJoin changes to support RowKeyJoin and Index Intersection.

    - Enhance Materializer to keep track of subscan association with a particular rowkey join.

  4. Index Planning utilities

    - Utility classes to perform RexNode analysis, including conversion to and from SchemaPath.

    - Utility class to analyze filter condition and an input collation to determine output collation.

    - Helper classes to maintain index contexts for logical and physical planning phase.

    - IndexPlanUtils utility class for various helper methods.

  5. Miscellaneous

    - Separate physical rel for DirectScan.

    - Modify LimitExchangeTranspose rule to handle SingleMergeExchange.

- MD-3880: Return correct status from RangePartitionRecordBatch setupNewSchema

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashJoinPOP.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillMergeProjectRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushProjectIntoScanRule.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScanRel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTrait.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PrelUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java

exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetPushDownFilter.java

exec/java-exec/src/main/resources/drill-module.conf

logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java

Resolve merge comflicts and compilation issues.

  1. … 93 more files in changeset.
DRILL-5922: - The QueryContext was never closed when the Foreman finished, so it's child allocator was never closed. Now it is. - The PlanSplitter created a QueryContext temporarily to construct an RPC message but never closed it. Now the temp QueryContext is closed. - The waitForExit method was error prone. Changed it to use the standard condition variable pattern. - Fixed timeouts in graceful shutdown tests

  1. … 6 more files in changeset.
DRILL-5716: Queue-driven memory allocation

* Creates new core resource management and query queue abstractions.

* Adds queue information to the Protobuf layer.

* Foreman and Planner changes

- Abstracts memory management out to the new resource management layer.

This means deferring generating the physical plan JSON to later in the

process after memory planning.

* Web UI changes

* Adds queue information to the main page and the profile page to each

query.

* Also sorts the list of options displayed in the Web UI.

- Added memory reserve

A new config parameter, exec.queue.memory_reserve_ratio, sets aside a

slice of total memory for operators that do not participate in the

memory assignment process. The default is 20% testing will tell us if

that value should be larger or smaller.

* Additional minor fixes

- Code cleanup.

- Added mechanism to abandon lease release during shutdown.

- Log queue configuration only when the config changes, rather than on

every query.

- Apply Boaz’ option to enforce a minimum memory allocation per

operator.

- Additional logging to help testers see what is happening.

closes #928

    • -1
    • +1
    ./HardAffinityFragmentParallelizer.java
    • -2
    • +1
    ./SoftAffinityFragmentParallelizer.java
    • -18
    • +6
    ./contrib/SplittingParallelizer.java
  1. … 52 more files in changeset.
DRILL-5547: Linking config options with system option manager

closes #868

  1. … 38 more files in changeset.
DRILL-5304: Queries fail intermittently when there is skew in data distribution

close #766

    • -1
    • +1
    ./SoftAffinityFragmentParallelizer.java
  1. … 1 more file in changeset.
DRILL-4911: Avoid plan serialization in SimpleParallelizer when debug logging is not enabled.

DRILL-4446: Support mandatory work assignment to endpoint requirements of operators

    • -0
    • +63
    ./DistributionAffinity.java
    • -0
    • +39
    ./FragmentParallelizer.java
    • -0
    • +150
    ./HardAffinityFragmentParallelizer.java
    • -0
    • +44
    ./ParallelizationParameters.java
    • -0
    • +167
    ./SoftAffinityFragmentParallelizer.java
  1. … 15 more files in changeset.
DRILL-4132 Ability to submit simple type of physical plan directly to EndPoint DrillBit for execution. There are multiple changes to achieve this: 1. During physical planning split single plan into multiple based on the number of minor fragments of the Leaf Major fragment. a. Removing exchange operators during planning b. Producing just root fragments (that will be also leaf fragments) 2. Each fragment can be executed against Drillbit it is assigned to, so to keep locality Design document can be found in the JIRA: DRILL-4132

    • -0
    • +96
    ./contrib/ExchangeRemoverMaterializer.java
    • -0
    • +55
    ./contrib/OperatorIdVisitor.java
    • -0
    • +228
    ./contrib/SplittingParallelizer.java
  1. … 24 more files in changeset.
DRILL-3079: Move execution fragment json parsing from RPC message to fragment start.

  1. … 13 more files in changeset.
DRILL-2902: Add support for context functions: user (synonyms session_user and system_user) and current_schema

  1. … 24 more files in changeset.
DRILL-2406: part 2 - Allow interpreted expression evaluation at planning time.

Changes needed after rebase to expose function determinism to calcite appropriately.

Address Jacques review comments.

Address chris' review comments.

Make things work now that BufferManager is AutoClosable.

Fixes tests that were creating plan fragments directly to create their own query start time,

as this information is now passed along from QueryContext during standard query initialization

(this enables the query start time and timezone to be available to planning time expression

evaluation).

Fix docs in BufferManger.

Update UDF interface to track determinism rather than randomness.

  1. … 14 more files in changeset.
DRILL-2275: Added support to get information about current cluster memory and threads

+ SystemRecordReader reads a SystemRecord e.g. MemoryRecord

+ Added generic data type for static tables

+ GroupScan can enforce width to be maximum width on ExcessiveExchangeRemover

+ GroupScan has minimum width for SimpleParallelizer

  1. … 19 more files in changeset.
DRILL-2245: Clean up query setup and execution kickoff in Foreman/WorkManager in order to ensure consistent handling, and avoid hangs and races, with the goal of improving Drillbit robustness.

I did my best to keep these clean when I split them up, but this core commit

may depend on some minor changes in the hygiene commit that is also

associated with this bug, so either both should be applied, or neither.

The core commit should be applied first.

protocol/pom.xml

- updated protocol buffer compiler version to 2.6

- this made slight modifications to the formats of a few committed protobuf

files

AutoCloseables

- created org.apache.drill.common.AutoCloseables to handle closing these

quietly

BaseTestQuery, and derivatives

- factored out pieces into QueryTestUtil so they can be reused

DeferredException:

- created this so we can collect exceptions during the shutdown process

Drillbit

- uses AutoCloseables for the WorkManager and for the storeProvider

- allow start() to take a RemoteServiceSet

- private, final, formatting

Foreman

- added new state CANCELLATION_REQUESTED (via UserBitShared.proto) to represent

the time between request of a cancellation, and acknowledgement from all

remote endpoints running fragments on a query's behalf

- created ForemanResult to manage interleaving cleanup effects/failure with

query result state

- does not need to implement Comparable

- does not need to implement Closeable

- thread blocking fixes

- add resultSent flag

- add code to log plan fragments with endpoint assignments

- added finals, cleaned up formatting

- do queue management in acquireQuerySemaphore; local tests pass

- rename getContext() to getQueryContext()

- retain DrillbitContext

- a couple of exception injections for testing

- minor formatting

- TODOs

FragmentContext

- added a DeferredException to collect errors during startup/shutdown sequences

FragmentExecutor

- eliminated CancelableQuery

- use the FragmentContext's DeferredException for errors

- common subexpression elimination

- cleaned up

QueryContext

- removed unnecessary functions (with some outside classes tweaked for this)

- finals, formatting

QueryManager

- merge in QueryStatus

- affects Foreman, ../batch/ControlHandlerImpl,

and ../../server/rest/ProfileResources

- made some methods private

- removed unused imports

- add finals and formatting

- variable renaming to improve readability

- formatting

- comments

- TODOs

QueryStatus

- getAsInfo() private

- member renaming

- member access changes

- formatting

- TODOs

QueryTestUtil, BaseTestQuery, TestDrillbitResilience

- make maxWidth a parameter to server startup

SelfCleaningRunnable

- created org.apache.drill.common.SelfCleaningRunnable

SingleRowListener

- created org.apache.drill.SingleRowListener results listener

- use in TestDrillbitResilience

TestComparisonFunctions

- fix not to close the FragmentContext multiple times

TestDrillbitResilience

- created org.apache.drill.exec.server.TestDrillbitResilience to test drillbit

resilience in the face of exceptions and failures during queries

TestWithZookeeper

- factor out work into ZookeeperHelper so that it can be reused by

TestDrillbitResilience

UserBitShared

- get rid of unused UNKNOWN_QUERY

WorkEventBus

- rename methods, affects Foreman and ControlHandlerImpl

- remove unused WorkerBee reference

- most members final

- formatting

WorkManager

- Closeable to AutoCloseable

- removed unused incomingFragments Set

- eliminated unnecessary eventThread and pendingTasks by posting Runnables

directly to executor

- use SelfCleaningRunnable for Foreman management

- FragmentExecutor management uses SelfCleaningRunnable

- runningFragments to be a ConcurrentHashMap; TestTpchDistributed passes

- other improvements due to bee no longer needed in various places

- most members final

- minor formatting

- comments

- TODOs

(*) Created exception injection classes to simulate exceptions for testing

- ExceptionInjection

- ExceptionInjector

- ExceptionInjectionUtil

- TestExceptionInjection

DRILL-2245-hygiene: General code cleanup encountered while working on the rest

of this commit. This includes

- making members final whenever possible

- making members private whenever possible

- making loggers private

- removing unused imports

- removing unused private functions

- removing unused public functions

- removing unused local variables

- removing unused private members

- deleting unused files

- cleaning up formatting

- adding spaces before braces in conditionals and loop bodies

- breaking up overly long lines

- removing extra blank lines

While I tried to keep this clean, this commit may have minor dependencies on

DRILL-2245-core that I missed. The intention is just to break this up for

review purposes. Either both commits should be applied, or neither.

  1. … 92 more files in changeset.
DRILL-2210 Introducing multithreading capability to PartitonerSender

  1. … 15 more files in changeset.
DRILL-2170: For fragment parallelization, use max cost of operators instead of total cost (this makes it consistent with what ExcessiveExchangeRemover uses).

  1. … 1 more file in changeset.
DRILL-133: LocalExchange planning and exec.

    • -0
    • +139
    ./ParallelizationInfo.java
  1. … 61 more files in changeset.
DRILL-1846: Use max receiver width during stats collection for parallelism planning. SingleMergeExchange and UnionExchange have a max receive width of 1. Others can go higher.

  1. … 5 more files in changeset.
DRILL-1846: Use max receiver width during stats collection for parallelism planning. SingleMergeExchange and UnionExchange have a max receive width of 1. Others can go higher.

  1. … 5 more files in changeset.