Clone Tools
  • last updated 20 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-3229: Implement Union type vector

  1. … 54 more files in changeset.
DRILL-3353: Fix dropping nested fields

Use the SchemaChangeCallBack in more places to track schema changes

Reset the ephemeral transfer pair when making a new transfer pair for Map or RepeatedMap

  1. … 17 more files in changeset.
DRILL-2053: Fix incorrect query result when join CTE, by making Project operator use case-insensitive matching for columns in input data stream.

  1. … 1 more file in changeset.
DRILL-2150: Create an abstraction for repeated value vectors.

  1. … 35 more files in changeset.
DRILL-2757: Verify operators correctly handle low memory conditions and cancellations

includes:

DRILL-2816: system error does not display the original Exception message

DRILL-2893: ScanBatch throws a NullPointerException instead of returning OUT_OF_MEMORY

DRILL-2894: FixedValueVectors shouldn't set it's data buffer to null when it fails to allocate it

DRILL-2895: AbstractRecordBatch.buildSchema() should properly handle OUT_OF_MEMORY outcome

DRILL-2905: RootExec implementations should properly handle IterOutcome.OUT_OF_MEMORY

DRILL-2920: properly handle OutOfMemoryException

DRILL-2947: AllocationHelper.allocateNew() doesn't have a consistent behavior when it can't allocate

also:

- added UserException.memoryError() with a pre assigned error message

- injection site in ScanBatch and unit test that runs various tpch queries and injects

an exception in the ScanBatch that will cause an OUT_OF_MEMORY outcome to be sent

  1. … 36 more files in changeset.
DRILL-2826: Simplify and centralize Operator Cleanup

- Remove cleanup method from RecordBatch interface

- Make OperatorContext creation and closing the management of FragmentContext

- Make OperatorContext an abstract class and the impl only available to FragmentContext

- Make RecordBatch closing the responsibility of the RootExec

- Make all closes be suppresing closes to maximize memory release in failure

- Add new CloseableRecordBatch interface used by RootExec

- Make RootExec AutoCloseable

- Update RecordBatchCreator to return CloseableRecordBatches so that RootExec can maintain list

- Generate list of operators through change in ImplCreator

  1. … 94 more files in changeset.
DRILL-2762: Update Fragment state reporting and error collection

DeferredException

- Add new throwAndClear operation on to allow checking for exceptions preClose in FragmentContext

- Add new getAndClear operation

BufferManager

- Ensure close() can be called multiple times by clearing managed buffer list on close().

FragmentContext/FragmentExecutor

- Update FragmentContext to have a preClose so that we can check closure state before doing final close.

- Update so that there is only a single state maintained between FragmentContext and FragmentExecutor

- Clean up FragmentExecutor run() method to better manage error states and have only single terminal point (avoiding multiple messages to Foreman).

- Add new CANCELLATION_REQUESTED state for FragmentState.

- Move all users of isCancelled or isFailed in main code to use shouldContinue()

- Update receivingFragmentFinished message to not cancel fragment (only inform root operator of cancellation)

WorkManager Updates

- Add new afterExecute command to the WorkManager ExecutorService so that we get log entries if a thread leaks an exception. (Otherwise logs don't show these exceptions and they only go to standard out.)

Profile Page

- Update profile page to show last update and last progress.

- Change durations to non-time presentation

Foreman/QueryManager

- Extract listenable interfaces into anonymous inner classes from body of Foreman

QueryManager

- Update QueryManager to track completed nodes rather than completed fragments using NodeTracker

- Update DrillbitStatusListener to decrement expected completion messages on Nodes that have died to avoid query hang when a node dies

FragmentData/MinorFragmentProfile

- Add ability to track last status update as well as last time fragment made progress

AbstractRecordBatch

- Update awareness of current cancellation state to avoid cancellation delays

Misc. Other changes

- Move ByteCode optimization code to only record assembly and code as trace messages

- Update SimpleRootExec to create fake ExecutorState to make existing tests work.

- Update sort to exit prematurely in the case that the fragment was asked to cancel.

- Add finals to all edited files.

- Modify control handler and FragmentManager to directly support receivingFragmentFinished

- Update receiver propagation message to avoid premature removal of fragment manager

- Update UserException.Builder to log a message if we're creating a new UserException (ERROR for System, INFO otherwise).

- Update Profile pages to use min and max instead of sorts.

  1. … 44 more files in changeset.
DRILL-2311: In ProjectRecordBatch, even if a column from incoming recordbatch does not need to be classified, the output name for this column is still ensured to be unique

  1. … 1 more file in changeset.
DRILL-2060: Constant folding rule

2060 update - Constant folding work completed.

Fix issue with date, time and timestamp literal creation.

Fix literal creation during expression interpretation to match nullability of incoming expression.

Fix decimal literals in interpreted expression eval.

Disable test with an exposed planning bug when the project instance of the constant folding rule is enabled. The rule is not actually influencing the final plan when the rule is firing and making expression reductions. This is due to our current cost model fro project which just counts the number of expressions and does not consider expression complexity. The issues have been logged in DRILL-2218 for further investigation, they do not need to be solved to merge the other constant folding rules and all of the interpreted expression work that has been done.

Get rid of clutter in RuleSets, explanation has been moved to the 2218 JIRA.

Belongs with 2060, fix constant expression executor to use the new constant expression interpreter interface that returns a ValueHolder instead a ValueVector with a single value filled in.

2060 update - change test baseline due to new column ordering (no functional or performance impacting changes to plan)

2060 - address Aman's comments.

add test ignore - DRILL-2218

Baseline update for project pushdown test (only column ordering on a scan, no functional or performance impacting plan changes)

Turn back on project instance.

Small casting bug in constant executor.

Don't fold hive UDFs.

Modify DrillBuf to allow a BufferManager to be the owning context for a DrilllBuf.

TODO - refactor to remove remaining common code from OperatorContext and FragmentContext,

have them both use the new BufferManager.

Add system option for disabling constant folding.

2060 update - test option to disable constant folding.

Update RuleSets to actually allow turning the constant folding rules on and off as well as establish general pattern for turning logical rules on an off, similar to how some physical rules can be already.

Change the estimated row count in EasyGroupScan to report a number of files in the case where the file size indicates an estimated total count of 0 records. Allows very small files to be pruned.

Fix folding expressions that result in null after refactoring the interpreted expression evaluation to return a ValueHolder in the case of a constant expression. Previously a value vector was returned in the same manner as the interpreter can still do when given an input VectorAccessible and an expression that may contain fild references. Calling getObject on the output vector previously gracefully handled nulls as they were passed into the Calciate API to create literals. This process has to be a bit more manual now.

Address Jinfeng's review comments.

A few more review comments.

Disable cost calculation change, complete fix will come in 2553.

Throw a runtime exception of there is an error materializing the expression, as the same materialization will take place at query execution time we should fail early.

Add a test that does prune appropriately, still have a test for the outstanding issue tracked in DRILL-2553.

Small fix for test to properly set session option and set it back after completion.

Fixing comment that was garbled somehow.

small fix for case where expression returns a null result during constant folding.

Add a little defensive code to give a good error message if a type that does not appear in the mapping from Drill to Calcite types attempts to be folded into a null value.

  1. … 21 more files in changeset.
DRILL-1960: Automatic reallocation

  1. … 63 more files in changeset.
DRILL-2021: In ProjectRecordBatch, for the case where expr != ref, allow duplicates; when projecting, keep track of the used output names

  1. … 10 more files in changeset.
DRILL-1825: In ProjectRecordBatch, avoid requesting memory (will result in memory leak) when no output column is needed

  1. … 1 more file in changeset.
DRILL-1839: Clean up complex writers per batch if project has a ComplexWriter function evaluation

DRILL-1839: Clean up complex writers per batch if project has a ComplexWriter function evaluation

DRILL-1811: select A, * from ... crashes JVM

  1. … 1 more file in changeset.
DRILL-1811: select A, * from ... crashes JVM

  1. … 1 more file in changeset.
DRILL-1781: Fast Complex Schema

  1. … 44 more files in changeset.
DRILL-1828: In ProjectRecordBatch, add a case for expression classification

  1. … 1 more file in changeset.
DRILL-1382: Fast schema return

  1. … 75 more files in changeset.
Patch for DRILL-705

Currently only supports partitioning/ordering, not yet preceding or

after offsets

  1. … 77 more files in changeset.
DRILL-1402: Add check-style rules for trailing space, TABs and blocks without braces

  1. … 440 more files in changeset.
DRILL-634: Cleanup/organize Java imports and trailing whitespaces from Drill code

  1. … 769 more files in changeset.
Fix issue introduced by DRILL-1202 where allocators are being closed after reporting success. Update ScreenRoot to cleanup before returning success. Update ScanBatch to cleanup reader in case of limit query to avoid memory leak in ParquetReader. Update allocators so that we don't have memory leak when using debug options. Update project record batch so that it doesn't try to return a released remainder.

  1. … 6 more files in changeset.
DRILL-1329: External sort memory fixes

  1. … 38 more files in changeset.
DRILL-1310: Fix assertion in ProjectRecordBatch for certain types of star queries.

DRILL-1324: Add mechanism to detect schema changes when adding a new primitive vector in a Map, RepeatedMap, RepeatedList vector

  1. … 13 more files in changeset.
DRILL-1278: Fix selecting scalar field from a map with join clause.

DRILL-1293: Fix assertion when selecting star column from view that also has star column.

  1. … 1 more file in changeset.
DRILL-1220: Process the star columns in ProjectRecordBatch by classifying the expressions appropriately based on the annotated expressions created by planner.

  1. … 3 more files in changeset.
DRILL-1060: Support ComplexToJson for Array Data Type

  1. … 9 more files in changeset.