Clone Tools
  • last updated 19 mins ago
Constraints: committers
Constraints: files
Constraints: dates
DRILL-6422: Replace guava imports with shaded ones

  1. … 984 more files in changeset.
DRILL-5318: Sub-operator test fixture

This commit depends on:

* DRILL-5323

This PR cannot be accepted (or built) until the above are pulled and

this PR is rebased on top of them. The PR is issued now so that reviews

can be done in parallel.

Provides the following:

* A new OperatorFixture to set up all the objects needed to test at the

sub-operator level. This relies on the refactoring to create the

required interfaces.

* Pulls the config builder code out of the cluster fixture builder so

that configs can be build for sub-operator tests.

* Modifies the QueryBuilder test tool to run a query and get back one

of the new row set objects to allow direct inspection of data returned

from a query.

* Modifies the cluster fixture to create a JDBC connection to the test

cluster. (Use requires putting the Drill JDBC project on the test class

path since exec does not depend on JDBC.)

Created a common subclass for the cluster and operator fixtures to

abstract out the allocator and config. Also provides temp directory

support to the operator fixture.

Merged with DRILL-5415 (Improve Fixture Builder to configure client


Moved row set tests here from DRILL-5323 so that DRILL-5323 is self

contained. (The tests depend on the fixtures defined here.)

Added comments where needed.

Puts code back as it was prior to a code review comment. The code is

redundant, but necessarily so due to code which is specific to several

primitive types.

closes #788

  1. … 14 more files in changeset.
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

  1. … 119 more files in changeset.
DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separate modules.

  1. … 138 more files in changeset.
DRILL-3987: (REFACTOR) Working TPCH unit tests

- Rename DecimalUtility.integerSize to INTEGER_SIZE as is constant

- Refactor BufferManager from BufferManagerImpl

  1. … 15 more files in changeset.
DRILL-3987: (REFACTOR) Common and Vector modules building.

- Extract Accountor interface from Implementation

- Separate FMPP modules to separate out Vector Needs versus external needs

- Separate out Vector classes from those that are VectorAccessible.

- Cleanup Memory Exception hiearchy

    • -0
    • +427
  1. … 102 more files in changeset.
DRILL-3987: (MOVE) Extract key vector, field reader, complex/field writer classes.

  1. … 184 more files in changeset.
DRILL-3874: flattening large JSON objects uses too much direct memory - add getBufferSizeFor() to ValueVector interface - add implememtations of getBufferSizeFor() for all ValueVector derivatives - add adaptive algorithm for adjusting batch size to flatten operator

  1. … 15 more files in changeset.
DRILL-3811: AtomicRemainder incorrectly accounts for transferred allocations

this closes #163

  1. … 1 more file in changeset.
DRILL-3598: use a factory to create the root allocator. - made the constructor for TopLevelAllocator package private to enforce this

Delete a test that had been commented out for over a year, it no longer compiles due to interface changes and there is plenty of other testing for hash aggregate.

    • -0
    • +67
  1. … 42 more files in changeset.
DRILL-3445: BufferAllocator.buffer() implementations should throw an OutOfMemoryRuntimeException

  1. … 12 more files in changeset.
DRILL-3035: Created ControlsInjector interface to enforce method implementations + DRILL-2867: Add ControlsValidator to VALIDATORS only if assertions are enabled + return in ExecutionControls ctor if assertions are not enabled + added InjectorFactory class to align with the logger pattern

  1. … 22 more files in changeset.
DRILL-3110: Coerce Java's Direct OutOfMemoryErrors to look like reservation failures to follow existing Drill behavior paths.

DRILL-3053: add unchecked exception injection site in ChildAllocator

  1. … 4 more files in changeset.
DRILL-2750: Running 1 or more queries against Drillbits having insufficient DirectMem renders the Drillbits in an unusable state

Fix memory leaks found at operator tree construction time if there is an

allocation failure.

  1. … 5 more files in changeset.
DRILL-3033: Add memory leak fixes found so far in DRILL-1942 to 1.0.0 Fixes some missing buffer retains() and missing vector clears().

  1. … 20 more files in changeset.
DRILL-2940, DRILL-2847: Improve Memory Characteristics

- Update Large Buffer allocation so Drill releases immediately rather than waiting for Garbage Collection

- Remove DrillBuf.wrap() and all references to it.

- Update Parquet Reader to reduce object churn and indirection.

- Add additional metric to memory iterator

- Add Large and small buffer metric historgram tracking

- Add memory tracking reporter

- Update Netty to 4.0.27

  1. … 32 more files in changeset.
DRILL-2826: Simplify and centralize Operator Cleanup

- Remove cleanup method from RecordBatch interface

- Make OperatorContext creation and closing the management of FragmentContext

- Make OperatorContext an abstract class and the impl only available to FragmentContext

- Make RecordBatch closing the responsibility of the RootExec

- Make all closes be suppresing closes to maximize memory release in failure

- Add new CloseableRecordBatch interface used by RootExec

- Make RootExec AutoCloseable

- Update RecordBatchCreator to return CloseableRecordBatches so that RootExec can maintain list

- Generate list of operators through change in ImplCreator

  1. … 95 more files in changeset.
DRILL-2406: part 2 - Allow interpreted expression evaluation at planning time.

Changes needed after rebase to expose function determinism to calcite appropriately.

Address Jacques review comments.

Address chris' review comments.

Make things work now that BufferManager is AutoClosable.

Fixes tests that were creating plan fragments directly to create their own query start time,

as this information is now passed along from QueryContext during standard query initialization

(this enables the query start time and timezone to be available to planning time expression


Fix docs in BufferManger.

Update UDF interface to track determinism rather than randomness.

  1. … 13 more files in changeset.
DRILL-2275: Added support to get information about current cluster memory and threads

+ SystemRecordReader reads a SystemRecord e.g. MemoryRecord

+ Added generic data type for static tables

+ GroupScan can enforce width to be maximum width on ExcessiveExchangeRemover

+ GroupScan has minimum width for SimpleParallelizer

  1. … 22 more files in changeset.
DRILL-2187: Single Broadcast Sender

Also includes:

1. Fix merge join planning issue (1c5c810 by jinfengni)

2. ExternalSort: Check the memory available for in-memory sorting or not in making decision to spill or not (36f9dd1)

3. Cleanup in ExternalSortBatch and its helper classes (36f9dd1)

4. MergeJoinBatch: Limit the outgoing record batch size to 2^15 (37dfeb8)

5. StreamingAggBatch: Limit outgoing record batch size to 2^15 (7d8a2e4)

  1. … 55 more files in changeset.
DRILL-1990: Add peak memory allocation in a operator to OperatorStats.

  1. … 6 more files in changeset.
DRILL-1700: Removed incorrect assertion in memory allocation code

DRILL-1578: Accountor message should be trace messages

DRILL-1601: Have a minimum allocation for variable length value vectors. Improve error messages in PartitionerTemplate and UnorderedRawBatchBuffer. Minor change in cleanup for HashJoin.

  1. … 4 more files in changeset.
DRILL-1382: Fast schema return

  1. … 75 more files in changeset.
DRILL-1474: [AtomicRemainder] Failure to allocate requested memory should be logged as warning

DRILL-1411: Disable feature by default. Add an overcommit factor to the fragment limit. Can be enabled and overcommit factor can be set as bootstrap parameters.

  1. … 3 more files in changeset.
DRILL-1504: Enabling fragment memory limit causes out of memory error

  1. … 2 more files in changeset.
Patch for DRILL-705

Currently only supports partitioning/ordering, not yet preceding or

after offsets

  1. … 77 more files in changeset.