Clone Tools
  • last updated 24 mins ago
Constraints: committers
Constraints: files
Constraints: dates
DRILL-5423: Refactor ScanBatch to allow unit testing record readers

Refactors ScanBatch to allow unit testing of record reader

implementations, especially the “writer” classes.

See JIRA for details.

closes #811

  1. … 8 more files in changeset.
DRILL-4257: Fix StoragePluginRegistry clean-up behavior and misc clean up.

- Create the storage plugin registry using a Drill configuration parameter to be able to replace the registry implementation.

- Change StoragePluginRegistry into an interface and move the implementation to an impl class.

- Write documentation for StoragePluginRegistry.

- Make StoragePluginRegistry and StoragePluginMap AutoCloseable and ensure that Drillbit closes registry.

- Misc Drillbit code reorganization so that fields are at top of class (and static methods at bottom).

- Update DrillConfig to support reflection-based creation of storage plugin objects

- Remove final from DrillConfig so that application developers can extend config object.

This closes #321.

  1. … 10 more files in changeset.
DRILL-4180: When clear() is called on BitVector, ensure valueCount is set to zero

  1. … 1 more file in changeset.
DRILL:3854: Ensure the readIndex of DrillBuf[] returned by ValueVector.getBuffers points at 0

  1. … 2 more files in changeset.
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

  1. … 118 more files in changeset.
DRILL-4134: Add new allocator

    • -0
    • +651
  1. … 27 more files in changeset.
DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separate modules.

  1. … 140 more files in changeset.
DRILL-3987: (REFACTOR) Working TPCH unit tests

- Rename DecimalUtility.integerSize to INTEGER_SIZE as is constant

- Refactor BufferManager from BufferManagerImpl

  1. … 16 more files in changeset.
DRILL-3987: (REFACTOR) Common and Vector modules building.

- Extract Accountor interface from Implementation

- Separate FMPP modules to separate out Vector Needs versus external needs

- Separate out Vector classes from those that are VectorAccessible.

- Cleanup Memory Exception hiearchy

  1. … 105 more files in changeset.
DRILL-3742: Classpath scanning and build improvement

Makes the classpath scanning a build time class discovery

Makes the fmpp generation incremental

Removes some slowness in DrillBit closing

Reduces the build time by 30%

This closes #148

  1. … 143 more files in changeset.
DRILL-3927: use OutOfMemoryException in more places Some code cleanup required for the upcoming introduction of the rewritten direct memory allocator. Chiefly the introduction of OutOfMemoryException handling in a few spots, the use of DrillAutoCloseables for some allocator close() calls, and some other minor cleanup. - removed the exclusion of exec/work from the jdbc-all jar, because it now depends on OutOfMemoryException, and that is derived from FragmentSetupException and Foreman Exception

this closes #199

  1. … 14 more files in changeset.
DRILL-3811: AtomicRemainder incorrectly accounts for transferred allocations

this closes #163

  1. … 1 more file in changeset.
DRILL-3598: use a factory to create the root allocator. - made the constructor for TopLevelAllocator package private to enforce this

Delete a test that had been commented out for over a year, it no longer compiles due to interface changes and there is plenty of other testing for hash aggregate.

  1. … 43 more files in changeset.
DRILL-3340: Added operator names and metric names to query profile before writing it to store

+ Rename: FragmentStats#getOperatorStats => newOperatorStats

+ Documentation

this closes #216

  1. … 12 more files in changeset.
DRILL-3445: BufferAllocator.buffer() implementations should throw an OutOfMemoryRuntimeException

  1. … 13 more files in changeset.
DRILL-2826: Simplify and centralize Operator Cleanup

- Remove cleanup method from RecordBatch interface

- Make OperatorContext creation and closing the management of FragmentContext

- Make OperatorContext an abstract class and the impl only available to FragmentContext

- Make RecordBatch closing the responsibility of the RootExec

- Make all closes be suppresing closes to maximize memory release in failure

- Add new CloseableRecordBatch interface used by RootExec

- Make RootExec AutoCloseable

- Update RecordBatchCreator to return CloseableRecordBatches so that RootExec can maintain list

- Generate list of operators through change in ImplCreator

  1. … 95 more files in changeset.
DRILL-2510: Fix unclosed allocators detected in Java shutdown hook


- fix warnings, including closing unclosed resources with try-with-resources


- close unclosed resources

doesn't use try-with-resources because this stresses interleaved allocation

operations that wouldn't necessarily be nested


- created convenience class for tracking stack traces


- used StackTrace to track location of Drillbit starts for shutdown hook

error messages so we can more easily identify errant tests from just the logs

in future

- added an isClosed flag to detect that the Drillbit has already been closed

to avoid complaints about double closures that happen in many tests that

explicitly close their drillbits (the shutdown hook then closes them again,

causing bogus complaints)

- synchronized close()

- made the id generator in the ShutdownThread an AtomicInteger

  1. … 3 more files in changeset.
DRILL-1411: Disable feature by default. Add an overcommit factor to the fragment limit. Can be enabled and overcommit factor can be set as bootstrap parameters.

  1. … 5 more files in changeset.
DRILL-1504: Enabling fragment memory limit causes out of memory error

  1. … 2 more files in changeset.
DRILL-1411 Fragment memory planning

    • -0
    • +189
  1. … 20 more files in changeset.
DRILL-634: Cleanup/organize Java imports and trailing whitespaces from Drill code

  1. … 769 more files in changeset.
Make tests extend shared base class. Add additional tracking in base class around memory usage per test.

  1. … 57 more files in changeset.
DRILL-334: Subdivide Drillbit control and data messages. Add support for socket backpressure. Add TopLevel and Child memory allocator with debug mode to capture memory leaks. Various memory leak fixes to get build to complete.

Also includes fixes from reviews by Tim.

  1. … 212 more files in changeset.
DRILL-221 Add license header to all files

  1. … 829 more files in changeset.
DRILL-165: Reorganize directories (moves only)

    • -0
    • +24
  1. … 1739 more files in changeset.