memory

closes #1207

closes #1045

This change includes:

DRILL-5783:

- A unit test is created for the priority queue in the TopN operator.

- The code generation classes passed around a completely unused function registry reference in some places so it is removed.

- The priority queue had unused parameters for some of its methods so it is removed.

DRILL-5841:

- Created standardized temp directory classes DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. And updated all unit tests to use them.

DRILL-5894:

- Removed the dfs_test storage plugin for tests and replaced it with the already existing dfs storage plugin.

Misc:

- General code cleanup.

- Removed unnecessary use of String.format in the tests.

This closes #984

1. Increased test parallelism and fixed associated bugs

2. Added test categories and categorized tests appropriately

- Don't exclude anything by default

- Increase test timeout

- Fixed flakey test

closes #940

Refactors ScanBatch to allow unit testing of record reader

implementations, especially the “writer” classes.

See JIRA for details.

closes #811

- Create the storage plugin registry using a Drill configuration parameter to be able to replace the registry implementation.

- Change StoragePluginRegistry into an interface and move the implementation to an impl class.

- Write documentation for StoragePluginRegistry.

- Make StoragePluginRegistry and StoragePluginMap AutoCloseable and ensure that Drillbit closes registry.

- Misc Drillbit code reorganization so that fields are at top of class (and static methods at bottom).

- Update DrillConfig to support reflection-based creation of storage plugin objects

- Remove final from DrillConfig so that application developers can extend config object.

This closes #321.

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

Makes the classpath scanning a build time class discovery

Makes the fmpp generation incremental

Removes some slowness in DrillBit closing

Reduces the build time by 30%

This closes #148

this closes #199

this closes #163

Delete a test that had been commented out for over a year, it no longer compiles due to interface changes and there is plenty of other testing for hash aggregate.

+ Rename: FragmentStats#getOperatorStats => newOperatorStats

+ Documentation

this closes #216

- Remove cleanup method from RecordBatch interface

- Make OperatorContext creation and closing the management of FragmentContext

- Make OperatorContext an abstract class and the impl only available to FragmentContext

- Make RecordBatch closing the responsibility of the RootExec

- Make all closes be suppresing closes to maximize memory release in failure

- Add new CloseableRecordBatch interface used by RootExec

- Make RootExec AutoCloseable

- Update RecordBatchCreator to return CloseableRecordBatches so that RootExec can maintain list

- Generate list of operators through change in ImplCreator

TestWriteToDisk

- fix warnings, including closing unclosed resources with try-with-resources

TestAllocators

- close unclosed resources

doesn't use try-with-resources because this stresses interleaved allocation

operations that wouldn't necessarily be nested

StackTrace

- created convenience class for tracking stack traces

Drillbit

- used StackTrace to track location of Drillbit starts for shutdown hook

error messages so we can more easily identify errant tests from just the logs

in future

- added an isClosed flag to detect that the Drillbit has already been closed

to avoid complaints about double closures that happen in many tests that

explicitly close their drillbits (the shutdown hook then closes them again,

causing bogus complaints)

- synchronized close()

- made the id generator in the ShutdownThread an AtomicInteger

Also includes fixes from reviews by Tim.