Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

    • -437
    • +0
    ./drill/exec/memory/AccountorImpl.java
    • -38
    • +0
    ./drill/exec/memory/AllocationPolicy.java
    • -69
    • +0
    ./drill/exec/memory/AllocationPolicyAgent.java
    • -152
    • +0
    ./drill/exec/memory/AllocationReservation.java
    • -31
    • +0
    ./drill/exec/memory/AllocatorClosedException.java
    • -40
    • +0
    ./drill/exec/memory/AllocatorOwner.java
    • -34
    • +0
    ./drill/exec/memory/AllocatorsStatsMXBean.java
    • -215
    • +0
    ./drill/exec/memory/AtomicRemainder.java
    • -1654
    • +0
    ./drill/exec/memory/BaseAllocator.java
    • -94
    • +0
    ./drill/exec/memory/BufferLedger.java
    • -60
    • +0
    ./drill/exec/memory/ChainedAllocatorOwner.java
    • -47
    • +0
    ./drill/exec/memory/ChildAllocator.java
    • -125
    • +0
    ./drill/exec/memory/RootAllocator.java
    • -43
    • +0
    ./drill/exec/memory/RootAllocatorFactory.java
    • -408
    • +0
    ./drill/exec/memory/TopLevelAllocator.java
  1. … 105 more files in changeset.
DRILL-4134: Add new allocator

    • -0
    • +38
    ./drill/exec/memory/AllocationPolicy.java
    • -0
    • +69
    ./drill/exec/memory/AllocationPolicyAgent.java
    • -0
    • +152
    ./drill/exec/memory/AllocationReservation.java
    • -0
    • +31
    ./drill/exec/memory/AllocatorClosedException.java
    • -0
    • +40
    ./drill/exec/memory/AllocatorOwner.java
    • -0
    • +34
    ./drill/exec/memory/AllocatorsStatsMXBean.java
    • -0
    • +1654
    ./drill/exec/memory/BaseAllocator.java
    • -0
    • +94
    ./drill/exec/memory/BufferLedger.java
    • -0
    • +60
    ./drill/exec/memory/ChainedAllocatorOwner.java
    • -0
    • +47
    ./drill/exec/memory/ChildAllocator.java
    • -0
    • +125
    ./drill/exec/memory/RootAllocator.java
    • -28
    • +4
    ./drill/exec/memory/RootAllocatorFactory.java
    • -71
    • +89
    ./drill/exec/memory/TopLevelAllocator.java
  1. … 16 more files in changeset.
DRILL-3987: (CLEANUP) Final cleanups to get complete working build/distribution

- small cleanups

- move Hook to drill-adbc

- update distribution assembly to include new modules

This closes #250

    • -3
    • +5
    ./drill/exec/memory/TopLevelAllocator.java
  1. … 31 more files in changeset.
DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separate modules.

    • -0
    • +437
    ./drill/exec/memory/AccountorImpl.java
    • -0
    • +215
    ./drill/exec/memory/AtomicRemainder.java
    • -0
    • +67
    ./drill/exec/memory/RootAllocatorFactory.java
    • -0
    • +388
    ./drill/exec/memory/TopLevelAllocator.java
  1. … 138 more files in changeset.