Clone Tools
  • last updated 27 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7442: Create multi-batch row set reader

Adds a ResultSetReader that works across multiple batches

in a result set. Reuses the same row set and readers if

schema is unchanged, creates a new set if the schema changes.

Adds a unit test for the result set reader.

Adds a "rebind" capability to the row set readers to focus

on new buffers under an existing set of vectors. Used when

a new batch arrives, if the schema is unchanged.

Extends row set classses to be aware of the BatchAccessor class

which encapsulates a container and optional selection vector,

and tracks schema changes.

Moves row set tests into the same package as the row sets.

(Row set classes were moved a while back, but the tests were

not moved.)

Renames some BatchAccessor methods.

closes #1897

  1. … 62 more files in changeset.
DRILL-7019: Add check for redundant imports

close apache/drill#1629

  1. … 23 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

  1. … 984 more files in changeset.
DRILL-6389: Fixed building javadocs - Added documentation about how to build javadocs - Fixed some of the javadoc warnings

closes #1276

  1. … 65 more files in changeset.
DRILL-5846: Improve parquet performance for Flat Data Types

closes #1060

  1. … 26 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

    • -1
    • +1
    ./netty/buffer/MutableWrappedByteBuf.java
  1. … 2063 more files in changeset.
DRILL-6230: Extend row set readers to handle hyper vectors

closes #1161

  1. … 64 more files in changeset.
DRILL-6202: Deprecate usage of IndexOutOfBoundsException to re-alloc vectors

closes #1144

  1. … 9 more files in changeset.
DRILL-6080: Sort incorrectly limits batch size to 65535 records

closes #1090

* Sort incorrectly limits batch size to 65535 records rather than 65536.

* This PR also includes a few code cleanup items.

* Fix for overflow in offset vector in row set writer

* Performance tool update

* Replace "unsafe" methods with "set" methods

* Also fixes an indexing issue with nullable writers

* Removed debug & timing code

* Increase strictness for batch size

  1. … 10 more files in changeset.
DRILL-6004: Direct buffer bounds checking should be disabled by default

This closes #1070

  1. … 7 more files in changeset.
DRILL-5830: Resolve regressions to MapR DB from DRILL-5546

- Back out HBase changes

- Code cleanup

- Test utilities

- Fix for DRILL-5829

closes #968

  1. … 22 more files in changeset.
DRILL-5657: Size-aware vector writer structure

- Vector and accessor layer

- Row Set layer

- Tuple and column models

- Revised write-time metadata

- "Result set loader" layer

this closes #914

    • -29
    • +33
    ./netty/buffer/PooledByteBufAllocatorL.java
  1. … 186 more files in changeset.
DRILL-5431: Upgrade Netty to 4.0.47

    • -9
    • +31
    ./netty/buffer/UnsafeDirectLittleEndian.java
  1. … 1 more file in changeset.
DRILL-5517: Size-aware set methods in value vectors

Please see DRILL-5517 for an explanation.

Also includes a workaround for DRILL-5529.

Implements a setEmpties method for repeated and non-nullable

variable-width types in support of the revised column accessors.

Unit test included. Without the setEmpties call, the tests fail with

vector corruption. With the call, things work properly.

closes #840

    • -19
    • +56
    ./netty/buffer/UnsafeDirectLittleEndian.java
  1. … 24 more files in changeset.
DRILL-5601: Rollup of external sort fixes an improvements

- DRILL-5513: Managed External Sort : OOM error during the merge phase

- DRILL-5519: Sort fails to spill and results in an OOM

- DRILL-5522: OOM during the merge and spill process of the managed external sort

- DRILL-5594: Excessive buffer reallocations during merge phase of external sort

- DRILL-5597: Incorrect "bits" vector allocation in nullable vectors allocateNew()

- DRILL-5602: Repeated List Vector fails to initialize the offset vector

- DRILL-5617: Spill file name collisions when spill file is on a shared file system

0 DRILL-5445: bug in repeated map vector deserialization

- Workaround for DRILL-5656: Streaming Agg Batch forces sort to retain in-memory batches past NONE

- Fixes for the "record batch sizer" to handle for UNION, MAP, LIST types

Fixes a longstanding bug in the deserialization of a repeated map

vector read from a spill file. A few minor code cleanups also.

All of the bugs have to do with handling low-memory conditions, and with

correctly estimating the sizes of vectors, even when those vectors come

from the spill file or from an exchange. Hence, the changes for all of

the above issues are interrelated.

Also includes some fixes for tests:

* Certain tests require the ability to enforce the output size of the

memory merge/sort. Restored this option.

* Resolve issue with TestDrillbitResilience

* A particular test injects a fault during in-memory sort, but used a

single-batch input (which does not need the merge phase.)

Rather than introduce a new config property (the earlier solution),

altered the test to use input that returns more than one batch.

Two fixes forBasicPhysicalOpUnitTest — a test that uses JMockit to create a “fake”

fragment context.

1. No drillbit endpoint is available, so the SpillSet change that adds

the node to the spill path failed. Solution was to omit the node path

segment in such tests.

2. The Dynamic UDF registry is null causing a crash. This has nothing

to do with sort. Perhaps some pre-existing error? Anyway, added a check

for this condition.

close #860

  1. … 63 more files in changeset.
DRILL-4131: Move RPC allocators under Drill's root allocator & accounting

- Allow settings to be set to ensure RPC reservation and maximums (currently unset by default). Defaults set in drill-module.conf

- Add new metrics to report RPC layer memory consumption.

- Check for memory leaks from RPC layer at shutdown.

- Add a multi-Drillbit single JVM safe DrillMetrics.register()

- Remove invalid verifyAllocator checks while RPC connection (and PING/PONG) are maintained

This closes #327.

  1. … 11 more files in changeset.
DRILL-4246: Fix Allocator concurrency bug and improve error detection

- Rename the internal DrillBuf field to udle to better express its purpose.

- Rename AllocatorManager to AllocationManager to better express its purpose.

- Address situation where dangling ledger could be transferred into while it was being released released by protecting association and release inside the AllocationManager.

- Add allocator assertions to ensure allocator operations are done while the allocator is open.

- Simplify AllocationManager locking model.

- Exclude HDFS reference to netty-all

- Improve debugging messages for allocators (and fix debug message bugs)

This closes #323.

  1. … 6 more files in changeset.
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

    • -527
    • +187
    ./netty/buffer/DrillBuf.java
    • -0
    • +55
    ./netty/buffer/ExpandableByteBuf.java
    • -167
    • +0
    ./netty/buffer/FakeAllocator.java
    • -0
    • +336
    ./netty/buffer/MutableWrappedByteBuf.java
    • -151
    • +161
    ./netty/buffer/PooledByteBufAllocatorL.java
    • -182
    • +134
    ./netty/buffer/UnsafeDirectLittleEndian.java
  1. … 113 more files in changeset.
DRILL-4134: Add new allocator

    • -101
    • +570
    ./netty/buffer/DrillBuf.java
    • -44
    • +24
    ./netty/buffer/FakeAllocator.java
    • -1
    • +59
    ./netty/buffer/UnsafeDirectLittleEndian.java
  1. … 26 more files in changeset.
DRILL-3987: (REFACTOR) Remove any parent Drill dependencies for drill-memory-base.

  1. … 2 more files in changeset.
DRILL-3987: (REFACTOR) Extract BoundsChecking check from AssertionUtil. Remove unused file.

  1. … 7 more files in changeset.
DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separate modules.

    • -0
    • +749
    ./netty/buffer/DrillBuf.java
    • -0
    • +187
    ./netty/buffer/FakeAllocator.java
    • -0
    • +350
    ./netty/buffer/LargeBuffer.java
    • -0
    • +257
    ./netty/buffer/PooledByteBufAllocatorL.java
    • -0
    • +247
    ./netty/buffer/UnsafeDirectLittleEndian.java
  1. … 137 more files in changeset.