Clone Tools
  • last updated a few minutes ago
Constraints: committers
Constraints: files
Constraints: dates
DRILL-6999: Fix the case that there's more than one join conditions

closes #1600

  1. … 1 more file in changeset.
DRILL-6785: DataClient is using RootAllocator in the bootstrap instead of dataPool

closes #1502

DRILL-6422: Replace guava imports with shaded ones

  1. … 982 more files in changeset.
DRILL-6385: Support JPPD feature

  1. … 61 more files in changeset.
DRILL-6386: Remove unused imports and star imports.

  1. … 231 more files in changeset.
DRILL-6255: Drillbit while sending control message to itself creates a connection instead of submitting locally

closes #1253

  1. … 18 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2054 more files in changeset.
DRILL-6187: Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on

This closes #1145

  1. … 22 more files in changeset.
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

  1. … 223 more files in changeset.
DRILL-5701: Fix drill.connections.rpc.<user/control/data>.<encrypted/unencrypted> metric behavior

closes #894

  1. … 8 more files in changeset.
DRILL-4335: Apache Drill should support network encryption.

NOTE: This pull request provides support for on-wire encryption using SASL framework. The communication channel that are covered are:

1) Between Drill JDBC client and Drillbit.

2) Between Drillbit to Drillbit i.e. control/data channels.

3) It has UI change to view encryption is enabled on which network channel and number of encrypted/unencrypted connections for

user/control/data connections.

close apache/drill#773

    • -0
    • +85
  1. … 55 more files in changeset.
DRILL-4280: CORE (bit to bit authentication, data)

+ Support authentication in DataServer and DataClient

+ Add AuthenticationCommand as an initial command after handshake

and before the command that initiates a connection

+ Add DataConnectionConfig to encapsulate configuration

+ Add DataServerRequestHandler to encapsulate all handling of

requests to DataServer


    • -0
    • +46
    • -0
    • +42
    • -0
    • +115
DRILL-4979: Make port of the DataConnection configurable

This closes #649

  1. … 1 more file in changeset.
DRILL-4499: Remove 16 unused classes

This closes #426

  1. … 15 more files in changeset.
DRILL-4131: Move RPC allocators under Drill's root allocator & accounting

- Allow settings to be set to ensure RPC reservation and maximums (currently unset by default). Defaults set in drill-module.conf

- Add new metrics to report RPC layer memory consumption.

- Check for memory leaks from RPC layer at shutdown.

- Add a multi-Drillbit single JVM safe DrillMetrics.register()

- Remove invalid verifyAllocator checks while RPC connection (and PING/PONG) are maintained

This closes #327.

  1. … 11 more files in changeset.
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

    • -0
    • +73
  1. … 114 more files in changeset.
DRILL-4134: Add new allocator

  1. … 28 more files in changeset.
DRILL-3742: Classpath scanning and build improvement

Makes the classpath scanning a build time class discovery

Makes the fmpp generation incremental

Removes some slowness in DrillBit closing

Reduces the build time by 30%

This closes #148

  1. … 143 more files in changeset.
DRILL-3581: Upgrade to Guava 18.0

- Replace Stopwatch constructors with .createStarted() or .createUnstarted()

- Stop using InputSupplier and Closeables.closeQuietly

- Clean up quiet closes to log or (preferably) propagate.

- Add log4j to enforcer exclusions.

- Update HBaseTestSuite to add patching of Closeables.closeQuietly() and Stopwatch legacy methods. Only needed when running HBaseMiniCluster.

- Remove log4j from HBase's pom to provide exception logging.

- Remove log4j from Hive's shaded pom.

- Update Catastrophic failures to use the same pattern to ensure reporting.

- Update test framework to avoid trying IPv6 resolution. (This removes 90s pause from HBase startup in my tests)

This closes #361.

This closes #157.

  1. … 64 more files in changeset.
DRILL-3242: Update RPC layer so that requests and response are managed on a secondary thread. - Create a separate serialized executor for fragment receiverFinished events. - Update serialized executor to pool object creation. - Ensure that FragmentExecutor acceptExternalEvents countdown occurs when only execution is cancellation.

  1. … 17 more files in changeset.
DRILL-3035: Created ControlsInjector interface to enforce method implementations + DRILL-2867: Add ControlsValidator to VALIDATORS only if assertions are enabled + return in ExecutionControls ctor if assertions are not enabled + added InjectorFactory class to align with the logger pattern

  1. … 22 more files in changeset.
DRILL-3147: tpcds-sf1-parquet query 73 causes memory leak

- each time a fragment A sends a "receiver finished" to fragment B, fragment B id will be

added to FragmentContext.ignoredSenders list

- refactored UnorderedReceiverBatch.informSenders() and MergingRecordBatch.informSenders()

by moving this method to FragmentContext

- DataServer.send() uses FragmentContext.ignoredSenders to decide if a batch should be

passed to the fragment or discarded right away

- BaseRawBatchBuffer methods enqueue() and kill() are now synchronized

- TestTpcdsSf1Leak test reproduces the leak, it's ignored by default because it requires

a large dataset

  1. … 5 more files in changeset.
DRILL-3081: Populate connection name as late as possible so RPC error messages are reported correctly.

  1. … 9 more files in changeset.
DRILL-3046: Release data buffers when interrupt occurs in DataTunnel.sendRecordBatch

  1. … 1 more file in changeset.
DRILL-3061: Fix memory leaks in TestDrillbitResilience

- fixes a race condition in WorkEventBus

- marking TestDrillbitResilience with @Ignore

  1. … 3 more files in changeset.
DRILL-2755: (part2) Use and handle InterruptedException during query processing

  1. … 26 more files in changeset.
DRILL-2971, DRILL-2886, DRILL-2778, DRILL-2545: Improve RPC connection detection failure. Add RPC timeout.

  1. … 27 more files in changeset.
DRILL-2755: Use and handle InterruptedException during query processing.

- Interrupt FragmentExecutor thread as part of FragmentExecutor.cancel()

- Handle InterruptedException in ExternalSortBatch.newSV2(). If the fragment status says

should not continue, then throw the InterruptedException to caller which returns IterOutcome.STOP

- Add comments reg not handling of InterruptedException in SendingAccountor.waitForSendComplete()

- Handle InterruptedException in OrderedPartitionRecordBatch.getPartitionVectors()

If interrupted in Thread.sleep calls and fragment status says should not run, then

return IterOutcome.STOP downstream.

- Interrupt partitioner threads if PartitionerRecordBatch is interrupted while waiting for

partitioner threads to complete.

- Preserve interrupt status if not handled

- Handle null RecordBatches returned by RawBatchBuffer.getNext() in MergingRecordBatch.buildSchema()

- Change timeout in Foreman to be proportional to the number of intermediate fragments sent instead

of hard coded limit of 90s.

- Change TimedRunnable to enforce a timeout of 15s per runnable.

Total timeout is (5s * numOfRunnableTasks) / parallelism.

- Add unit tests

* Testing cancelling a query interrupts the query fragments which are currently blocked

* Testing interrupting the partitioner sender which in turn interrupts its helper threads

* Testing TimedRunanble enforeces timeout for the whole task list.

  1. … 28 more files in changeset.
DRILL-2940, DRILL-2847: Improve Memory Characteristics

- Update Large Buffer allocation so Drill releases immediately rather than waiting for Garbage Collection

- Remove DrillBuf.wrap() and all references to it.

- Update Parquet Reader to reduce object churn and indirection.

- Add additional metric to memory iterator

- Add Large and small buffer metric historgram tracking

- Add memory tracking reporter

- Update Netty to 4.0.27

  1. … 32 more files in changeset.
DRILL-2762: Update Fragment state reporting and error collection


- Add new throwAndClear operation on to allow checking for exceptions preClose in FragmentContext

- Add new getAndClear operation


- Ensure close() can be called multiple times by clearing managed buffer list on close().


- Update FragmentContext to have a preClose so that we can check closure state before doing final close.

- Update so that there is only a single state maintained between FragmentContext and FragmentExecutor

- Clean up FragmentExecutor run() method to better manage error states and have only single terminal point (avoiding multiple messages to Foreman).

- Add new CANCELLATION_REQUESTED state for FragmentState.

- Move all users of isCancelled or isFailed in main code to use shouldContinue()

- Update receivingFragmentFinished message to not cancel fragment (only inform root operator of cancellation)

WorkManager Updates

- Add new afterExecute command to the WorkManager ExecutorService so that we get log entries if a thread leaks an exception. (Otherwise logs don't show these exceptions and they only go to standard out.)

Profile Page

- Update profile page to show last update and last progress.

- Change durations to non-time presentation


- Extract listenable interfaces into anonymous inner classes from body of Foreman


- Update QueryManager to track completed nodes rather than completed fragments using NodeTracker

- Update DrillbitStatusListener to decrement expected completion messages on Nodes that have died to avoid query hang when a node dies


- Add ability to track last status update as well as last time fragment made progress


- Update awareness of current cancellation state to avoid cancellation delays

Misc. Other changes

- Move ByteCode optimization code to only record assembly and code as trace messages

- Update SimpleRootExec to create fake ExecutorState to make existing tests work.

- Update sort to exit prematurely in the case that the fragment was asked to cancel.

- Add finals to all edited files.

- Modify control handler and FragmentManager to directly support receivingFragmentFinished

- Update receiver propagation message to avoid premature removal of fragment manager

- Update UserException.Builder to log a message if we're creating a new UserException (ERROR for System, INFO otherwise).

- Update Profile pages to use min and max instead of sorts.

  1. … 44 more files in changeset.