Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6422: Replace guava imports with shaded ones

  1. … 979 more files in changeset.
DRILL-6656: Disallow extra semicolons and multiple statements on the same line.

closes #1415

  1. … 144 more files in changeset.
DRILL-6386: Remove unused imports and star imports.

  1. … 230 more files in changeset.
DRILL-6255: Drillbit while sending control message to itself creates a connection instead of submitting locally

closes #1253

    • -0
    • +236
    ./LocalControlConnectionManager.java
    • -0
    • +41
    ./RemoteControlConnectionManager.java
  1. … 13 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2057 more files in changeset.
DRILL-6053: Avoid excessive locking in LocalPersistentStore

closes #1163

  1. … 18 more files in changeset.
DRILL-6187: Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on

This closes #1145

  1. … 21 more files in changeset.
DRILL-5961: For long running queries (> 10 min) Drill may raise FragmentSetupException for completed/cancelled fragments

This closes #1041

  1. … 5 more files in changeset.
DRILL-5701: Fix drill.connections.rpc.<user/control/data>.<encrypted/unencrypted> metric behavior

closes #894

  1. … 8 more files in changeset.
DRILL-5355: Misc. code cleanup closes #784

  1. … 23 more files in changeset.
DRILL-4335: Apache Drill should support network encryption.

NOTE: This pull request provides support for on-wire encryption using SASL framework. The communication channel that are covered are:

1) Between Drill JDBC client and Drillbit.

2) Between Drillbit to Drillbit i.e. control/data channels.

3) It has UI change to view encryption is enabled on which network channel and number of encrypted/unencrypted connections for

user/control/data connections.

close apache/drill#773

    • -0
    • +84
    ./ControlRpcMetrics.java
  1. … 56 more files in changeset.
DRILL-4280: CORE (bit to bit authentication, control)

+ Support authentication in ControlServer and ControlClient

+ Add AuthenticationCommand as an initial command after handshake

and before the command that initiates a connection

+ Add ControlConnectionConfig to encapsulate configuration

+ ControlMessageHandler now implements RequestHandler

control

    • -22
    • +16
    ./ConnectionManagerRegistry.java
    • -0
    • +48
    ./ControlConnectionConfig.java
  1. … 7 more files in changeset.
DRILL-4499: Remove 16 unused classes

This closes #426

  1. … 13 more files in changeset.
DRILL-4358: Fix NPE in UserServer.close()

- Also remove untested CustomSerDe's from CustomTunnel.

- Fix GuavaPatcher copy-paste comment mistake.

closes #362

  1. … 2 more files in changeset.
DRILL-4297: Enable custom serializers and deserializers when using CustomTunnel

- Adds support for customer serializers and deserializers

- Adds pre-built serializers and deserializers for Protobuf, Jackson, Protostuff (protobuf) and Protostuff (json)

  1. … 1 more file in changeset.
DRILL-3581: Upgrade HPPC to 0.7.1

  1. … 14 more files in changeset.
DRILL-4131: Move RPC allocators under Drill's root allocator & accounting

- Allow settings to be set to ensure RPC reservation and maximums (currently unset by default). Defaults set in drill-module.conf

- Add new metrics to report RPC layer memory consumption.

- Check for memory leaks from RPC layer at shutdown.

- Add a multi-Drillbit single JVM safe DrillMetrics.register()

- Remove invalid verifyAllocator checks while RPC connection (and PING/PONG) are maintained

This closes #327.

  1. … 8 more files in changeset.
DRILL-4238: Add a custom RPC interface on the Control channel for extensible communication between bits.

This closes #313.

    • -0
    • +124
    ./CustomHandlerRegistry.java
  1. … 6 more files in changeset.
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

  1. … 118 more files in changeset.
DRILL-3581: Upgrade to Guava 18.0

- Replace Stopwatch constructors with .createStarted() or .createUnstarted()

- Stop using InputSupplier and Closeables.closeQuietly

- Clean up quiet closes to log or (preferably) propagate.

- Add log4j to enforcer exclusions.

- Update HBaseTestSuite to add patching of Closeables.closeQuietly() and Stopwatch legacy methods. Only needed when running HBaseMiniCluster.

- Remove log4j from HBase's pom to provide exception logging.

- Remove log4j from Hive's shaded pom.

- Update Catastrophic failures to use the same pattern to ensure reporting.

- Update test framework to avoid trying IPv6 resolution. (This removes 90s pause from HBase startup in my tests)

This closes #361.

This closes #157.

  1. … 63 more files in changeset.
DRILL-2903: General improvements to tests in TestDrillbitResilience + Added RepeatTestRule to tests that are flaky + Added Controls.Builder to create controls string in tests + Added @Ignore to failing tests (filed JIRAs)

Other fixes:

+ Added @Override to ScanBatch#close to avoid potential bugs

+ Added docs link in ProtobufLengthDecoder

+ Fixed logging issue in CountDownLatchImpl

  1. … 17 more files in changeset.
DRILL-3242: Update RPC layer so that requests and response are managed on a secondary thread. - Create a separate serialized executor for fragment receiverFinished events. - Update serialized executor to pool object creation. - Ensure that FragmentExecutor acceptExternalEvents countdown occurs when only execution is cancellation.

  1. … 20 more files in changeset.
DRILL-3081: Populate connection name as late as possible so RPC error messages are reported correctly.

  1. … 9 more files in changeset.
DRILL-3061: Fix memory leaks in TestDrillbitResilience

- fixes a race condition in WorkEventBus

- marking TestDrillbitResilience with @Ignore

  1. … 3 more files in changeset.
DRILL-2977, DRILL-2978: Swap fragment execution method implementations, and cancellation changes

Execution: In WorkManager,

+ swap implementations of startFragmentPendingRemote() and addFragmentRunner()

+ warn if there are running fragments in close()

Cancellation:

+ for fragments waiting on data, delegate cancellations to WorkEventBus (in Foreman and ControlMessageHandler)

+ documentation

  1. … 4 more files in changeset.
DRILL-2971, DRILL-2886, DRILL-2778, DRILL-2545: Improve RPC connection detection failure. Add RPC timeout.

  1. … 28 more files in changeset.
DRILL-2697: Pauses sites wait indefinitely for a resume signal DrillClient sends a resume signal to UserServer. UserServer triggers a resume call in the correct Foreman. Foreman resumes all pauses related to the query through the Control layer.

+ Better error messages and more tests in TestDrillbitResilience and TestPauseInjection

+ Added execution controls to operator context

+ Removed ControlMessageHandler interface, renamed ControlHandlerImpl to ControlMessageHandler

+ Added CountDownLatchInjection, useful in cases like ParititionedSender that spawns multiple threads

  1. … 32 more files in changeset.
DRILL-2383: Support to inject exceptions and pauses in various components of Drill + Controls are fired only if assertions are enabled + Controls can be introduced in any class that has access to FragmentContext/QueryContext + Controls can be fired by altering the DRILLBIT_CONTROL_INJECTIONS session option + Renames: SimulatedExceptions => ExecutionControls, ExceptionInjector => ExecutionControlsInjector + Added injection sites in Foreman, DrillSqlWorker, FragmentExecutor + Unit tests in TestDrillbitResilience, TestExceptionInjection and TestPauseInjection

Other commits included:

+ DRILL-2437: Moved ExecutionControls from DrillbitContext to FragmentContext/QueryContext

+ DRILL-2382: Added address and port to Injection to specify drillbit

+ DRILL-2384: Added QueryState to SingleRowListener and assert that state is COMPLETED while testing

Other edits:

+ Support for short lived session options in SessionOptionManager (using TTL in OptionValidator)

+ Introduced query count in UserSession

+ Added QueryState to queryCompleted() in UserResultsListener to check if COMPLETED/CANCELED

+ Added JSONStringValidator to TypeValidators

+ Log query id as string in DrillClient, WorkEventBus, QueryResultHandler

+ Use try..catch block only around else clause for OptionList in FragmentContext

+ Fixed drillbitContext spelling error in QueryContext

+ Fixed state transition when cancel() before run() in FragmentExecutor

+ Do not call setLocalOption twice in FallbackOptionManager

+ Show explicitly that submitWork() returns queryId in UserServer

+ Updated protocol/readme.txt to include an alternative way to generate sources

  1. … 45 more files in changeset.
DRILL-2245: Clean up query setup and execution kickoff in Foreman/WorkManager in order to ensure consistent handling, and avoid hangs and races, with the goal of improving Drillbit robustness.

I did my best to keep these clean when I split them up, but this core commit

may depend on some minor changes in the hygiene commit that is also

associated with this bug, so either both should be applied, or neither.

The core commit should be applied first.

protocol/pom.xml

- updated protocol buffer compiler version to 2.6

- this made slight modifications to the formats of a few committed protobuf

files

AutoCloseables

- created org.apache.drill.common.AutoCloseables to handle closing these

quietly

BaseTestQuery, and derivatives

- factored out pieces into QueryTestUtil so they can be reused

DeferredException:

- created this so we can collect exceptions during the shutdown process

Drillbit

- uses AutoCloseables for the WorkManager and for the storeProvider

- allow start() to take a RemoteServiceSet

- private, final, formatting

Foreman

- added new state CANCELLATION_REQUESTED (via UserBitShared.proto) to represent

the time between request of a cancellation, and acknowledgement from all

remote endpoints running fragments on a query's behalf

- created ForemanResult to manage interleaving cleanup effects/failure with

query result state

- does not need to implement Comparable

- does not need to implement Closeable

- thread blocking fixes

- add resultSent flag

- add code to log plan fragments with endpoint assignments

- added finals, cleaned up formatting

- do queue management in acquireQuerySemaphore; local tests pass

- rename getContext() to getQueryContext()

- retain DrillbitContext

- a couple of exception injections for testing

- minor formatting

- TODOs

FragmentContext

- added a DeferredException to collect errors during startup/shutdown sequences

FragmentExecutor

- eliminated CancelableQuery

- use the FragmentContext's DeferredException for errors

- common subexpression elimination

- cleaned up

QueryContext

- removed unnecessary functions (with some outside classes tweaked for this)

- finals, formatting

QueryManager

- merge in QueryStatus

- affects Foreman, ../batch/ControlHandlerImpl,

and ../../server/rest/ProfileResources

- made some methods private

- removed unused imports

- add finals and formatting

- variable renaming to improve readability

- formatting

- comments

- TODOs

QueryStatus

- getAsInfo() private

- member renaming

- member access changes

- formatting

- TODOs

QueryTestUtil, BaseTestQuery, TestDrillbitResilience

- make maxWidth a parameter to server startup

SelfCleaningRunnable

- created org.apache.drill.common.SelfCleaningRunnable

SingleRowListener

- created org.apache.drill.SingleRowListener results listener

- use in TestDrillbitResilience

TestComparisonFunctions

- fix not to close the FragmentContext multiple times

TestDrillbitResilience

- created org.apache.drill.exec.server.TestDrillbitResilience to test drillbit

resilience in the face of exceptions and failures during queries

TestWithZookeeper

- factor out work into ZookeeperHelper so that it can be reused by

TestDrillbitResilience

UserBitShared

- get rid of unused UNKNOWN_QUERY

WorkEventBus

- rename methods, affects Foreman and ControlHandlerImpl

- remove unused WorkerBee reference

- most members final

- formatting

WorkManager

- Closeable to AutoCloseable

- removed unused incomingFragments Set

- eliminated unnecessary eventThread and pendingTasks by posting Runnables

directly to executor

- use SelfCleaningRunnable for Foreman management

- FragmentExecutor management uses SelfCleaningRunnable

- runningFragments to be a ConcurrentHashMap; TestTpchDistributed passes

- other improvements due to bee no longer needed in various places

- most members final

- minor formatting

- comments

- TODOs

(*) Created exception injection classes to simulate exceptions for testing

- ExceptionInjection

- ExceptionInjector

- ExceptionInjectionUtil

- TestExceptionInjection

DRILL-2245-hygiene: General code cleanup encountered while working on the rest

of this commit. This includes

- making members final whenever possible

- making members private whenever possible

- making loggers private

- removing unused imports

- removing unused private functions

- removing unused public functions

- removing unused local variables

- removing unused private members

- deleting unused files

- cleaning up formatting

- adding spaces before braces in conditionals and loop bodies

- breaking up overly long lines

- removing extra blank lines

While I tried to keep this clean, this commit may have minor dependencies on

DRILL-2245-core that I missed. The intention is just to break this up for

review purposes. Either both commits should be applied, or neither.

  1. … 91 more files in changeset.
DRILL-2004: Foreman should account for fragment cancellations or query hangs

  1. … 4 more files in changeset.