Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-5749: solve deadlock occured between foreman and netty threads

closes #943

DRILL-5425: Support HTTP Kerberos auth using SPNEGO

closes #1040

  1. … 25 more files in changeset.
DRILL-5701: Fix drill.connections.rpc.<user/control/data>.<encrypted/unencrypted> metric behavior

closes #894

  1. … 10 more files in changeset.
DRILL-5599: Notify StatusHandler that batch sending has failed even if channel is still open

close #857

DRILL-5457: Spill implementation for Hash Aggregate

closes #822

  1. … 35 more files in changeset.
DRILL-5431: SSL Support (Java) - Java client server SSL implementation

Also enable OpenSSL support

Also fix exclusions and java-exec pom file to eliminate netty-tcnative as a transitive dependency on all projects

    • -9
    • +18
    ./drill/exec/rpc/AbstractRemoteConnection.java
    • -108
    • +41
    ./drill/exec/rpc/BasicClient.java
    • -0
    • +235
    ./drill/exec/rpc/ConnectionMultiListener.java
  1. … 26 more files in changeset.
DRILL-4280: CORE (revert DRILL-3242)

+ DRILL-3242 aims to provide offloading request handling to a secondary thread, but this feature is disabled by default due to concurrency issues

+ One of the implications of the feature was to ignore exceptions that were not of UserRpcException type. But exceptions must not be ignored, they should be handled properly, specially in the context of security

DRILL-4335: Apache Drill should support network encryption.

NOTE: This pull request provides support for on-wire encryption using SASL framework. The communication channel that are covered are:

1) Between Drill JDBC client and Drillbit.

2) Between Drillbit to Drillbit i.e. control/data channels.

3) It has UI change to view encryption is enabled on which network channel and number of encrypted/unencrypted connections for

user/control/data connections.

close apache/drill#773

    • -3
    • +105
    ./drill/exec/rpc/AbstractRemoteConnection.java
    • -0
    • +99
    ./drill/exec/rpc/ChunkCreationHandler.java
    • -1
    • +4
    ./drill/exec/rpc/ClientConnection.java
    • -0
    • +37
    ./drill/exec/rpc/EncryptionContext.java
    • -0
    • +95
    ./drill/exec/rpc/EncryptionContextImpl.java
    • -0
    • +2
    ./drill/exec/rpc/RemoteConnection.java
    • -0
    • +29
    ./drill/exec/rpc/RpcMetrics.java
    • -0
    • +34
    ./drill/exec/rpc/SaslCodec.java
    • -0
    • +160
    ./drill/exec/rpc/SaslDecryptionHandler.java
    • -0
    • +177
    ./drill/exec/rpc/SaslEncryptionHandler.java
    • -1
    • +3
    ./drill/exec/rpc/ServerConnection.java
  1. … 46 more files in changeset.
DRILL-4280: REFACTOR

+ Extract RemoteConnection interface, and add AbstractRemoteConnection

+ Add ServerConnection and ClientConnection interfaces

+ Add RequestHandler interface to decouple connections from how requests are handled

+ Add NonTransientRpcException

+ Remove unused classes and methods

+ Code style changes

    • -0
    • +227
    ./drill/exec/rpc/AbstractRemoteConnection.java
    • -35
    • +53
    ./drill/exec/rpc/BasicClient.java
    • -39
    • +24
    ./drill/exec/rpc/BasicServer.java
    • -0
    • +30
    ./drill/exec/rpc/ClientConnection.java
    • -29
    • +0
    ./drill/exec/rpc/InvalidConnectionInfoException.java
    • -0
    • +33
    ./drill/exec/rpc/NonTransientRpcException.java
    • -66
    • +24
    ./drill/exec/rpc/ReconnectingConnection.java
    • -194
    • +31
    ./drill/exec/rpc/RemoteConnection.java
    • -0
    • +50
    ./drill/exec/rpc/RequestHandler.java
    • -4
    • +4
    ./drill/exec/rpc/RpcConnectionHandler.java
    • -0
    • +37
    ./drill/exec/rpc/ServerConnection.java
  1. … 3 more files in changeset.
DRILL-5098: Improving fault tolerance for connection between client and foreman node.

Adding tries config option in connection string. Improving fault tolerance in Drill client when trying to make first connection with foreman. The client will try to connect to min(tries, num_drillbits) unique drillbits unless a successfull connection is established.

HYGIENE: Refactoring BasicClient::close to call RemoteConnection::close

close apache/drill#679

    • -4
    • +16
    ./drill/exec/rpc/RemoteConnection.java
  1. … 4 more files in changeset.
DRILL-5015: Randomly select the drillbit from the list provided by user in connection string Note: Improved the connection string validation and error handling during parsing. Added unit test for the new parsing mechanism.

close apache/drill#648

    • -0
    • +29
    ./drill/exec/rpc/InvalidConnectionInfoException.java
  1. … 3 more files in changeset.
DRILL-4956: Temporary tables support

close apache/drill#666

    • -1
    • +8
    ./drill/exec/rpc/RemoteConnection.java
  1. … 40 more files in changeset.
DRILL-4654: Add new metrics to the MetricRegistry

+ New metrics:

- drill.queries.enqueued

number of queries that have been submitted to the drillbit but have

not started

- drill.queries.running

number of running queries for which this drillbit is the foreman

- drill.queries.completed

number of completed queries (or cancelled or failed) for which this

drillbit was the foreman

- drill.fragments.running

number of query fragments that are running in the drillbit

- drill.allocator.root.used

amount of memory used in bytes by the internal memory allocator

- drill.allocator.root.peak

peak amount of memory used in bytes by the internal memory allocator

- fd.usage

ratio of used to total file descriptors (on *nix systems)

+ Rename "current" to "used" for RPC allocator current memory usage to

follow convention

+ Borrow SystemPropertyUtil class from Netty

+ Configure DrillMetrics through system properties

+ Remove unused methods and imports

closes #495

  1. … 13 more files in changeset.
DRILL-4623: Disable epoll transport by default

closes #486

  1. … 1 more file in changeset.
DRILL-3714: Avoid cascading disconnection when a single connection is broken.

- Move the coordination id management to be connection level instead of RpcBus level

- Rename CoordinationQueue to a more appropriate name: RequestIdMap

- Simplify locking and memory overhead of RequestIdMap. It used to be that this would accessed by a large number of threads concurrently. We modified the behavior so that it is only accessed by two threads at most. Rather than have memory overhead of ConcurrentHashMap, switch to simple locking approach since contention should be low.

- Update all methods associated with coordination to improve names as well as add javadocs. Move these methods to the RemoteConnection.

- Consolidate the two different close handlers into a single, ordered close handler managed inside the connection.

- Add better javadoc around the close method of RemoteConnection

- Add some preconditions checks.

- Update the HPPC version in the base memory module since it conflicts with the one in the java-exec module.

This closes #463.

    • -160
    • +0
    ./drill/exec/rpc/CoordinationQueue.java
    • -39
    • +0
    ./drill/exec/rpc/PositiveAtomicInteger.java
    • -30
    • +81
    ./drill/exec/rpc/RemoteConnection.java
    • -0
    • +192
    ./drill/exec/rpc/RequestIdMap.java
  1. … 1 more file in changeset.
DRILL-4327: Fix rawtypes warnings in drill codebase

Fixing most rawtypes warning issues in drill modules.

Closes #347

  1. … 76 more files in changeset.
DRILL-4278: Heap memory leak issues

- Fix issue where WorkspaceConfig was not returning consistent hashCode()s for equal objects.

- Fix issue where we were misusing recycler causing object reference leaks

This closes #331.

    • -21
    • +2
    ./drill/common/SerializedExecutor.java
  1. … 4 more files in changeset.
Fix issue where FutureBitComment didn't return buffer as part of future.

    • -0
    • +1
    ./drill/exec/rpc/FutureBitCommand.java
    • -3
    • +6
    ./drill/exec/rpc/RpcCheckedFuture.java
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

  1. … 119 more files in changeset.
DRILL-3987: (MOVE) Move logical expressions and operators out of common. Move to new drill-logical model.

    • -0
    • +144
    ./drill/common/SerializedExecutor.java
  1. … 213 more files in changeset.
DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separate modules.

    • -0
    • +69
    ./drill/exec/rpc/AbstractHandshakeHandler.java
    • -0
    • +27
    ./drill/exec/rpc/Acks.java
    • -0
    • +41
    ./drill/exec/rpc/BaseRpcOutcomeListener.java
    • -0
    • +316
    ./drill/exec/rpc/BasicClient.java
    • -0
    • +76
    ./drill/exec/rpc/BasicClientWithConnection.java
    • -0
    • +237
    ./drill/exec/rpc/BasicServer.java
    • -0
    • +39
    ./drill/exec/rpc/ChannelClosedException.java
    • -0
    • +25
    ./drill/exec/rpc/ChannelListenerWithCoordinationId.java
    • -0
    • +22
    ./drill/exec/rpc/ConnectionThrottle.java
    • -0
    • +160
    ./drill/exec/rpc/CoordinationQueue.java
    • -0
    • +28
    ./drill/exec/rpc/DrillRpcFuture.java
    • -0
    • +80
    ./drill/exec/rpc/DrillRpcFutureImpl.java
    • -0
    • +79
    ./drill/exec/rpc/FutureBitCommand.java
    • -0
    • +66
    ./drill/exec/rpc/InboundRpcMessage.java
    • -0
    • +70
    ./drill/exec/rpc/ListeningCommand.java
  1. … 127 more files in changeset.
DRILL-4085: Disable RPC thread offload until concurrency bug is found.

DRILL-3581: Upgrade to Guava 18.0

- Replace Stopwatch constructors with .createStarted() or .createUnstarted()

- Stop using InputSupplier and Closeables.closeQuietly

- Clean up quiet closes to log or (preferably) propagate.

- Add log4j to enforcer exclusions.

- Update HBaseTestSuite to add patching of Closeables.closeQuietly() and Stopwatch legacy methods. Only needed when running HBaseMiniCluster.

- Remove log4j from HBase's pom to provide exception logging.

- Remove log4j from Hive's shaded pom.

- Update Catastrophic failures to use the same pattern to ensure reporting.

- Update test framework to avoid trying IPv6 resolution. (This removes 90s pause from HBase startup in my tests)

This closes #361.

This closes #157.

  1. … 64 more files in changeset.