Clone
 

jacques nadeau <jacques@apache.org> in drill

DRILL-3714: Avoid cascading disconnection when a single connection is broken.

- Move the coordination id management to be connection level instead of RpcBus level

- Rename CoordinationQueue to a more appropriate name: RequestIdMap

- Simplify locking and memory overhead of RequestIdMap. It used to be that this would accessed by a large number of threads concurrently. We modified the behavior so that it is only accessed by two threads at most. Rather than have memory overhead of ConcurrentHashMap, switch to simple locking approach since contention should be low.

- Update all methods associated with coordination to improve names as well as add javadocs. Move these methods to the RemoteConnection.

- Consolidate the two different close handlers into a single, ordered close handler managed inside the connection.

- Add better javadoc around the close method of RemoteConnection

- Add some preconditions checks.

- Update the HPPC version in the base memory module since it conflicts with the one in the java-exec module.

This closes #463.

DRILL-4474: Ensure that ConvertCountToDirectScan only pushes through project when project is trivial. This closes #406

    • binary
    /exec/java-exec/src/test/resources/directcount.parquet
DRILL-4467: Fix field ordering issue in PrelUtil

DRILL-4465: Simplify Calcite parsing & planning integration

- Canonicalize Planning phases with PlannerPhase enumeration

- Canonicalize PlannerType transforms

- Remove dependency on Calcite's Frameworks.Planner since Drill need stop heavily customize interactions

- Update AbstractStoragePlugin to implement a phase-aware planning rule injection behavior.

- Avoid (or at least reduce) duplicated registerSchemas() invocations

This closes #401.

  1. … 12 more files in changeset.
DRILL-4384: Fix Profile regressions

DRILL-4358: Fix NPE in UserServer.close()

- Also remove untested CustomSerDe's from CustomTunnel.

- Fix GuavaPatcher copy-paste comment mistake.

closes #362

DRILL-4297: Enable custom serializers and deserializers when using CustomTunnel

- Adds support for customer serializers and deserializers

- Adds pre-built serializers and deserializers for Protobuf, Jackson, Protostuff (protobuf) and Protostuff (json)

DRILL-3581: Upgrade Jackson to version 2.7.1

DRILL-3581: Upgrade HPPC to 0.7.1

DRILL-4291: Fix Missing classes when trying to query varchar[] using JDBC

- Create a new simplified version of the Hadoop Text class that doesn't include massive dependencies.

- Update Vectors to use new Text class.

- Update the jdbc-all module to have a test which also includes complex types.

- Clean up exclusions in Jdbc jar file to reduce file size.

- Add an enforcer rule that ensures the adbc-all jar exclusions are maintained in the future.

This closes #336.

    • -0
    • +25
    /exec/jdbc-all/src/test/resources/types.json
DRILL-4278: Heap memory leak issues

- Fix issue where WorkspaceConfig was not returning consistent hashCode()s for equal objects.

- Fix issue where we were misusing recycler causing object reference leaks

This closes #331.

DRILL-4131: Move RPC allocators under Drill's root allocator & accounting

- Allow settings to be set to ensure RPC reservation and maximums (currently unset by default). Defaults set in drill-module.conf

- Add new metrics to report RPC layer memory consumption.

- Check for memory leaks from RPC layer at shutdown.

- Add a multi-Drillbit single JVM safe DrillMetrics.register()

- Remove invalid verifyAllocator checks while RPC connection (and PING/PONG) are maintained

This closes #327.

DRILL-4277: Fix for JdbcPrel serialization issue.

This closes #326.

    • -0
    • +1
    /contrib/storage-jdbc/src/test/resources/jdbcmulti/file1.json
    • -0
    • +1
    /contrib/storage-jdbc/src/test/resources/jdbcmulti/file2.json
DRILL-4246: Fix Allocator concurrency bug and improve error detection

- Rename the internal DrillBuf field to udle to better express its purpose.

- Rename AllocatorManager to AllocationManager to better express its purpose.

- Address situation where dangling ledger could be transferred into while it was being released released by protecting association and release inside the AllocationManager.

- Add allocator assertions to ensure allocator operations are done while the allocator is open.

- Simplify AllocationManager locking model.

- Exclude HDFS reference to netty-all

- Improve debugging messages for allocators (and fix debug message bugs)

This closes #323.

DRILL-4257: Fix StoragePluginRegistry clean-up behavior and misc clean up.

- Create the storage plugin registry using a Drill configuration parameter to be able to replace the registry implementation.

- Change StoragePluginRegistry into an interface and move the implementation to an impl class.

- Write documentation for StoragePluginRegistry.

- Make StoragePluginRegistry and StoragePluginMap AutoCloseable and ensure that Drillbit closes registry.

- Misc Drillbit code reorganization so that fields are at top of class (and static methods at bottom).

- Update DrillConfig to support reflection-based creation of storage plugin objects

- Remove final from DrillConfig so that application developers can extend config object.

This closes #321.

Fix issue where FutureBitComment didn't return buffer as part of future.

DRILL-4238: Add a custom RPC interface on the Control channel for extensible communication between bits.

This closes #313.

    • -0
    • +7
    /protocol/src/main/protobuf/BitControl.proto
Blog post for Drill 1.4 release

    • -0
    • +53
    /blog/_posts/2015-12-14-drill-1.4-released.md
DRILL-4241: Add Single Tablet Writer

- Also move to a test bootstrap

- Update to the latest kudu and Drill

- Add plugin to Drill distribution

- Checkstyle and directory cleanup

This closes #314.

    • -0
    • +23
    /contrib/storage-kudu/src/main/codegen/config.fmpp
  1. … 3 more files in changeset.
DRILL-4241: Add table metadata and DROP table support

[maven-release-plugin] prepare release drill-1.3.0

    • -3
    • +2
    /contrib/data/tpch-sample-data/pom.xml
  1. … 6 more files in changeset.
DRILL-4134: Allocator Improvements

- make Allocator mostly lockless

- change BaseAllocator maps to direct references

- add documentation around memory management model

- move transfer and ownership methods to DrillBuf

- Improve debug messaging.

- Fix/revert sort changes

- Remove unused fragment limit flag

- Add time to HistoricalLog events

- Remove reservation amount from RootAllocator constructor (since not allowed)

- Fix concurrency issue where allocator is closing at same moment as incoming batch transfer, causing leaked memory and/or query failure.

- Add new AutoCloseables.close(Iterable<AutoCloseable>)

- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to use smarter mock of FragmentManager)

- Remove the concept of poison pill record batches, using instead FragmentContext.isOverMemoryLimit()

- Update incoming data batches so that they are transferred under protection of a close lock

- Improve field names in IncomingBuffers and move synchronization to collectors as opposed to IncomingBuffers (also change decrementing to decrementToZero rather than two part check).

This closes #238.

  1. … 105 more files in changeset.
Roll forward to 1.4.0-SNAPSHOT

    • -1
    • +1
    /contrib/data/tpch-sample-data/pom.xml
  1. … 12 more files in changeset.
DRILL-4065: Maintain a cache of created UserGroupInformation objects so that we don't accidentally generate thousands of FileSystem objects & threads.

DRILL-4065: Maintain a cache of created UserGroupInformation objects so that we don't accidentally generate thousands of FileSystem objects & threads.

DRILL-3987: (CLEANUP) Final cleanups to get complete working build/distribution

- small cleanups

- move Hook to drill-adbc

- update distribution assembly to include new modules

This closes #250

    • -1
    • +13
    /common/src/main/resources/drill-module.conf
  1. … 17 more files in changeset.
DRILL-3987: (MOVE) Move logical expressions and operators out of common. Move to new drill-logical model.

  1. … 199 more files in changeset.
DRILL-3987: (REFACTOR) Remove any parent Drill dependencies for drill-memory-base.

DRILL-3987: (CLEANUP) Delete unused files

DRILL-3987: (REFACTOR) Extract BoundsChecking check from AssertionUtil. Remove unused file.