Clone Tools
  • last updated 13 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7272: Drill Metastore Read / Write API and Drill Iceberg Metastore implementation

1. Drill Metastore Read / Write API.

2. Drill Iceberg Metastore implementation in iceberg-metastore module.

3. Patches Guava Preconditions class for Apache Iceberg.

4. General refactoring.

5. Unit tests.

6. Documentation.

  1. … 97 more files in changeset.
DRILL-7046: Support for loading and parsing new RM config file closes #1652

    • -0
    • +49
    ./ConfigConstants.java
    • -0
    • +33
    ./ConfigFileInfo.java
    • -0
    • +41
    ./DrillExecConfigFileInfo.java
    • -0
    • +41
    ./DrillRMConfigFileInfo.java
  1. … 57 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

  1. … 983 more files in changeset.
DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM

closes #1446

  1. … 23 more files in changeset.
DRILL-6494: Drill Plugins Handler

- Storage Plugins Handler service is used op the Drill start-up stage and it updates storage plugins configs from

storage-plugins-override.conf file. If plugins configs are present in the persistence store - they are updated,

otherwise bootstrap plugins are updated and the result configs are loaded to persistence store. If the enabled

status is absent in the storage-plugins-override.conf file, the last plugin config enabled status persists.

- 'drill.exec.storage.action_on_plugins_override_file' Boot option is added. This is the action, which should be

performed on the storage-plugins-override.conf file after successful updating storage plugins configs.

Possible values are: "none" (default), "rename" and "remove".

- The "NULL" issue with updating Hive plugin config by REST is solved. But clients are still being instantiated for disabled

plugins - DRILL-6412.

- "org.honton.chas.hocon:jackson-dataformat-hocon" library is added for the proper deserializing HOCON conf file

- additional refactoring: "com.typesafe:config" and "org.apache.commons:commons-lang3" are placed into DependencyManagement

block with proper versions; correct properties for metrics in "drill-override-example.conf" are specified

closes #1345

  1. … 33 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2062 more files in changeset.
DRILL-6250: Sqlline start command with password appears in the sqlline.log

closes #1174

DRILL-5730: Mock testing improvements and interface improvements

closes #1045

  1. … 222 more files in changeset.
DRILL-5431: SSL Support (Java) - Java client server SSL implementation

Also enable OpenSSL support

Also fix exclusions and java-exec pom file to eliminate netty-tcnative as a transitive dependency on all projects

  1. … 30 more files in changeset.
DRILL-5560: Create configuration file for distribution specific configuration This closes #848

  1. … 1 more file in changeset.
DRILL-4335: Apache Drill should support network encryption.

NOTE: This pull request provides support for on-wire encryption using SASL framework. The communication channel that are covered are:

1) Between Drill JDBC client and Drillbit.

2) Between Drillbit to Drillbit i.e. control/data channels.

3) It has UI change to view encryption is enabled on which network channel and number of encrypted/unencrypted connections for

user/control/data connections.

close apache/drill#773

  1. … 60 more files in changeset.
DRILL-4280: CORE (user to bit authentication, Java)

+ Add logic for authentication in UserClient and UserServer with

backward compatibility in both directions

+ Add abstract extension to ServerConnection and ClientConnection

+ Add concrete extensions to abstract connections:

BitToUserConnection and UserToBitConnection

+ Add ConnectionConfig interface with abstract and concrete

implementations to encapsulate configuration for server-side

connections

+ Encapsulate all requests handled by UserServer in

UserServerRequestHandler

+ Clear UserSession when connection is closed either by user or

bit

+ Add DrillProperties to encapsulate all connection properties

used during connection time

    • -0
    • +186
    ./DrillProperties.java
  1. … 15 more files in changeset.
DRILL-4726: Dynamic UDF Support

1) Configuration / parsing / options / protos

2) Zookeeper integration

3) Registration / unregistration / lazy-init

4) Unit tests

This closes #574

  1. … 71 more files in changeset.
DRILL-3510: Add ANSI_QUOTES option so that Drill's SQL Parser will recognize ANSI_SQL identifiers

- added supporing of quoting identifiers with DOUBLE_QUOTES or BRACKETS via setting new

sys/sess EnumString option QUOTING_IDENTIFIERS;

- added possibility of setting QUOTING_IDENTIFIERS by the jdbc connection URL string;

- added relevant unit tests;

close #520

  1. … 20 more files in changeset.
DRILL-4257: Fix StoragePluginRegistry clean-up behavior and misc clean up.

- Create the storage plugin registry using a Drill configuration parameter to be able to replace the registry implementation.

- Change StoragePluginRegistry into an interface and move the implementation to an impl class.

- Write documentation for StoragePluginRegistry.

- Make StoragePluginRegistry and StoragePluginMap AutoCloseable and ensure that Drillbit closes registry.

- Misc Drillbit code reorganization so that fields are at top of class (and static methods at bottom).

- Update DrillConfig to support reflection-based creation of storage plugin objects

- Remove final from DrillConfig so that application developers can extend config object.

This closes #321.

  1. … 10 more files in changeset.
DRILL-3987: (MOVE) Move logical expressions and operators out of common. Move to new drill-logical model.

  1. … 213 more files in changeset.
DRILL-3742: Classpath scanning and build improvement

Makes the classpath scanning a build time class discovery

Makes the fmpp generation incremental

Removes some slowness in DrillBit closing

Reduces the build time by 30%

This closes #148

    • -0
    • +65
    ./LogicalPlanPersistence.java
  1. … 141 more files in changeset.
DRILL-3927: use OutOfMemoryException in more places Some code cleanup required for the upcoming introduction of the rewritten direct memory allocator. Chiefly the introduction of OutOfMemoryException handling in a few spots, the use of DrillAutoCloseables for some allocator close() calls, and some other minor cleanup. - removed the exclusion of exec/work from the jdbc-all jar, because it now depends on OutOfMemoryException, and that is derived from FragmentSetupException and Foreman Exception

this closes #199

  1. … 14 more files in changeset.
DRILL-3581: Upgrade to Guava 18.0

- Replace Stopwatch constructors with .createStarted() or .createUnstarted()

- Stop using InputSupplier and Closeables.closeQuietly

- Clean up quiet closes to log or (preferably) propagate.

- Add log4j to enforcer exclusions.

- Update HBaseTestSuite to add patching of Closeables.closeQuietly() and Stopwatch legacy methods. Only needed when running HBaseMiniCluster.

- Remove log4j from HBase's pom to provide exception logging.

- Remove log4j from Hive's shaded pom.

- Update Catastrophic failures to use the same pattern to ensure reporting.

- Update test framework to avoid trying IPv6 resolution. (This removes 90s pause from HBase startup in my tests)

This closes #361.

This closes #157.

  1. … 65 more files in changeset.
DRILL-2743: Parquet file metadata caching

rebasing on top of master required conflict resolution in Parser.tdd and parserImpls.ftl

this closes #114

  1. … 11 more files in changeset.
DRILL-1942-hygiene

- Formatting

- @Overrides

- finals

- some AutoCloseable additions

- new isCancelled() abstract method on FragmentManager, implemented on subclasses

Added missing new abstract method isCancelled()

Close apache/drill#120

  1. … 24 more files in changeset.
DRILL-3496: Updates from review comments

DRILL-3496: Update: Multiple log msgs. -> single, aligned multi-line message.

Main: Changed avoidance of original single hard-to-read list from

first draft's multiple single-item log calls (resulting in multiple log

calls/entries) to single log call with aligned multi-line message.

Also revised message wording, etc.

DRILL-3496: Update 2: Some DEBUG -> INFO; another single multi-line msg.

- logger.debug(...) -> logger.info(...) for config. file messages

- changed drill-module.conf to single multi-line message like

scanForImplementations cases

DRILL-3496: Update 3: Various review comment responses.

- Removed isTraceEnabled/etc. guards.

- Simplified logging line break/indentation code.

- Added some more "final".

- Undid logger cleanup. :-(

  1. … 7 more files in changeset.
DRILL-3496: Augment logging in DrillConfig and classpath scanning.

Main: Added and augmented logging calls.

Code Hygiene:

- Added/revised doc. comments, code comments.

- Renamed some locals.

- Added "final".

- Wrapped lines.

Also fixed currentTimeMillis -> Stopwatch in Drillbit.

Conflicts:

common/src/main/java/org/apache/drill/common/config/DrillConfig.java

  1. … 7 more files in changeset.
DRILL-3483: Clarify CommonConstants' constants. [CommonConstants, DrillConfig, PathScanner]

Renamed constants.

Documented constants.

Removed extraneous "public static final" (redudant and abnormal since in interface).

Edit: Unwrapped a no-longer-long line.

closes #88

  1. … 1 more file in changeset.
DRILL-2006: Updated Text reader. Increases variations of text files Drill can work with.

Text reader is heavily inspired by uniVocity parser although it is now byte based and customized for Drill's memory representations.

Also updated the RecordReader interface so that OperatorContext is presented at setup time rather than being a separate call.

  1. … 43 more files in changeset.
DRILL-1385, along with some cleanup

Cleaned up option handling. This includes using finals, making member variables

private whenever possible, and some formatting.

- fixed a bug in the string formatting for the double range validator

- OptionValidator, OptionValue, and their implementations now conspire not to

allow the creation of malformed options because the OptionType has been added

to validator calls to handle OptionValues that are created on demand.

Started with updated byte code rewrite from Jacques

Fixed several problems with scalar value replacement:

- use consistent ASM api version throughout

- stop using deprecated ASM methods (actually causes bugs)

- visitMethodInsn()

- added a couple of missing super.visitEnd()s

- fixed a couple of minor FindBugs issues

- accounted for required stack size increases when replacing holders for

longs and doubles

- added accounting for frame offsets to cope with long and double local

variables and value holder members

- fixed a few minor bugs found with FindBugs

- stop using carrotlabs' hash map lget() method on shared constant data

- fixed an incorrect use of DUP2 on objectrefs when copying long or double

holder members into locals

- fixed a problem with redundant POP instructions left behind after replacement

- fixed a problem with incorrect DUPs in multiple assignment statements

- fixed a problem with DUP_X1 replacement when handling constants in multiple

assignment statements

- fixed a problem with non-replaced holder member post-decrements

- don't replace holders passed to static functions as "out" parameters

(common with Accessors on repeated value vectors)

- increased the maximum required stack size when transferring holder members to

locals

- changed the code generation block type mappings for constants for external

sorts

- fixed problems handling constant and non-constant member variables in

operator classes

- in general, if a holder is assigned to or from an operator member variable,

it can't be replaced (at least not until we replace those as well)

- Use a derived ASM Analyzer (MethodAnalyzer) and Frame

(AssignmentTrackingFrame) in order to establish relationships between

assignments of holders through chains of local variables. This effectively

back-propagates non-replaceability attributes so that if a holder variable

that can't be replaced is assigned to from another holder variable, that

second one cannot be replaced either, and so on through longer chains of

assignments.

- code for dumping generated source code

- MergeAdapter dumps before and after results of scalar replacement

(if it's on)

- fixed some problems in ReplacingBasicValue by replacing HashSet with

IdentityHashMap

- made loggers private

- added a retry strategy for scalar replacement

if a scalar replacement code rewriting fails, then this will try to

regenerate the bytecode again without the scalar replacement.

- bytecode verification is always on now (required for the retry strategy)

- use system option to determine whether scalar replacement should be used

- default option: if scalar replacement fails, retry without it

- force replacement on or off

- unit tests for the retry strategy are based on a single known failure case,

covered by DRILL-2326.

- add tests TestConvertFunctions to test the three scalar replacement options

for the failing test case (testVarCharReturnTripConvertLogical)

- made it possible to set a SYSTEM option as a java property in Drillbit

- added a command line argument to force scalar replacement to be on during

testing in the rootmost pom.xml

In the course of this, added increased checking of intermediate stages of code

rewriting, as well as logging of classes that cause failures.

- work around a bug in ASM's CheckClassAdapter that doesn't allow for checking

of inner classes

Added comments, tidied up formatting, and added "final" in a number of places.

Signed-off-by: vkorukanti <venki.korukanti@gmail.com>

  1. … 53 more files in changeset.
DRILL-2245: Clean up query setup and execution kickoff in Foreman/WorkManager in order to ensure consistent handling, and avoid hangs and races, with the goal of improving Drillbit robustness.

I did my best to keep these clean when I split them up, but this core commit

may depend on some minor changes in the hygiene commit that is also

associated with this bug, so either both should be applied, or neither.

The core commit should be applied first.

protocol/pom.xml

- updated protocol buffer compiler version to 2.6

- this made slight modifications to the formats of a few committed protobuf

files

AutoCloseables

- created org.apache.drill.common.AutoCloseables to handle closing these

quietly

BaseTestQuery, and derivatives

- factored out pieces into QueryTestUtil so they can be reused

DeferredException:

- created this so we can collect exceptions during the shutdown process

Drillbit

- uses AutoCloseables for the WorkManager and for the storeProvider

- allow start() to take a RemoteServiceSet

- private, final, formatting

Foreman

- added new state CANCELLATION_REQUESTED (via UserBitShared.proto) to represent

the time between request of a cancellation, and acknowledgement from all

remote endpoints running fragments on a query's behalf

- created ForemanResult to manage interleaving cleanup effects/failure with

query result state

- does not need to implement Comparable

- does not need to implement Closeable

- thread blocking fixes

- add resultSent flag

- add code to log plan fragments with endpoint assignments

- added finals, cleaned up formatting

- do queue management in acquireQuerySemaphore; local tests pass

- rename getContext() to getQueryContext()

- retain DrillbitContext

- a couple of exception injections for testing

- minor formatting

- TODOs

FragmentContext

- added a DeferredException to collect errors during startup/shutdown sequences

FragmentExecutor

- eliminated CancelableQuery

- use the FragmentContext's DeferredException for errors

- common subexpression elimination

- cleaned up

QueryContext

- removed unnecessary functions (with some outside classes tweaked for this)

- finals, formatting

QueryManager

- merge in QueryStatus

- affects Foreman, ../batch/ControlHandlerImpl,

and ../../server/rest/ProfileResources

- made some methods private

- removed unused imports

- add finals and formatting

- variable renaming to improve readability

- formatting

- comments

- TODOs

QueryStatus

- getAsInfo() private

- member renaming

- member access changes

- formatting

- TODOs

QueryTestUtil, BaseTestQuery, TestDrillbitResilience

- make maxWidth a parameter to server startup

SelfCleaningRunnable

- created org.apache.drill.common.SelfCleaningRunnable

SingleRowListener

- created org.apache.drill.SingleRowListener results listener

- use in TestDrillbitResilience

TestComparisonFunctions

- fix not to close the FragmentContext multiple times

TestDrillbitResilience

- created org.apache.drill.exec.server.TestDrillbitResilience to test drillbit

resilience in the face of exceptions and failures during queries

TestWithZookeeper

- factor out work into ZookeeperHelper so that it can be reused by

TestDrillbitResilience

UserBitShared

- get rid of unused UNKNOWN_QUERY

WorkEventBus

- rename methods, affects Foreman and ControlHandlerImpl

- remove unused WorkerBee reference

- most members final

- formatting

WorkManager

- Closeable to AutoCloseable

- removed unused incomingFragments Set

- eliminated unnecessary eventThread and pendingTasks by posting Runnables

directly to executor

- use SelfCleaningRunnable for Foreman management

- FragmentExecutor management uses SelfCleaningRunnable

- runningFragments to be a ConcurrentHashMap; TestTpchDistributed passes

- other improvements due to bee no longer needed in various places

- most members final

- minor formatting

- comments

- TODOs

(*) Created exception injection classes to simulate exceptions for testing

- ExceptionInjection

- ExceptionInjector

- ExceptionInjectionUtil

- TestExceptionInjection

DRILL-2245-hygiene: General code cleanup encountered while working on the rest

of this commit. This includes

- making members final whenever possible

- making members private whenever possible

- making loggers private

- removing unused imports

- removing unused private functions

- removing unused public functions

- removing unused local variables

- removing unused private members

- deleting unused files

- cleaning up formatting

- adding spaces before braces in conditionals and loop bodies

- breaking up overly long lines

- removing extra blank lines

While I tried to keep this clean, this commit may have minor dependencies on

DRILL-2245-core that I missed. The intention is just to break this up for

review purposes. Either both commits should be applied, or neither.

  1. … 93 more files in changeset.
DRILL-2127: make JDBC tests re-use bit instances; reset cached bits on failure

  1. … 14 more files in changeset.
DRILL-2126: make jdbc connection impl honor parameters passed from user

  1. … 3 more files in changeset.
DRILL-1904 - Part 1: Package level docs for the common module and a few packages in exec.

Fix line wrapping on package level docs to standardize at 80 character wrap.

Patch updated to address review comments.

Patch updated again to address Sudheesh's review comments.

  1. … 28 more files in changeset.