common

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Update version to 1.14.0-SNAPSHOT

  1. … 30 more files in changeset.
DRILL-6053: Avoid excessive locking in LocalPersistentStore

closes #1163

  1. … 17 more files in changeset.
DRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet

  1. … 30 more files in changeset.
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

  1. … 222 more files in changeset.
DRILL-6049: Misc. hygiene and code cleanup changes

close apache/drill#1085

  1. … 121 more files in changeset.
Update version to 1.13.0-SNAPSHOT

  1. … 29 more files in changeset.
[maven-release-plugin] prepare release drill-1.12.0

  1. … 29 more files in changeset.
DRILL-5989: Categories some tests to speed up smoke tests. Made travis run tests.

closes #1053

  1. … 31 more files in changeset.
DRILL-5987: Use one version of javassist

closes #1048

  1. … 2 more files in changeset.
DRILL-5986: Update jackson-databind version to 2.7.9.1

closes #1046

  1. … 4 more files in changeset.
DRILL-5922: - The QueryContext was never closed when the Foreman finished, so it's child allocator was never closed. Now it is. - The PlanSplitter created a QueryContext temporarily to construct an RPC message but never closed it. Now the temp QueryContext is closed. - The waitForExit method was error prone. Changed it to use the standard condition variable pattern. - Fixed timeouts in graceful shutdown tests

  1. … 6 more files in changeset.
DRILL-5783, DRILL-5841, DRILL-5894: Rationalize test temp directories

This change includes:

DRILL-5783:

- A unit test is created for the priority queue in the TopN operator.

- The code generation classes passed around a completely unused function registry reference in some places so it is removed.

- The priority queue had unused parameters for some of its methods so it is removed.

DRILL-5841:

- Created standardized temp directory classes DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. And updated all unit tests to use them.

DRILL-5894:

- Removed the dfs_test storage plugin for tests and replaced it with the already existing dfs storage plugin.

Misc:

- General code cleanup.

- Removed unnecessary use of String.format in the tests.

This closes #984

    • -0
    • +46
    ./src/main/java/org/apache/drill/common/util/DrillFileUtils.java
    • -0
    • +167
    ./src/test/java/org/apache/drill/test/DirTestWatcher.java
    • -0
    • +193
    ./src/test/java/org/apache/drill/test/SubDirTestWatcher.java
    • -0
    • +104
    ./src/test/java/org/apache/drill/test/TestTools.java
  1. … 358 more files in changeset.
DRILL-5819: Default value of security.admin.user_groups and security.admin.users is true

closes #983

  1. … 10 more files in changeset.
DRILL-5832: Change OperatorFixture to use system option manager

- Rename FixtureBuilder to ClusterFixtureBuilder

- Provide alternative way to reset system/session options

- Fix for DRILL-5833: random failure in TestParquetWriter

- Provide strict, but clear, errors for missing options

closes #970

  1. … 51 more files in changeset.
DRILL-5809 made the storage format of system options backwards compatible, and avoided storing unnecessary option info.

- Fixed forward compatability issue

- Added error message for debugging

- Fix flakey test

- Cleaned up bad logging

- Applied comments

closes #957

    • -0
    • +61
    ./src/test/java/org/apache/drill/testutils/DirTestWatcher.java
  1. … 19 more files in changeset.
DRILL-5694: Handle OOM in HashAggr by spill and retry, reserve memory, spinner

  1. … 19 more files in changeset.
DRILL-5761: Disable Lilith ClassicMultiplexSocketAppender by default. Unify logback files.

    • -0
    • +53
    ./src/test/resources/logback-test.xml
  1. … 12 more files in changeset.
DRILL-5752 this change includes:

1. Increased test parallelism and fixed associated bugs

2. Added test categories and categorized tests appropriately

- Don't exclude anything by default

- Increase test timeout

- Fixed flakey test

closes #940

    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/HiveStorageTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/JdbcStorageTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/JdbcTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/KuduStorageTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/MemoryTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/OperatorTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/OptionsTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/ParquetTest.java
    • -0
    • +26
    ./src/test/java/org/apache/drill/categories/PlannerTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/SecurityTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/SlowTest.java
    • -0
    • +25
    ./src/test/java/org/apache/drill/categories/SqlFunctionTest.java
  1. … 251 more files in changeset.
DRILL-5723: Added System Internal Options That can be Modified at Runtime Changes include:

1. Addition of internal options.

2. Refactoring of OptionManagers and OptionValidators.

3. Fixed ambiguity in the meaning of an option type, and changed its name to accessibleScopes.

4. Updated javadocs in the Option System classes.

5. Added RestClientFixture for testing the Rest API.

6. Fixed flakey test in TestExceptionInjection caused by race condition.

7. Fixed various tests which started zookeeper but failed to shut it down at the end of tests.

8. Added port hunting to the Drill Webserver for testing

9. Fixed various flaky tests

10. Fix compile issue

closes #923

  1. … 85 more files in changeset.
Update version to 1.12.0-SNAPSHOT

  1. … 27 more files in changeset.
[maven-release-plugin] prepare release drill-1.11.0

  1. … 27 more files in changeset.
DRILL-5431: SSL Support (Java) - Java client server SSL implementation

Also enable OpenSSL support

Also fix exclusions and java-exec pom file to eliminate netty-tcnative as a transitive dependency on all projects

  1. … 30 more files in changeset.
DRILL-5560: Create configuration file for distribution specific configuration This closes #848

  1. … 1 more file in changeset.
DRILL-5546: Handle schema change exception failure caused by empty input or empty batch.

1. Modify ScanBatch's logic when it iterates list of RecordReader.

1) Skip RecordReader if it returns 0 row && present same schema. A new schema (by calling Mutator.isNewSchema() ) means either a new top level field is added, or a field in a nested field is added, or an existing field type is changed.

2) Implicit columns are presumed to have constant schema, and are added to outgoing container before any regular column is added in.

3) ScanBatch will return NONE directly (called as "fast NONE"), if all its RecordReaders haver empty input and thus are skipped, in stead of returing OK_NEW_SCHEMA first.

2. Modify IteratorValidatorBatchIterator to allow

1) fast NONE ( before seeing a OK_NEW_SCHEMA)

2) batch with empty list of columns.

2. Modify JsonRecordReader when it get 0 row. Do not insert a nullable-int column for 0 row input. Together with ScanBatch, Drill will skip empty json files.

3. Modify binary operators such as join, union to handle fast none for either one side or both sides. Abstract the logic in AbstractBinaryRecordBatch, except for MergeJoin as its implementation is quite different from others.

4. Fix and refactor union all operator.

1) Correct union operator hanndling 0 input rows. Previously, it will ignore inputs with 0 row and put nullable-int into output schema, which causes various of schema change issue in down-stream operator. The new behavior is to take schema with 0 into account

in determining the output schema, in the same way with > 0 input rows. By doing that, we ensure Union operator will not behave like a schema-lossy operator.

2) Add a UnionInputIterator to simplify the logic to iterate the left/right inputs, removing significant chunk of duplicate codes in previous implementation.

The new union all operator reduces the code size into half, comparing the old one.

5. Introduce UntypedNullVector to handle convertFromJson() function, when the input batch contains 0 row.

Problem: The function convertFromJSon() is different from other regular functions in that it only knows the output schema after evaluation is performed. When input has 0 row, Drill essentially does not have

a way to know the output type, and previously will assume Map type. That works under the assumption other operators like Union would ignore batch with 0 row, which is no longer

the case in the current implementation.

Solution: Use MinorType.NULL at the output type for convertFromJSON() when input contains 0 row. The new UntypedNullVector is used to represent a column with MinorType.NULL.

6. HBaseGroupScan convert star column into list of row_key and column family. HBaseRecordReader should reject column star since it expectes star has been converted somewhere else.

In HBase a column family always has map type, and a non-rowkey column always has nullable varbinary type, this ensures that HBaseRecordReader across different HBase regions will have the same top level schema, even if the region is

empty or prune all the rows due to filter pushdown optimization. In other words, we will not see different top level schema from different HBaseRecordReader for the same table.

However, such change will not be able to handle hard schema change : c1 exists in cf1 in one region, but not in another region. Further work is required to handle hard schema change.

7. Modify scan cost estimation when the query involves * column. This is to remove the planning randomness since previously two different operators could have same cost.

8. Add a new flag 'outputProj' to Project operator, to indicate if Project is for the query's final output. Such Project is added by TopProjectVisitor, to handle fast NONE when all the inputs to the query are empty

and are skipped.

1) column star is replaced with empty list

2) regular column reference is replaced with nullable-int column

3) An expression will go through ExpressionTreeMaterializer, and use the type of materialized expression as the output type

4) Return an OK_NEW_SCHEMA with the schema using the above logic, then return a NONE to down-stream operator.

9. Add unit test to test operators handling empty input.

10. Add unit test to test query when inputs are all empty.

DRILL-5546: Revise code based on review comments.

Handle implicit column in scan batch. Change interface in ScanBatch's constructor.

1) Ensure either the implicit column list is empty, or all the reader has the same set of implicit columns.

2) We could skip the implicit columns when check if there is a schema change coming from record reader.

3) ScanBatch accept a list in stead of iterator, since we may need go through the implicit column list multiple times, and verify the size of two lists are same.

ScanBatch code review comments. Add more unit tests.

Share code path in ProjectBatch to handle normal setupNewSchema() and handleNullInput().

- Move SimpleRecordBatch out of TopNBatch to make it sharable across different places.

- Add Unit test verify schema for star column query against multilevel tables.

Unit test framework change

- Fix memory leak in unit test framework.

- Allow SchemaTestBuilder to pass in BatchSchema.

close #906

  1. … 67 more files in changeset.
DRILL-5512: Standardize error handling in ScanBatch

Standardizes error handling to throw a UserException. Prior code threw

various exceptions, called the fail() method, or returned a variety of

status codes.

closes #838

  1. … 5 more files in changeset.
DRILL-5325: Unit tests for the managed sort

Uses the sub-operator test framework (DRILL-5318), including the test

row set abstraction (DRILL-5323) to enable unit testing of the

“managed” external sort. This PR allows early review of the code, but

cannot be pulled until the dependencies (mentioned above) are pulled.

Refactors the external sort code into small chunks that can be unit

tested, then “wraps” that code in tests for all interesting data types,

record batch sizes, and so on.

Refactors some of the operator definitions to more easily allow

programmatic setup in the unit tests.

Fixes a number of bugs discovered by the unit tests. The biggest

changes were in the new code: the code that computes spilling and

merging based on memory levels.

Otherwise, although GitHub will show many files change, most of the

changes are simply moving blocks of code around to create smaller units

that can be tested independently.

Includes a refactoring of the code that does spilling, along with a

complete set of low-level unit tests.

Excludes long-running sort tests.

Defines a test category for long-running tests.

First attempt to provide a way to run such tests from Maven.

closes #808

    • -0
    • +70
    ./src/main/java/org/apache/drill/test/SecondaryTest.java
  1. … 50 more files in changeset.
DRILL-5419: Calculate return string length for literals & some string functions

1. Revisited calculation logic for string literals and some string functions

(cast, upper, lower, initcap, reverse, concat, concat operator, rpad, lpad, case statement,

coalesce, first_value, last_value, lag, lead).

Synchronized return type length calculation logic between limit 0 and regular queries.

2. Deprecated width and changed it to precision for string types in MajorType.

3. Revisited FunctionScope and splitted it into FunctionScope and ReturnType.

FunctionScope will indicate only function usage in term of number of in / out rows, (n -> 1, 1 -> 1, 1->n).

New annotation in UDFs ReturnType will indicate which return type strategy should be used.

4. Changed MAX_VARCHAR_LENGTH from 65536 to 65535.

5. Updated calculation of precision and display size for INTERVALYEAR & INTERVALDAY.

6. Refactored part of function code-gen logic (ValueReference, WorkspaceReference, FunctionAttributes, DrillFuncHolder).

This closes #819

  1. … 78 more files in changeset.
Update version to 1.11.0-SNAPSHOT

  1. … 27 more files in changeset.
[maven-release-plugin] prepare release drill-1.10.0

  1. … 27 more files in changeset.
DRILL-5326: Unit tests failures related to the SERVER_METADTA

- adding of the sql type name for the "GENERIC_OBJECT";

- changing "NullCollation" in the "ServerMetaProvider" to the correct default value;

- changing RpcType to GET_SERVER_META in the appropriate ServerMethod

close #775

  1. … 3 more files in changeset.