Clone Tools
  • last updated 25 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7294: Prevent generating java beans using protostuff to avoid overriding classes with the same simple name declared as nested in the proto files

closes #1808

    • -2
    • +2
    ./drill/exec/store/jdbc/JdbcSubScan.java
  1. … 7 more files in changeset.
DRILL-6734: JDBC storage plugin returns null for fields without aliases closes #1642 - Add output column names to JdbcRecordReader and use them for storing the results since column names in result set may differ when aliases aren't specified

    • -1
    • +2
    ./drill/exec/store/jdbc/JdbcBatchCreator.java
    • -12
    • +16
    ./drill/exec/store/jdbc/JdbcGroupScan.java
    • -19
    • +32
    ./drill/exec/store/jdbc/JdbcRecordReader.java
    • -3
    • +12
    ./drill/exec/store/jdbc/JdbcSubScan.java
  1. … 2 more files in changeset.
DRILL-6850: Allow configuring table names case sensitivity for JDBC storage plugin

closes #1542

    • -3
    • +13
    ./drill/exec/store/jdbc/JdbcStorageConfig.java
    • -9
    • +20
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 7 more files in changeset.
DRILL-6850: Force setting DRILL_LOGICAL Convention for DrillRelFactories and DrillFilterRel

- Fix workspace case insensitivity for JDBC storage plugin

    • -10
    • +11
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 13 more files in changeset.
DRILL-6724: Dump operator context to logs when error occurs during query execution

closes #1455

    • -1
    • +7
    ./drill/exec/store/jdbc/JdbcRecordReader.java
  1. … 102 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

    • -4
    • +5
    ./drill/exec/store/jdbc/DrillJdbcRuleBase.java
    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcBatchCreator.java
    • -3
    • +3
    ./drill/exec/store/jdbc/JdbcRecordReader.java
    • -6
    • +7
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 981 more files in changeset.
DRILL-6656: Disallow extra semicolons and multiple statements on the same line.

closes #1415

    • -1
    • +0
    ./drill/exec/store/jdbc/JdbcExpressionCheck.java
    • -2
    • +4
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 143 more files in changeset.
DRILL-6617: Changing name of implicit RowId column from implicitColumn to implicitRIDColumn.

closes #1401

  1. … 20 more files in changeset.
DRILL-6617: Planner Side changed to propagate $drill_implicit_field$ information.

  1. … 36 more files in changeset.
DRILL-6647: Update Calcite version to 1.17.0

closes #1425

    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 16 more files in changeset.
DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.

1. Added protobuf files (incl generated C++ and Java)

2. Updated JdbcSubScan's getOperatorType API

closes #1297

    • -1
    • +2
    ./drill/exec/store/jdbc/JdbcSubScan.java
  1. … 5 more files in changeset.
DRILL-6450: Visualized plans for profiles querying JDBC sources is broken

When viewing a profile for a query against a JDBC source, the visualized plan is not rendered. This is because the generated SQL pushed down to the JDBC source has a line break injected just before the FROM clause.

The workaround is to strip away any injected newlines ('\\n') at least for the SQL defined in the text plan, so that the backend Javascript can render it correctly.

In addition, any single line comments are also removed, but any block comments (i.e. /* .. */ ) are retained as they might carry hints.

This closes #1295

    • -2
    • +13
    ./drill/exec/store/jdbc/JdbcPrel.java
DRILL-6386: Remove unused imports and star imports.

    • -2
    • +0
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 231 more files in changeset.
DRILL-6422: Update guava to 23.0 and shade it

- Fix compilation errors for new version of Guava.

- Remove usage of deprecated API

- Shade guava and add dependencies to the shaded version

- Ban unshaded package

- Introduce drill-shaded module and move guava-shaded under it

- Add methods to convert shaded guava lists to the unshaded ones

- Add instruction for publishing artifacts to the Apache repository

    • -2
    • +1
    ./drill/exec/store/jdbc/DrillJdbcRuleBase.java
    • -3
    • +2
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 81 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcBatchCreator.java
    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcGroupScan.java
    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcRecordReader.java
    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcStorageConfig.java
    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcSubScan.java
  1. … 2060 more files in changeset.
DRILL-6094: Decimal data type enhancements

Add ExprVisitors for VARDECIMAL

Modify writers/readers to support VARDECIMAL

- Added usage of VarDecimal for parquet, hive, maprdb, jdbc;

- Added options to store decimals as int32 and int64 or fixed_len_byte_array or binary;

Add UDFs for VARDECIMAL data type

- modify type inference rules

- remove UDFs for obsolete DECIMAL types

Enable DECIMAL data type by default

Add unit tests for DECIMAL data type

Fix mapping for NLJ when literal with non-primitive type is used in join conditions

Refresh protobuf C++ source files

Changes in C++ files

Add support for decimal logical type in Avro.

Add support for date, time and timestamp logical types.

Update Avro version to 1.8.2.

    • -31
    • +53
    ./drill/exec/store/jdbc/JdbcRecordReader.java
  1. … 201 more files in changeset.
DRILL-6189: Security: passwords logging and file permisions

1. Overrided serialization methods for instances with passwords

2. Changed file permissions for configuration files

closes #1139

    • -0
    • +2
    ./drill/exec/store/jdbc/JdbcStorageConfig.java
  1. … 6 more files in changeset.
DRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin

closes #1282

    • -14
    • +3
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 11 more files in changeset.
DRILL-6294: Changes to support Calcite 1.16.0 , and remove deprecated API usage

closes #1198

    • -2
    • +4
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 38 more files in changeset.
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

    • -4
    • +3
    ./drill/exec/store/jdbc/JdbcBatchCreator.java
    • -8
    • +1
    ./drill/exec/store/jdbc/JdbcRecordReader.java
  1. … 222 more files in changeset.
DRILL-3993: Changes after code review.

  1. … 4 more files in changeset.
DRILL-3993: Move Drill-specific commits 'CALCITE-628' and 'Drill-specific change: Add back AbstractConverter in RelSet.java' from Calcite into DRILL

    • -2
    • +2
    ./drill/exec/store/jdbc/DrillJdbcRuleBase.java
    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 12 more files in changeset.
DRILL-3993: Use custom RelBuilder implementation in rules

After the changes, made in CALCITE-1056 if the filter has a predicate that is always false, RelBuilder.filter() method returns values rel node instead of filter rel node. In order to preserve column types, DrillRelBuilder.empty() method, which is returned by filter method was overridden, and now it returns filter with a false predicate. (advice to override this method was in its javadoc) The goal of all other changes in this commit is to use our custom RelBuilder for all rules that are used in Drill.

    • -4
    • +6
    ./drill/exec/store/jdbc/DrillJdbcRuleBase.java
    • -8
    • +10
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 34 more files in changeset.
DRILL-3993: Changes to support Calcite 1.13

- fixed all compiling errors (main changes were: Maven changes, chenges RelNode -> RelRoot, implementing some new methods from updated interfaces, chenges some literals, logger changes);

- fixed unexpected column errors, validation errors and assertion errors after Calcite update;

- fixed describe table/schema statement according to updated logic;

- added fixes with time-intervals;

- changed precision of BINARY to 65536 (was 1048576) according to updated logic (Calcite overrides bigger precision to own maxPrecision);

- ignored some incorrect tests with DRILL-3244;

- changed "Table not found" message to "Object not found within" according to new Calcite changes.

    • -1
    • +19
    ./drill/exec/store/jdbc/JdbcExpressionCheck.java
  1. … 69 more files in changeset.
DRILL-5546: Handle schema change exception failure caused by empty input or empty batch.

1. Modify ScanBatch's logic when it iterates list of RecordReader.

1) Skip RecordReader if it returns 0 row && present same schema. A new schema (by calling Mutator.isNewSchema() ) means either a new top level field is added, or a field in a nested field is added, or an existing field type is changed.

2) Implicit columns are presumed to have constant schema, and are added to outgoing container before any regular column is added in.

3) ScanBatch will return NONE directly (called as "fast NONE"), if all its RecordReaders haver empty input and thus are skipped, in stead of returing OK_NEW_SCHEMA first.

2. Modify IteratorValidatorBatchIterator to allow

1) fast NONE ( before seeing a OK_NEW_SCHEMA)

2) batch with empty list of columns.

2. Modify JsonRecordReader when it get 0 row. Do not insert a nullable-int column for 0 row input. Together with ScanBatch, Drill will skip empty json files.

3. Modify binary operators such as join, union to handle fast none for either one side or both sides. Abstract the logic in AbstractBinaryRecordBatch, except for MergeJoin as its implementation is quite different from others.

4. Fix and refactor union all operator.

1) Correct union operator hanndling 0 input rows. Previously, it will ignore inputs with 0 row and put nullable-int into output schema, which causes various of schema change issue in down-stream operator. The new behavior is to take schema with 0 into account

in determining the output schema, in the same way with > 0 input rows. By doing that, we ensure Union operator will not behave like a schema-lossy operator.

2) Add a UnionInputIterator to simplify the logic to iterate the left/right inputs, removing significant chunk of duplicate codes in previous implementation.

The new union all operator reduces the code size into half, comparing the old one.

5. Introduce UntypedNullVector to handle convertFromJson() function, when the input batch contains 0 row.

Problem: The function convertFromJSon() is different from other regular functions in that it only knows the output schema after evaluation is performed. When input has 0 row, Drill essentially does not have

a way to know the output type, and previously will assume Map type. That works under the assumption other operators like Union would ignore batch with 0 row, which is no longer

the case in the current implementation.

Solution: Use MinorType.NULL at the output type for convertFromJSON() when input contains 0 row. The new UntypedNullVector is used to represent a column with MinorType.NULL.

6. HBaseGroupScan convert star column into list of row_key and column family. HBaseRecordReader should reject column star since it expectes star has been converted somewhere else.

In HBase a column family always has map type, and a non-rowkey column always has nullable varbinary type, this ensures that HBaseRecordReader across different HBase regions will have the same top level schema, even if the region is

empty or prune all the rows due to filter pushdown optimization. In other words, we will not see different top level schema from different HBaseRecordReader for the same table.

However, such change will not be able to handle hard schema change : c1 exists in cf1 in one region, but not in another region. Further work is required to handle hard schema change.

7. Modify scan cost estimation when the query involves * column. This is to remove the planning randomness since previously two different operators could have same cost.

8. Add a new flag 'outputProj' to Project operator, to indicate if Project is for the query's final output. Such Project is added by TopProjectVisitor, to handle fast NONE when all the inputs to the query are empty

and are skipped.

1) column star is replaced with empty list

2) regular column reference is replaced with nullable-int column

3) An expression will go through ExpressionTreeMaterializer, and use the type of materialized expression as the output type

4) Return an OK_NEW_SCHEMA with the schema using the above logic, then return a NONE to down-stream operator.

9. Add unit test to test operators handling empty input.

10. Add unit test to test query when inputs are all empty.

DRILL-5546: Revise code based on review comments.

Handle implicit column in scan batch. Change interface in ScanBatch's constructor.

1) Ensure either the implicit column list is empty, or all the reader has the same set of implicit columns.

2) We could skip the implicit columns when check if there is a schema change coming from record reader.

3) ScanBatch accept a list in stead of iterator, since we may need go through the implicit column list multiple times, and verify the size of two lists are same.

ScanBatch code review comments. Add more unit tests.

Share code path in ProjectBatch to handle normal setupNewSchema() and handleNullInput().

- Move SimpleRecordBatch out of TopNBatch to make it sharable across different places.

- Add Unit test verify schema for star column query against multilevel tables.

Unit test framework change

- Fix memory leak in unit test framework.

- Allow SchemaTestBuilder to pass in BatchSchema.

close #906

    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcBatchCreator.java
  1. … 67 more files in changeset.
DRILL-4678: Tune metadata by generating a dispatcher at runtime

main code changes are in Calcite library.

update drill's calcite version to 1.4.0-drill-r20.

close #793

  1. … 39 more files in changeset.
DRILL-4277: Fix for JdbcPrel serialization issue.

This closes #326.

  1. … 3 more files in changeset.
DRILL-4198: Enhance StoragePlugin interface to expose logical space rules for planning purpose

Also move Hive partition pruning rules to logical storage plugin rulesets.

this closes #300

    • -1
    • +1
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java
  1. … 9 more files in changeset.
DRILL-4124: Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs

This closes #281

    • -4
    • +2
    ./drill/exec/store/jdbc/JdbcRecordReader.java
  1. … 6 more files in changeset.
DRILL-4031: Log warning and ignore columns returned from JDBC source that are unsupported.

This closes #240

    • -4
    • +29
    ./drill/exec/store/jdbc/JdbcRecordReader.java
    • -6
    • +18
    ./drill/exec/store/jdbc/JdbcStoragePlugin.java