Clone Tools
  • last updated 22 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7764: Cleanup warning messages in GuavaPatcher class (#2093)

* DRILL-7764: Cleanup warning messages in GuavaPatcher class

DRILL-7711: Add data path, parameter filter pushdown to HTTP plugin

Adds an option to specify the path to data so the plugin will

ignore REST message "overhead" except the actual data.

Allows specifying HTTP URL parameters as filter push-downs from

SQL.

* Revised scan cost model to allow a scan more freedom

to define cost. Needed to ensure that filter push-down

is actually accepted by Calcite.

* Reduced size of HTTP scan plan by including on the one

needed connection config rather than all of them.

* Revised URL building so that the proxy sees the full

URL after adding parameters, etc.

* Many code refinements.

* Added more details to README

  1. … 34 more files in changeset.
DRILL-7702: Update shaded guava

    • -0
    • +99
    ./concurrent/AbstractCheckedFuture.java
    • -0
    • +72
    ./concurrent/CheckedFuture.java
  1. … 8 more files in changeset.
DRILL-7696: EVF v2 scan schema resolution

Provides the mechanism to resolve the scan schema from a

projection list, provided schema, early reader schema and

actual reader schema.

  1. … 75 more files in changeset.
DRILL-7693: Updated protobuf version to 3.11.1

  1. … 33 more files in changeset.
DRILL-7632: Improve user exception formatting

Adds a colon in the "getMessage()" format of a User

Exception between the context name and value:

My Context: value

closes #2017

    • -6
    • +15
    ./exceptions/UserExceptionContext.java
DRILL-7637: Add an option to retrieve MapR SSL truststore/keystore credentials using MapR Web Security Manager

  1. … 9 more files in changeset.
DRILL-7615: UNION ALL query returns the wrong result for the decimal value

closes #2006

  1. … 4 more files in changeset.
DRILL-7590: Refactor plugin registry

Major cleanup of the plugin registry to split it into components

in preparation for a proper plugin API.

Better coordinates the named and ephemeral plugin caches.

Cleans up the registry API. Sharpens rules for modifying

plugin configs.

closes #1988

    • -0
    • +119
    ./PlanStringBuilder.java
    • -4
    • +4
    ./exceptions/DrillRuntimeException.java
  1. … 154 more files in changeset.
DRILL-7634: Rollup of code cleanup changes

Collection of code cleanup changes. The most significant

is to create constants for function names.

closes #2020

    • -5
    • +3
    ./exceptions/DrillConfigurationException.java
    • -2
    • +3
    ./exceptions/RetryAfterSpillException.java
  1. … 123 more files in changeset.
DRILL-7576: Fail fast for operator errors

Converts operators to fail with a UserException rather than using

the STOP iterator status. The result is clearer error messages

and simpler code.

closes #1975

    • -1
    • +0
    ./exceptions/DrillRuntimeException.java
  1. … 66 more files in changeset.
DRILL-7549: Fix validation error when querying absent sub folder in embedded mode

closes #1963

    • -4
    • +4
    ./exceptions/DrillRuntimeException.java
  1. … 2 more files in changeset.
DRILL-7527: DROP METADATA doesn't work with table name starting with '/' inside workspace

closes #1958

  1. … 10 more files in changeset.
DRILL-7467: Jdbc plugin enhancements and fixes

1. Added logic to close data source when plugin is closed.

2. Added disabled jdbc plugin template to the bootstrap storage plugins.

3. Added new jdbc storage plugin configuration property sourceParameters which would allow setting data source parameters described in BasicDataSource Configuration Parameters.

4. Upgraded commons-dbcp2 version and added it to the dependency management section in common pom.xml.

closes #1956

  1. … 17 more files in changeset.
DRILL-7506: Simplify code gen error handling

Pushes code gen error handling close to the code gen itself to

allow clearer error messages. Doing so avoids the need to bubble

code gen exceptions up the call stack, resulting in cleaner

operator code.

closes #1948

  1. … 40 more files in changeset.
DRILL-7502: Invalid codegen for typeof() with UNION

Also fixes DRILL-6362: typeof() reports NULL for primitive

columns with a NULL value.

typeof() is meant to return "NULL" if a UNION has a NULL

value, but the column type when known, such as for non-UNION

columns.

Also fixes DRILL-7499: sqltypeof() function with an array returns

"ARRAY", not type. This was due to treating REPEATED like LIST.

Handling of the Union vector in code gen is problematic

with about three special cases. Existing code handled two

of the cases. This change handles the third case.

Figuring out the change required poking around quite a bit

of unclear code. Added comments and restructuring to make

that code a bit more clear.

The fix modified code gen for the Union Holder. It can now

"go back in time" to add the union reader at the point we

need it.

closes #1945

  1. … 54 more files in changeset.
DRILL-7494: Unable to connect to Drill using JDBC driver when using custom authenticator

closes #1938

  1. … 5 more files in changeset.
DRILL-6332: Allow to provide two-component Kerberos principals

closes #1931

  1. … 1 more file in changeset.
DRILL-6832: Removes the old "unmanaged" external sort

When the "managed" external sort was implemented a couple

of years back, we retained the original "unmanaged" version

out of an abundance of caution. The new version is now

battle tested and it is time to retire the original one.

closes #1929

  1. … 64 more files in changeset.
DRILL-7393: Revisit Drill tests to ensure that patching is executed before any test run

- Added BaseTest with patchers and extended all tests from it.

- Added a test to java-exec module to ensure that all tests there are inherited from BaseTest.

- Revised exception handling in the patchers, now it's individual for each patching method.

closes #1910

  1. … 138 more files in changeset.
DRILL-7445: Create batch copier based on result set framework

The result set framework now provides both a reader and writer.

This PR provides a copier that copies batches using this

framework. Such a copier can:

- Copy selected records

- Copy all records, such as for an SV2 or SV4

The copier uses the result set loader to create uniformly-sized

output batches from input batches of any size. It does this

by merging or splitting input batches as needed.

Since the result set reader handles both SV2 and SV4s, the

copier can filter or reorder rows based on the SV associated

with the input batch.

This version assumes single stream of input batches, and handles

any schema changes in that input by creating output batches

that track the input schema. This would be used in, say, the

selection vector remover. A different design is needed for merging

such as in the merging receiver.

Adds a "copy" method to the column writers. Copy is implemented

by doing a direct memory copy from source to destination vectors.

A unit test verifies functionality for various use cases

and data types.

closes #1899

  1. … 39 more files in changeset.
DRILL-6540: Updated Hadoop and HBase libraries to the latest versions

Hadoop: 3.2.1

HBase: 2.2.2

closes #1895

  1. … 82 more files in changeset.
DRILL-7412: Minor unit test improvements

Many tests intentionally trigger errors. A debug-only log setting

sent those errors to stdout. The resulting stack dumps simply cluttered

the test output, so disabled error output to the console.

Drill can apply bounds checks to vectors. Tests run via Maven

enable bounds checking. Now, bounds checking is also enabled in

"debug mode" (when assertions are enabled, as in an IDE.)

Drill contains two test frameworks. The older BaseTestQuery was

marked as deprecated, but many tests still use it and are unlikely

to be changed soon. So, removed the deprecated marker to reduce the

number of spurious warnings.

Also includes a number of minor clean-ups.

closes #1876

  1. … 16 more files in changeset.
DRILL-7357: Expose Drill Metastore data through information_schema

1. Add additional columns to TABLES and COLUMNS tables.

2. Add PARTITIONS table.

3. General refactoring to adjust information_schema data retrieval from multiple sources.

closes #1860

  1. … 33 more files in changeset.
DRILL-7314: Use TupleMetadata instead of concrete implementation

1. Add ser / de implementation for TupleMetadata interface based on types.

2. Replace TupleSchema usage where possible.

3. Move patcher classes into commons.

4. Upgrade some dependencies and general refactoring.

    • -0
    • +91
    ./util/GuavaPatcher.java
    • -0
    • +188
    ./util/ProtobufPatcher.java
  1. … 39 more files in changeset.
DRILL-7315: Revise precision and scale order in the method arguments

  1. … 28 more files in changeset.
DRILL-7273: Introduce operators for handling metadata

closes #1886

    • -2
    • +24
    ./util/function/CheckedSupplier.java
  1. … 155 more files in changeset.
DRILL-7272: Drill Metastore Read / Write API and Drill Iceberg Metastore implementation

1. Drill Metastore Read / Write API.

2. Drill Iceberg Metastore implementation in iceberg-metastore module.

3. Patches Guava Preconditions class for Apache Iceberg.

4. General refactoring.

5. Unit tests.

6. Documentation.

  1. … 96 more files in changeset.
DRILL-7181: Improve V3 text reader (row set) error messages

Adds an error context to the User Error mechanism. The context allows

information to be passed through an intermediate layer and applied when

errors are raised in lower-level code; without the need for that

low-level code to know the details of the error context information.

Modifies the scan framework and V3 text plugin to use the framework to

improve error messages.

Refines how the `columns` column can be used with the text reader. If

headers are used, then `columns` is just another column. An error is

raised, however, if `columns[x]` is used when headers are enabled.

Added another builder abstraction where a constructor argument list

became too long.

Added the drill file system and split to the file schema negotiator

to simplify reader construction.

Added additional unit tests to fully define the `columns` column

behavior.

    • -0
    • +41
    ./exceptions/ChildErrorContext.java
    • -0
    • +44
    ./exceptions/CustomErrorContext.java
    • -0
    • +26
    ./exceptions/EmptyErrorContext.java
  1. … 31 more files in changeset.
DRILL-7143: Support default value for empty columns

Modifies the prior work to add default values for columns. The prior work added defaults

when the entire column is missing from a reader (the old Nullable Int column). The Row

Set mechanism now will also "fill empty" slots with the default value.

Added default support for the column writers. The writers automatically obtain the

default value from the column schema. The default can also be set explicitly on

the column writer.

Updated the null column mechanism to use this feature rather than the ad-hoc

implemention in the prior commit.

Semantics changed a bit. Only Required columns take a default. The default value

is ignored or nullable columns since nullable columns already have a file default: NULL.

Other changes:

* Updated the CSV-with-schema tests to illustrate the new behavior.

* Made multiple fixes for Boolean and Decimal columns and added unit tests.

* Upgraded Fremarker to version 2.3.28 to allow use of the continue statement.

* Reimplemented the Bit column reader and writer to use the BitVector directly since this vector is rather special.

* Added get/set Boolean methods for column accessors

* Moved the BooleanType class to the common package

* Added more CSV unit tests to explore decimal types, booleans, and defaults

* Add special handling for blank fields in from-string conversions

* Added options to the conversion factory to specify blank-handling behavior.

CSV uses a mapping of blanks to null (nullable) or default value (non-nullable)

closes #1726

    • -0
    • +92
    ./types/BooleanType.java
  1. … 71 more files in changeset.