Clone
 

paul rogers <par0328@yahoo.com> in drill

Revisions from review comments

DRILL-7734: Revise the result set reader

Revised into two forms: push (for streaming JSON results) and

pull (for one operator reading from another).

closes #2077

  1. … 7 more files in changeset.
DRILL-7730: Improve web query efficiency

Implements a direct transfer of batches from Screen to web client.

Cleans up web client query processing to avoid duplicate schema

info.

Much related code cleanup.

  1. … 24 more files in changeset.
DRILL-7725: Updates to the EVF2 framework

* Supports internal implicit columns

* Better support for standard conversions

* Handle several reader corner cases

* Simplified file reader

closes #2073

    • -0
    • +26
    /common/src/test/java/org/apache/drill/categories/EvfTest.java
  1. … 67 more files in changeset.
DRILL-7724: Refactor metadata controller batch

Also changed for (;;) infinite loops to

while (true) as preferred by IntelliJ.

  1. … 11 more files in changeset.
DRILL-7717: Support Mongo extended types in V2 JSON loader

Adds support for Mongo extended types to the JSON loader.

Refactors the JSON code to make such extensions easier.

Added support for provided schema with extended types.

  1. … 83 more files in changeset.
DRILL-7711: Add data path, parameter filter pushdown to HTTP plugin

Adds an option to specify the path to data so the plugin will

ignore REST message "overhead" except the actual data.

Allows specifying HTTP URL parameters as filter push-downs from

SQL.

* Revised scan cost model to allow a scan more freedom

to define cost. Needed to ensure that filter push-down

is actually accepted by Calcite.

* Reduced size of HTTP scan plan by including on the one

needed connection config rather than all of them.

* Revised URL building so that the proxy sees the full

URL after adding parameters, etc.

* Many code refinements.

* Added more details to README

    • -85
    • +335
    /contrib/storage-http/README.md
  1. … 20 more files in changeset.
DRILL-7703: Support for 3+D arrays in EVF JSON loader

Revises the EVF-based JSON loader to support nested

repeated lists.

DRILL-7701: EVF V2 Scan Framework

Revises the scan framework to use the revised schema resolution

introduced in DRILL-7696.

  1. … 43 more files in changeset.
DRILL-6168: Revise format plugin table functions

Allows table functions to inherit properties from a

defined format plugin.

Also DRILL-7612: enforces immutability for all format plugins.

  1. … 32 more files in changeset.
DRILL-7603 and DRILL-7604: Add schema, options to REST query

Update and revision of work originally done by dobesv.

DRILL-7603: Allow default schema to be set for HTTP queries

DRILL-7604: Allow session options to be set in HTTP queries

Merges the above two. Separates running a REST query from the

JSON representation. Allows setting all option types from

a string (as required by DRILL-7604).

Added default schema to query profile query editor.

Made the two query editors a bit more similar visually,

but see DRILL-7697 for more work needed.

Added a utility to run a server for UI teseting without

a full build.

  1. … 6 more files in changeset.
DRILL-7696: EVF v2 scan schema resolution

Provides the mechanism to resolve the scan schema from a

projection list, provided schema, early reader schema and

actual reader schema.

  1. … 61 more files in changeset.
DRILL-7675: Work around for partitions sender memory use

Adds an ad-hoc system/session option to limit partition sender

memory use. See DRILL-7686 for the underlying issue.

Also includes code cleanup and diagnostic tools.

closes #2047

DRILL-7683: Add "message parsing" to new JSON loader

Adds the ability to parse "extra" JSON around the data payload,

as often needed for a REST API.

closes #2045

DRILL-7680: Place UDFs before plugins in contrib

DRILL-7640: EVF-based JSON Loader

Builds on the JSON structure parser and several other PRs

to provide an enhanced, robust mechanism to read JSON data

into value vectors via the EVF. This is not the JSON reader,

rather it is the "V2" version of the JsonProcessor which

does the actual JSON parsing/loading work.

closes #2023

  1. … 27 more files in changeset.
DRILL-7633: Fixes for union and repeated list accessors

Minor fixes and cleanup for the obscure union and

repeated list types in the column accesor framework.

Added variant type typeString() function

DRILL-7632: Improve user exception formatting

Adds a colon in the "getMessage()" format of a User

Exception between the context name and value:

My Context: value

closes #2017

DRILL-7631: Updates to the Json Structure Parser

Revised how "look-ahead" works. Added support for unknown

types.

closes #2016

  1. … 11 more files in changeset.
DRILL-7620: Fix plugin mutability issues

A recent commit made the plugin registry more strict about

the rule that, once a plugin is registered, it must be

immutable. A flaw enforcing that rule in the UI put the

registry in an inconsistent state.

Also

* Registry-specific errors

* Push more operations from UI layer into registry

* Clean up semantics of "resolve" for plugins

* Add more unit tests

* Better handling of "bad" plugins

* Force plugin names to lower case

* Fix comparison bugs in some format plugins

  1. … 87 more files in changeset.
DRILL-7617: Disabled plugins not showing in Web UI

Fixes a misunderstanding in prior PR: getConfigs()

should return both enabled and disabled configs.

DRILL-7601: Shift column conversion to reader from scan framework

Allows the column writers to be generic, moves scan-specific

conversions into each reader where needed, implemented in

a reader-specific way.

Adds a revised way of handling projections in the result set

loader that is not coupled with conversion, as the prior

design was.

Updates the CSV, Avro, Log and HDF5 readers.

closes #1993

  1. … 206 more files in changeset.
DRILL-7583: Remove STOP status from operator outcome

Now that all operators have been converted to throw

exceptions on error condistions, the STOP status is

unused. This patch removes the STOP status and the

related kill() and killIncoming() methods. The

"kill" methods are replaced by "cancel" methods which

handle "normal" case cancellation, such as for

LIMIT.

closes #1981

  1. … 63 more files in changeset.
DRILL-7590: Refactor plugin registry

Major cleanup of the plugin registry to split it into components

in preparation for a proper plugin API.

Better coordinates the named and ephemeral plugin caches.

Cleans up the registry API. Sharpens rules for modifying

plugin configs.

closes #1988

  1. … 149 more files in changeset.
DRILL-7634: Rollup of code cleanup changes

Collection of code cleanup changes. The most significant

is to create constants for function names.

closes #2020

  1. … 114 more files in changeset.
DRILL-7576: Fail fast for operator errors

Converts operators to fail with a UserException rather than using

the STOP iterator status. The result is clearer error messages

and simpler code.

closes #1975

  1. … 52 more files in changeset.
DRILL-7574: Generalize the projection parser

Adds support for multi-dimensional arrays, and columns

projected as both an array and a map.

closes #1974

  1. … 34 more files in changeset.
DRILL-7572: JSON structure parser

Provides a structure-driven parser for JSON. Provides an intermediate

level beteen the Jackson JSON parser and the "shims" that write to

Drill value vectors via the EVF.

close #1971

  1. … 6 more files in changeset.
DRILL-7507: Convert fragment interrupts to exceptions

Modifies fragment interrupt handling to throw a specialized

exception, rather than relying on the complex and cumbersome

STOP iterator status.

closes #1949

DRILL-7506: Simplify code gen error handling

Pushes code gen error handling close to the code gen itself to

allow clearer error messages. Doing so avoids the need to bubble

code gen exceptions up the call stack, resulting in cleaner

operator code.

closes #1948

  1. … 26 more files in changeset.