Clone
Paul Rogers <par0328@yahoo.com>
committed
on 17 May
DRILL-7278: Refactor result set loader projection mechanism
Drill 1.16 added a enhanced scan framework based on the row set
mechanisms, and … Show more
DRILL-7278: Refactor result set loader projection mechanism

Drill 1.16 added a enhanced scan framework based on the row set

mechanisms, and a "provisioned schema" feature build on top

of that framework. Conversion of the log reader plugin to use

the framework identified additional features we wish to add,

such as marking a column as "special" (not expanded in a wildcard

query.)

This work identified that the code added for provisioned schemas in

Drill 1.16 worked, but is a bit overly complex, making it hard to add

the desired new feature.

This patch refactors the "reader" projection code:

* Create a "projection set" mechanism that the reader can query to ask,

 "the caller just added a column. Should it be projected or not?"

* Unifies the type conversion mechanism added as part of provisioned

 schemas.

* Added the "special column" property for both "reader" and "provided"

 schemas.

* Verified that provisioned schemas work with maps (at least on the scan

 framework side.)

* Replaced the previous "schema transformer" mechanism with a new "type

 conversion" mechanism that unifies type conversion, provided schemas

 and an optional custom type conversion mechanism.

* Column writers can report if they are projected. Moved this query

 from metadata to the column writer itself.

* Extended and clarified documentation of the feature.

* Revised and/or added unit tests.

closes #1797

Show less