dbarclay <> in drill

DRILL-2288: Fix ScanBatch violation of IterOutcome protocol and downstream chain of bugs.


2288: Pt. 1 Core: Added unit test. [Drill2288GetColumnsMetadataWhenNoRowsTest, empty.json]

2288: Pt. 1 Core: Changed HBase test table #1's # of regions from 1 to 2. [HBaseTestsSuite]

Also added TODO(DRILL-3954) comment about # of regions.

2288: Pt. 2 Core: Documented IterOutcome much more clearly. [RecordBatch]

Also edited some related Javadoc.

2288: Pt. 2 Hyg.: Edited doc., added @Override, etc. [AbstractRecordBatch, RecordBatch]

Purged unused SetupOutcome.

Added @Override.

Edited comments.

Fix some comments to doc. comments.

2288: Pt. 3 Core&Hyg.: Added validation of IterOutcome sequence. [IteratorValidatorBatchIterator]


Renamed internal members for clarity.

Added comments.

2288: Pt. 4 Core: Fixed a NONE -> OK_NEW_SCHEMA in [ScanBatch]

(With nearby comments.)

2288: Pt. 4 Hyg.: Edited comments, reordered, whitespace. [ScanBatch]


Added comments.


2288: Pt. 4 Core+: Fixed UnionAllRecordBatch to receive IterOutcome sequence right. (3659) [UnionAllRecordBatch]

2288: Pt. 5 Core: Fixed ScanBatch.Mutator.isNewSchema() to stop spurious "new schema" reports (fix short-circuit OR, to call resetting method right). [ScanBatch]

2288: Pt. 5 Hyg.: Renamed, edited comments, reordered. [ScanBatch, SchemaChangeCallBack, AbstractSingleRecordBatch]

Renamed getSchemaChange -> getSchemaChangedAndReset.

Renamed schemaChange -> schemaChanged.

Added doc. comments.


2288: Pt. 6 Core: Avoided dummy Null.IntVec. column in JsonReader when not needed (MapWriter.isEmptyMap()). [JsonReader, 3 vector files]

2288: Pt. 6 Hyg.: Edited comments, message. Fixed message formatting. [RecordReader, JSONFormatPlugin, JSONRecordReader, AbstractMapVector, JsonReader]

Fixed message formatting.

Edited comments.

Edited message.

Fixed spurious line break.

2288: Pt. 7 Core: Added column families in HBaseRecordReader* to avoid dummy Null.IntVec. clash. [HBaseRecordReader]

2288: Pt. 8 Core.1: Cleared recordCount in OrderedPartitionRecordBatch.innerNext(). [OrderedPartitionRecordBatch]

2288: Pt. 8 Core.2: Cleared recordCount in ProjectRecordBatch.innerNext. [ProjectRecordBatch]

2288: Pt. 8 Core.3: Cleared recordCount in TopNBatch.innerNext. [TopNBatch]

2288: Pt. 9 Core: Had UnorderedReceiverBatch reset RecordBatchLoader's record count. [UnorderedReceiverBatch, RecordBatchLoader]

2288: Pt. 9 Hyg.: Added comments. [RecordBatchLoader]

2288: Pt. 10 Core: Worked around mismatched map child vectors in MapVector.getObject(). [MapVector]

2288: Pt. 11 Core: Added OK_NEW_SCHEMA schema comparison for HashAgg. [HashAggTemplate]

2288: Pt. 12 Core: Fixed memory leak in BaseTestQuery's printing.

Fixed bad skipping of RecordBatchLoader.clear(...) and

QueryDataBatch.load(...) for zero-row batches in printResult(...).

Also, dropped suppression of call to

VectorUtil.showVectorAccessibleContent(...) (so zero-row batches are

as visible as others).

2288: Pt. 13 Core: Fixed test that used unhandled periods in column alias identifiers.

2288: Misc.: Added # of rows to showVectorAccessibleContent's output. [VectorUtil]

2288: Misc.: Added simple/partial toString() [VectorContainer, AbstractRecordReader, JSONRecordReader, BaseValueVector, FieldSelection, AbstractBaseWriter]

2288: Misc. Hyg.: Added doc. comments to VectorContainer. [VectorContainer]

2288: Misc. Hyg.: Edited comment. [DrillStringUtils]

2288: Misc. Hyg.: Clarified message for unhandled identifier containing period.

2288: Pt. 3 Core&Hyg. Upd.: Added schema comparison result to logging. [IteratorValidatorBatchIterator]

2288: Pt. 7 Core Upd.: Handled HBase columns too re NullableIntVectors. [HBaseRecordReader, TestTableGenerator, TestHBaseFilterPushDown]

Created map-child vectors for requested columns.

Added unit test method testDummyColumnsAreAvoided, adding new row to test table,

updated some row counts.

2288: Pt. 7 Hyg. Upd.: Edited comment. [HBaseRecordReader]

2288: Pt. 11 Core Upd.: REVERTED all of bad OK_NEW_SCHEMA schema comparison for HashAgg. [HashAggTemplate]

This reverts commit 0939660f4620c03da97f4e1bf25a27514e6d0b81.

2288: Pt. 6 Core Upd.: Added isEmptyMap override in new (just-rebased-in) PromotableWriter. [PromotableWriter]

Adjusted definition and default implementation of isEmptyMap (to handle MongoDB

storage plugin's use of JsonReader).

2288: Pt. 6 Hyg. Upd.: Purged old atLeastOneWrite flag. [JsonReader]

2288: Pt. 14: Disabled newly dying test testNestedFlatten().

  1. … 25 more files in changeset.
DRILL-3766: Fix ResultSet.getRow() (make 1-based; return 0 at end).

Corrected for Avatica's non-JDBC-compliant behavior.

Also purged unused imports.

DRILL-3822: Have PathScanner use own, not thread-context, class loader.

this closes #166

DRILL-3778: Add missed part of DRILL-3160 (making JDBC Javadoc available).

This closes #158


Configured Javadoc generation (title, package groups, version in headers).

Added link to JDBC page in Drill documentation site.

Edited/fixed some JDBC Javadoc comments.

Added explicit SQLConversionOverflowException to throws clauses for Javadoc


Added some imports for Javadoc references.


Fixed a couple Javadoc syntax errors.

Fixed POM indentation.

DRILL-3661: Update: Added a @link and a @code.

This closes #119

DRILL-3661: Update: Fixed "DRILL-xxxx" reference.

DRILL-2769: Fix most non-SQLException not-supported-yet exceptions.


Added (auto-scanning) unit test. [Drill2769UnsupportedReportsUseSqlExceptionTest]

Added translation of lots of UnsupportedOperationExceptions (and some

RuntimeExceptions) from Avatica code to SQLFeatureNotSupportedExceptions (tons

of method overrides).


Added explicit bounds checks in ResultSetMetaData methods and checking of

last-accessed column in DrillAccessorList.wasNull() (to fix other

RuntimeExceptions to SQLExceptions).

Added resetting of last-accessed column to fix latent bug in DrillAccessorList.


- Renamed some zero-based index/ordinal-position parameters to "...Offset".

- Renamed some one-based index/ordinal-position parameters to "...Number".

- Renamed DrillAccessorList lastColumn to rowLastColumnOffset; declared

explicit logical null value for rowLastColumnOffset.

DRILL-3661: Update: Fixed bad import.

DRILL-2489: Throw exception from remaining methods for closed JDBC objects.

Refactored unit test to check all methods per interface. (Replaced individual,

static test methods with bulk reflection-based checking.)


Added DrillResultSetMetaDataImpl.

Added method overrides to check state for remaining methods from Connection,

Statement, PreparedStatement, ResultSet, ResultSetMetaData and DatabaseMetaData.


- renamed checkNotClosed to throwIfClosed.

DRILL-3661: Edit JDBC doc. for clarity, more consistency, bug fixes.

Also fixed two serialVersionUID literals.

  1. … 13 more files in changeset.
DRILL-3347: VARCHAR ResultSet.getObject returned, not String.

this closes #144

Core fix:

- Fixed {,Nullable}VarCharAccessor's getObject() to return String instead of

value vector's internal

- Updated unit tests (to expect only String now).

[DatabaseMetaDataGetColumnsTest, ResultSetMetaDataTest]

Also Added getObject check in tracing proxy test. [TracingProxyDriverTest]

Changed hard references to Hadoop's Text and JodaTime's Period to strings in

warning check in tracing proxy. [InvocationReporterImpl]


- Added @Override annotations. [SqlAccessors]

- (Unintentionally) fixed (undetected) missing comma. [ValueVectorTypes.tdd]

DRILL-3566: Fix: PreparedStatement.executeQuery() got ClassCastException.


Restored DrillResultSetImpl(...)'s statement parameter from overly

restrictive DrillStatementImpl to AvaticaStatement and removed caller

cast that was throwing. (Relatedly, adjusted getStatement() and moved

internal casting from statement to connection.)

Added basic test of querying via PreparedStatement. [PreparedStatementTest]

Added some case test of statement-creation methods. [ConnectionTest]

Removed now-unneeded DrillPrepareResult cast.

Eliminated redundant getConnection() and getClient calls, intermediate variables, etc.

Fixed comment error.

this closes #143

DRILL-3153: Fix JDBC's getIdentifierQuoteString() to report Drill's backquote.

Added override of Avatica's default implementation returning SQL

standard value. Added Javadoc. Added unit test.

Also moved unit test for nullsAreSortedXxx methods.

DRILL-3153: Update: Comment fixes and whitespace only.

Close apache/drill#99

DRILL-3496: Updates from review comments

DRILL-3496: Update: Multiple log msgs. -> single, aligned multi-line message.

Main: Changed avoidance of original single hard-to-read list from

first draft's multiple single-item log calls (resulting in multiple log

calls/entries) to single log call with aligned multi-line message.

Also revised message wording, etc.

DRILL-3496: Update 2: Some DEBUG -> INFO; another single multi-line msg.

- logger.debug(...) -> for config. file messages

- changed drill-module.conf to single multi-line message like

scanForImplementations cases

DRILL-3496: Update 3: Various review comment responses.

- Removed isTraceEnabled/etc. guards.

- Simplified logging line break/indentation code.

- Added some more "final".

- Undid logger cleanup. :-(

DRILL-3496: Augment logging in DrillConfig and classpath scanning.

Main: Added and augmented logging calls.

Code Hygiene:

- Added/revised doc. comments, code comments.

- Renamed some locals.

- Added "final".

- Wrapped lines.

Also fixed currentTimeMillis -> Stopwatch in Drillbit.



DRILL-2815: Augment PathScanner logging; clean up code.

Added some DEBUG- and TRACE-level log calls.

Augmented/edited existing log call.

Misc. code hygiene:

- Added documentation of forResources(...).

- Renamed to clarify a number of local names.

- Made logger private.

- Added "final".

- Fixed indentation; wrapped some long lines; etc.



DRILL-3483: Clarify CommonConstants' constants. [CommonConstants, DrillConfig, PathScanner]

Renamed constants.

Documented constants.

Removed extraneous "public static final" (redudant and abnormal since in interface).

Edit: Unwrapped a no-longer-long line.

closes #88

DRILL-3151: Fix many ResultSetMetaData method return values.

Added ~unit test for ResultSetMetaData implementation.

Made getObject return classes available to implementation of getColumnClassName:

- Added SqlAccessor.getObjectClass() (to put that metadata right next to code

to which it corresponds rather than in far-away parallel code).

- Added similar AvaticaDrillSqlAccessor.getObjectClass().

- Changed DrillAccessorList.accessors from Accessor[] to

AvaticaDrillSqlAccessor[] for better access to JDBC getObject return class.

- Extracted return classes from accessors to pass to updateColumnMetaData.

Reworked some data type mapping and utilities:

- Added Added Types.getSqlTypeName(...).

- Renamed Types.getJdbcType(...) to getJdbcTypeCode(...)

- Replaced Types.isUnSigned with isJdbcSignedType.

- Fixed various bogus RPC-type XXX -> java.sql.Types.SMALLINT mappings.

- Removed DrillColumnMetaDataList.getJdbcTypeName.

- Moved getAvaticaType up (for bottom-up order).

- Revised DrillColumnMetaDataList.getAvaticaType(...).


- Updated updateColumnMetaData(...) to change many calculations of metadata

input to ColumnMetaData construction. [DrillColumnMetaDataList]

Updated other metadata tests per changes.

DRILL-2494: Have PreparedStmt. set-param. methods throw "unsupported".

Added (integration-level) unit test.

Modified set-parameter methods to throw SQLFeatureNotSupportedException.

(Intercepted common getParameter method.)

Inserted DrillPreparedStatement into hierarchy for place for documentation.

Documented that parameter-setting methods are not supported.

DRILL-3285: Part 3--Invert beforeFirstBatch -> ! afterFirstBatch.

DRILL-3285: Part 2--Renaming.

Renamed state/control-flow members:

- started -> initialSchemaLoaded

- first -> beforeFirstBatch

- redoFirstNext -> returnTrueForNextCallToNext

- finished -> afterLastRow

Renamed other items:

- changed -> schemaChanged

- currentBatch -> currentBatchHolder

- DrillResultSet's currentBatch -> batchLoader

DRILL-3285: Part 4--Reorder fields, updateColumns.

DRILL-3285: Part 1--Prep., Hygiene: Mainly, adding comments.

Added/edited comments:

- field doc. comments

- method doc. comments

- branch/block comments

Removed unused recordBatchCount and getRecordBatchCount().

Added logger call for spurious batch.

Various cleanup:

- Cleaned up logger.

- Added "final" on updateColumns().

- Wrapped some lines

- Misc. comment whitespace.

DRILL-3285: Part 5--Split old hacky next() into separate methods.

Split the original public next() method (which was hacked to handle an extra,

initial call to read the schema batch) into:

- new loadInitialSchema() (for handling the call for the schema)

- modified next() (for handling normal calls from

- new private nextRowInternally() (for common code)

Pulled invariant afterFirstBatch up out of bogus-batch loop.

DRILL-3262: Rename DrillDatabaseMetaData -> DrillDatabaseMetaDataImpl.

Renamed org.apache.drill.jdbc.impl.DrillDatabaseMetaData to

org.apache.drill.jdbc.impl.DrillDatabaseMetaDataImpl (re creating

org.apache.drill.jdbc.DrillDatabaseMetaDataImpl for DRILL-3198).

[DrillDatabaseMetaData -> DrillDatabaseMetaDataImpl, DrillJdbc41Factory]

DRILL-3198: Fix: DatabaseMetaData.getTypeInfo(...), etc., return null.

Fixed DatabaseMetaData.getTypeInfo(...) and similar methods enough to not return

null. (Each method now returns a ResultSet representing a result set with zero

rows, though not with JDBC-specified columns.)

- Changed utility method getEmptyResult().

- Added basic test methods.

- Added interface DrillDatabaseMetaData (for place to document current

Drill-specific behavior).

- Added Javadoc for current behavior.

DRILL-3216: Part 2--Fix existing(+) INFORMATION_SCHEMA.COLUMNS columns.

Added unit test. [TempInformationSchemaColumnsTest]


- Changed -1 to NULL.

- Moved column NUMERIC_PRECISION to correct position.

- Fixed column ORDINAL_POSITION from 0-based to 1-based.

- Moved CHAR length to column CHARACTER_MAXIMUM_LENGTH.


- Added precision and scale values for integer types and floating-point types.

- Added column COLUMN_DEFAULT.



- Added column INTERVAL_TYPE.


- Fixed column DATA_TYPE to correct form of type names:


- short (e.g., "CHAR") to specified (e.g., "CHARACTER")

Applied COLUMNS to JDBC DatabaseMetaData.getColumns() implementation:

- Changed some getColumns() result values:

- type names (DATA_TYPE changes)

- precision radix for some cases

- Adapted existing uses of COLUMNS data.

- Applied new COLUMNS data (e.g., interval precision).

- Updated getColumns() test (re changed result values).

- Augmented getColumns() test (e.g., more intervals).

Updated other tests using INFORMATION_SCHEMA.COLUMNS:

- -1 -> NULL

- integer types have precision and scale values

- "CHAR" -> "CHARACTER", etc.

[TestInfoSchema, TestInfoSchemaOnHiveStorage, TestInformationSchemaColumns,


DRILL-3216: Part 1--Pre-core code hygiene.

Many-instance changes: [DatabaseMetaDataGetColumnsTest]

- Renamed "...opt..." columns and methods to match "...mdrOpt" variables.

- Renamed "...req..." columns and methods to match "...mdrReq" variables.

- Renamed "testRow..." variables to "mdrUnk".

- Added/applied getIntOrNull(...) especially to simplify null cases.

- Put REAL, FLOAT, and DOUBLE in that order (smallest, ~variable, largest).

- (Edited "nullability code" -> "nullability code:".)

Renamed "type" -> "relDataType"; "sqlType" -> "sqlTypeName".

Changed "dfs_test.tmp" -> VIEW_SCHEMA.

Re-aligned query SQL.

Added 2 "final".

Edited 3253 (test plugin) TODOs.

Purged several comments.

Wrapped some lines.

DRILL-3159: Part 1--Prep., Hyg. for: Make JDBC throttling threshold configurable.

Cleaned, enhanced DrillResultSet:

- Enhanced ResultsListener logging:

- Added instance ID; added batch numbers.

- Added logging at close (pairing with logging at construction).

- Fixed 2-integer query ID to UUID form.

- Renamed qrb -> qdb; q -> qdb (per recent QueryDataBatch change).

- Added "final" on ResultsListener's logger.

Reduced Avatica-vs.-Drill casting:

- DrillStatementImpl's (Drill)Connection(Impl).

- DrillResultSetImpl's (Drill)Statement(Impl).

Converted a comment in ExecConstants.

DRILL-3159: Part 2--Core: Make JDBC throttling threshold configurable.

Added configuration/option "drill.jdbc.batch_queue_throttling_threshold".

Applied "drill.jdbc.batch_queue_throttling_threshold" to DrillResultSetImpl.