drill

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
minor typo fix

DRILL-4025: Create FileSelection object with the appropriate file statuses

DRILL-3810: Add Avro schema support

Also updates the FileFormatPlugin to support providing schema.

DRILL-3956: Add support for MySQL text type

DRILL-4006: Reallocate offset vector in repeated vectors when index is beyond the current capacity

Author: Steven Phillips <smp@apache.org>

This closes #243, #242

DRILL-4000: Ensure storage plugins are not needlessly created. Add start and close capability to storage plugins.

This closes #227

Add a new configuration based map so any configurations that are passed to a node can leverage existing storage plugins.

Update FileSystemConfig to correctly implement hashcode()

Update StoragePlugin interface to extends Autocloseable and add start() method.

Update Mongo plugin to close clients when closing plugin.

DRILL-3994: make classpath scanning work on windows

DRILL-3634: Add number of partitions information in the plan for hive scans

closes #231

DRILL-4053: Reduce metadata cache file size. Save a merged schema instead of repeating for every row group. Save maxValue in a row group iff minVal equals maxVal and null otherwise. Maintain backward compatibility with older version. This closes #254

DRILL-3992: Add/fix support for JDBC schemas (tested against oracle and derby)

This closes #225

DRILL-2288: Fix ScanBatch violation of IterOutcome protocol and downstream chain of bugs.

Increments:

2288: Pt. 1 Core: Added unit test. [Drill2288GetColumnsMetadataWhenNoRowsTest, empty.json]

2288: Pt. 1 Core: Changed HBase test table #1's # of regions from 1 to 2. [HBaseTestsSuite]

Also added TODO(DRILL-3954) comment about # of regions.

2288: Pt. 2 Core: Documented IterOutcome much more clearly. [RecordBatch]

Also edited some related Javadoc.

2288: Pt. 2 Hyg.: Edited doc., added @Override, etc. [AbstractRecordBatch, RecordBatch]

Purged unused SetupOutcome.

Added @Override.

Edited comments.

Fix some comments to doc. comments.

2288: Pt. 3 Core&Hyg.: Added validation of IterOutcome sequence. [IteratorValidatorBatchIterator]

Also:

Renamed internal members for clarity.

Added comments.

2288: Pt. 4 Core: Fixed a NONE -> OK_NEW_SCHEMA in ScanBatch.next(). [ScanBatch]

(With nearby comments.)

2288: Pt. 4 Hyg.: Edited comments, reordered, whitespace. [ScanBatch]

Reordered

Added comments.

Aligned.

2288: Pt. 4 Core+: Fixed UnionAllRecordBatch to receive IterOutcome sequence right. (3659) [UnionAllRecordBatch]

2288: Pt. 5 Core: Fixed ScanBatch.Mutator.isNewSchema() to stop spurious "new schema" reports (fix short-circuit OR, to call resetting method right). [ScanBatch]

2288: Pt. 5 Hyg.: Renamed, edited comments, reordered. [ScanBatch, SchemaChangeCallBack, AbstractSingleRecordBatch]

Renamed getSchemaChange -> getSchemaChangedAndReset.

Renamed schemaChange -> schemaChanged.

Added doc. comments.

Aligned.

2288: Pt. 6 Core: Avoided dummy Null.IntVec. column in JsonReader when not needed (MapWriter.isEmptyMap()). [JsonReader, 3 vector files]

2288: Pt. 6 Hyg.: Edited comments, message. Fixed message formatting. [RecordReader, JSONFormatPlugin, JSONRecordReader, AbstractMapVector, JsonReader]

Fixed message formatting.

Edited comments.

Edited message.

Fixed spurious line break.

2288: Pt. 7 Core: Added column families in HBaseRecordReader* to avoid dummy Null.IntVec. clash. [HBaseRecordReader]

2288: Pt. 8 Core.1: Cleared recordCount in OrderedPartitionRecordBatch.innerNext(). [OrderedPartitionRecordBatch]

2288: Pt. 8 Core.2: Cleared recordCount in ProjectRecordBatch.innerNext. [ProjectRecordBatch]

2288: Pt. 8 Core.3: Cleared recordCount in TopNBatch.innerNext. [TopNBatch]

2288: Pt. 9 Core: Had UnorderedReceiverBatch reset RecordBatchLoader's record count. [UnorderedReceiverBatch, RecordBatchLoader]

2288: Pt. 9 Hyg.: Added comments. [RecordBatchLoader]

2288: Pt. 10 Core: Worked around mismatched map child vectors in MapVector.getObject(). [MapVector]

2288: Pt. 11 Core: Added OK_NEW_SCHEMA schema comparison for HashAgg. [HashAggTemplate]

2288: Pt. 12 Core: Fixed memory leak in BaseTestQuery's printing.

Fixed bad skipping of RecordBatchLoader.clear(...) and

QueryDataBatch.load(...) for zero-row batches in printResult(...).

Also, dropped suppression of call to

VectorUtil.showVectorAccessibleContent(...) (so zero-row batches are

as visible as others).

2288: Pt. 13 Core: Fixed test that used unhandled periods in column alias identifiers.

2288: Misc.: Added # of rows to showVectorAccessibleContent's output. [VectorUtil]

2288: Misc.: Added simple/partial toString() [VectorContainer, AbstractRecordReader, JSONRecordReader, BaseValueVector, FieldSelection, AbstractBaseWriter]

2288: Misc. Hyg.: Added doc. comments to VectorContainer. [VectorContainer]

2288: Misc. Hyg.: Edited comment. [DrillStringUtils]

2288: Misc. Hyg.: Clarified message for unhandled identifier containing period.

2288: Pt. 3 Core&Hyg. Upd.: Added schema comparison result to logging. [IteratorValidatorBatchIterator]

2288: Pt. 7 Core Upd.: Handled HBase columns too re NullableIntVectors. [HBaseRecordReader, TestTableGenerator, TestHBaseFilterPushDown]

Created map-child vectors for requested columns.

Added unit test method testDummyColumnsAreAvoided, adding new row to test table,

updated some row counts.

2288: Pt. 7 Hyg. Upd.: Edited comment. [HBaseRecordReader]

2288: Pt. 11 Core Upd.: REVERTED all of bad OK_NEW_SCHEMA schema comparison for HashAgg. [HashAggTemplate]

This reverts commit 0939660f4620c03da97f4e1bf25a27514e6d0b81.

2288: Pt. 6 Core Upd.: Added isEmptyMap override in new (just-rebased-in) PromotableWriter. [PromotableWriter]

Adjusted definition and default implementation of isEmptyMap (to handle MongoDB

storage plugin's use of JsonReader).

2288: Pt. 6 Hyg. Upd.: Purged old atLeastOneWrite flag. [JsonReader]

2288: Pt. 14: Disabled newly dying test testNestedFlatten().

  1. … 25 more files in changeset.
minor edit

typos, add direct to drillbit connection

add example of flatten

formatting

    • -0
    • +11
    /_docs/install/050-starting-drill-in-distributed-mode.md
DRILL-3983: Small test improvements improve error message when SQL parsing error add a simple test to Parquet writer make errors verbose by default in tests

This closes #221

Also includes:

- make the checkstyle conf not tied to the execution to allow running mvn checkstyle:check from the command line

- fix verbose output in tests

DRILL-3921: Initialize the underlying record reader lazily in HiveRecordReader

This closes #197

+ OperatorContextImpl has a lazily initialized executor service that is

a decorator around the worker pool. This service is used through

#runCallableAs to run a callable task as the given proxy user.

+ Unit tests use partition student table with chained impersonation to

exercise running multiple delegate callables (creates multiple readers,

a reader per partition).

+ Remove unused WorkManager#shutdownAndAwaitTermination method

DRILL-3937: Handle the case where min/max columns in metadata cache file are string or binary values.

Simplify serialization and check for nulls. Remove byte array comparison for min/max since it is not needed anymore.

close apache/drill#220

DRILL-3340: Part 2: Reverting 1a589ab and committing latest patch

Add operator metrics registry for metric definitions

+ Display metrics as a table within an operator profile panel

+ Rename FragmentStats#getOperatorStats to newOperatorStats

DRILL-3975: Make sure to strip scheme and authority from partition location.

This closes #217

Filter push-down support for JSON tables.

misc edits

DRILL-3941: Add timing instrumentation in different phases of partition pruning Add basic timing instrumentation while initializing FileSelection and reading the metadata cache file

closes #230

DRILL-3980: Upgrade hadoop dependency version to 2.7.x in mapr profile

DRILL-3952: Improve Window Functions performance when not all batches are required to process the current batch

this closes #222

DRILL-3229: Miscellaneous Union-type fixes

closes #207

closes #180

  1. … 24 more files in changeset.
DRILL-3963: Add Sequence file support.

- File Support

- Add test for sequencefile reader in Impersonation test suite.

- Create hadoop based record reader under user proxy.

- Fix impersonation test for sequence file and add one for Avro.

This closes #214

    • binary
    /exec/java-exec/src/test/resources/sequencefiles/simple.seq
DRILL-3739: Fix issues in reading Hive tables with StorageHandler configuration (eg. Hive-HBase tables)

DRILL-3893: Change Hive metadata cache invalidation policy to "1 min after last write".

remove web ui security straggler, reorg plugins, add jdbc

minor edits

more edits

more edits

fix curl cmd

fix rn date, link to jdbc plugin doc

minor edit

another web ui security straggler

reference to web ui security

web ui security leftover

minor edit

formatting

yet another web ui security reference

Ted's changes

edits to workarounds

minor tweaks

complex nested data edit

wordsmithing

Uwe's changes

minor tweak

minor edit

consistency issue

pointer to rn for DRILL 3956

nav changes

javadoc

Ted's table alias workaround

links to javadoc

  1. … 65 more files in changeset.
DRILL-3505: Add the _id when using a * projection in Mongo

This closes #210

DRILL-3965: Fix IOOB while trying to perform partition pruning

DRILL-1752: Fixing query failures of unsharded db/collection in sharded cluster

- Add logback.xml in test/resources

- Add testcases and upgrading pom version of mongo and mongo test dependency

    • -0
    • +59
    /contrib/storage-mongo/src/test/resources/logback.xml