Clone Tools
  • last updated 25 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-4237, DRILL-4478: Implement hash to use murmur3 and add correspondent unit tests

+ Avoid object or extra buffer creation

+ Clean up tests

closes #485

  1. … 13 more files in changeset.
DRILL-4184: Support variable length decimal fields in parquet

  1. … 68 more files in changeset.
DRILL-4203: Fix DrillVersionInfo to make it provide a valid version number even during the unit tests.

This is now a build-time generated class, rather than one that looks on the

classpath for META-INF files.

This pattern for file generation with parameters passed from the POM files

was borrowed from parquet-mr.

  1. … 2 more files in changeset.
DRILL-2618: handle queries over empty folders consistently so that they report table not found rather than failing. Refactor FileSelection to eliminate redundancy, make it more managable Fix WorkspaceSchemaFactory to handle empty folders. Introduce ParquetFileSelection, a sub-class of FileSelection that carries along metadata cache Fix MagicStringMatcher so that it operate on files only. Unit test file selection

  1. … 13 more files in changeset.
DRILL-3987: (MOVE) Move logical expressions and operators out of common. Move to new drill-logical model.

  1. … 212 more files in changeset.
DRILL-2288: Fix ScanBatch violation of IterOutcome protocol and downstream chain of bugs.

Increments:

2288: Pt. 1 Core: Added unit test. [Drill2288GetColumnsMetadataWhenNoRowsTest, empty.json]

2288: Pt. 1 Core: Changed HBase test table #1's # of regions from 1 to 2. [HBaseTestsSuite]

Also added TODO(DRILL-3954) comment about # of regions.

2288: Pt. 2 Core: Documented IterOutcome much more clearly. [RecordBatch]

Also edited some related Javadoc.

2288: Pt. 2 Hyg.: Edited doc., added @Override, etc. [AbstractRecordBatch, RecordBatch]

Purged unused SetupOutcome.

Added @Override.

Edited comments.

Fix some comments to doc. comments.

2288: Pt. 3 Core&Hyg.: Added validation of IterOutcome sequence. [IteratorValidatorBatchIterator]

Also:

Renamed internal members for clarity.

Added comments.

2288: Pt. 4 Core: Fixed a NONE -> OK_NEW_SCHEMA in ScanBatch.next(). [ScanBatch]

(With nearby comments.)

2288: Pt. 4 Hyg.: Edited comments, reordered, whitespace. [ScanBatch]

Reordered

Added comments.

Aligned.

2288: Pt. 4 Core+: Fixed UnionAllRecordBatch to receive IterOutcome sequence right. (3659) [UnionAllRecordBatch]

2288: Pt. 5 Core: Fixed ScanBatch.Mutator.isNewSchema() to stop spurious "new schema" reports (fix short-circuit OR, to call resetting method right). [ScanBatch]

2288: Pt. 5 Hyg.: Renamed, edited comments, reordered. [ScanBatch, SchemaChangeCallBack, AbstractSingleRecordBatch]

Renamed getSchemaChange -> getSchemaChangedAndReset.

Renamed schemaChange -> schemaChanged.

Added doc. comments.

Aligned.

2288: Pt. 6 Core: Avoided dummy Null.IntVec. column in JsonReader when not needed (MapWriter.isEmptyMap()). [JsonReader, 3 vector files]

2288: Pt. 6 Hyg.: Edited comments, message. Fixed message formatting. [RecordReader, JSONFormatPlugin, JSONRecordReader, AbstractMapVector, JsonReader]

Fixed message formatting.

Edited comments.

Edited message.

Fixed spurious line break.

2288: Pt. 7 Core: Added column families in HBaseRecordReader* to avoid dummy Null.IntVec. clash. [HBaseRecordReader]

2288: Pt. 8 Core.1: Cleared recordCount in OrderedPartitionRecordBatch.innerNext(). [OrderedPartitionRecordBatch]

2288: Pt. 8 Core.2: Cleared recordCount in ProjectRecordBatch.innerNext. [ProjectRecordBatch]

2288: Pt. 8 Core.3: Cleared recordCount in TopNBatch.innerNext. [TopNBatch]

2288: Pt. 9 Core: Had UnorderedReceiverBatch reset RecordBatchLoader's record count. [UnorderedReceiverBatch, RecordBatchLoader]

2288: Pt. 9 Hyg.: Added comments. [RecordBatchLoader]

2288: Pt. 10 Core: Worked around mismatched map child vectors in MapVector.getObject(). [MapVector]

2288: Pt. 11 Core: Added OK_NEW_SCHEMA schema comparison for HashAgg. [HashAggTemplate]

2288: Pt. 12 Core: Fixed memory leak in BaseTestQuery's printing.

Fixed bad skipping of RecordBatchLoader.clear(...) and

QueryDataBatch.load(...) for zero-row batches in printResult(...).

Also, dropped suppression of call to

VectorUtil.showVectorAccessibleContent(...) (so zero-row batches are

as visible as others).

2288: Pt. 13 Core: Fixed test that used unhandled periods in column alias identifiers.

2288: Misc.: Added # of rows to showVectorAccessibleContent's output. [VectorUtil]

2288: Misc.: Added simple/partial toString() [VectorContainer, AbstractRecordReader, JSONRecordReader, BaseValueVector, FieldSelection, AbstractBaseWriter]

2288: Misc. Hyg.: Added doc. comments to VectorContainer. [VectorContainer]

2288: Misc. Hyg.: Edited comment. [DrillStringUtils]

2288: Misc. Hyg.: Clarified message for unhandled identifier containing period.

2288: Pt. 3 Core&Hyg. Upd.: Added schema comparison result to logging. [IteratorValidatorBatchIterator]

2288: Pt. 7 Core Upd.: Handled HBase columns too re NullableIntVectors. [HBaseRecordReader, TestTableGenerator, TestHBaseFilterPushDown]

Created map-child vectors for requested columns.

Added unit test method testDummyColumnsAreAvoided, adding new row to test table,

updated some row counts.

2288: Pt. 7 Hyg. Upd.: Edited comment. [HBaseRecordReader]

2288: Pt. 11 Core Upd.: REVERTED all of bad OK_NEW_SCHEMA schema comparison for HashAgg. [HashAggTemplate]

This reverts commit 0939660f4620c03da97f4e1bf25a27514e6d0b81.

2288: Pt. 6 Core Upd.: Added isEmptyMap override in new (just-rebased-in) PromotableWriter. [PromotableWriter]

Adjusted definition and default implementation of isEmptyMap (to handle MongoDB

storage plugin's use of JsonReader).

2288: Pt. 6 Hyg. Upd.: Purged old atLeastOneWrite flag. [JsonReader]

2288: Pt. 14: Disabled newly dying test testNestedFlatten().

  1. … 39 more files in changeset.
DRILL-3742: Classpath scanning and build improvement

Makes the classpath scanning a build time class discovery

Makes the fmpp generation incremental

Removes some slowness in DrillBit closing

Reduces the build time by 30%

This closes #148

  1. … 143 more files in changeset.
DRILL-2726: Add Drill version, from Manifest, same as SqlLine in the sys.version.

This closes #198

    • -0
    • +58
    ./DrillVersionInfo.java
  1. … 1 more file in changeset.
DRILL-3822: Have PathScanner use own, not thread-context, class loader.

this closes #166

DRILL-3778: Add missed part of DRILL-3160 (making JDBC Javadoc available).

This closes #158

Main:

Configured Javadoc generation (title, package groups, version in headers).

Added link to JDBC page in Drill documentation site.

Edited/fixed some JDBC Javadoc comments.

Added explicit SQLConversionOverflowException to throws clauses for Javadoc

effect.

Added some imports for Javadoc references.

Misc.:

Fixed a couple Javadoc syntax errors.

Fixed POM indentation.

  1. … 11 more files in changeset.
DRILL-3589: Update JDBC driver to shade and minimize dependencies.

Update build process to use shading, stop using Proguard.

Add simple integration test that verifies that the JDBC driver works

correctly with a clean classpath.

  1. … 9 more files in changeset.
DRILL-3496: Updates from review comments

DRILL-3496: Update: Multiple log msgs. -> single, aligned multi-line message.

Main: Changed avoidance of original single hard-to-read list from

first draft's multiple single-item log calls (resulting in multiple log

calls/entries) to single log call with aligned multi-line message.

Also revised message wording, etc.

DRILL-3496: Update 2: Some DEBUG -> INFO; another single multi-line msg.

- logger.debug(...) -> logger.info(...) for config. file messages

- changed drill-module.conf to single multi-line message like

scanForImplementations cases

DRILL-3496: Update 3: Various review comment responses.

- Removed isTraceEnabled/etc. guards.

- Simplified logging line break/indentation code.

- Added some more "final".

- Undid logger cleanup. :-(

  1. … 7 more files in changeset.
DRILL-2815: Augment PathScanner logging; clean up code.

Added some DEBUG- and TRACE-level log calls.

Augmented/edited existing log call.

Misc. code hygiene:

- Added documentation of forResources(...).

- Renamed to clarify a number of local names.

- Made logger private.

- Added "final".

- Fixed indentation; wrapped some long lines; etc.

Conflicts:

common/src/main/java/org/apache/drill/common/util/PathScanner.java

DRILL-3483: Clarify CommonConstants' constants. [CommonConstants, DrillConfig, PathScanner]

Renamed constants.

Documented constants.

Removed extraneous "public static final" (redudant and abnormal since in interface).

Edit: Unwrapped a no-longer-long line.

closes #88

  1. … 2 more files in changeset.
DRILL-2903: General improvements to tests in TestDrillbitResilience + Added RepeatTestRule to tests that are flaky + Added Controls.Builder to create controls string in tests + Added @Ignore to failing tests (filed JIRAs)

Other fixes:

+ Added @Override to ScanBatch#close to avoid potential bugs

+ Added docs link in ProtobufLengthDecoder

+ Fixed logging issue in CountDownLatchImpl

    • -0
    • +72
    ./RepeatTestRule.java
  1. … 16 more files in changeset.
DRILL-2848: Hygiene - ParquetToDrillConverter cleanup

  1. … 1 more file in changeset.
DRILL-1385, along with some cleanup

Cleaned up option handling. This includes using finals, making member variables

private whenever possible, and some formatting.

- fixed a bug in the string formatting for the double range validator

- OptionValidator, OptionValue, and their implementations now conspire not to

allow the creation of malformed options because the OptionType has been added

to validator calls to handle OptionValues that are created on demand.

Started with updated byte code rewrite from Jacques

Fixed several problems with scalar value replacement:

- use consistent ASM api version throughout

- stop using deprecated ASM methods (actually causes bugs)

- visitMethodInsn()

- added a couple of missing super.visitEnd()s

- fixed a couple of minor FindBugs issues

- accounted for required stack size increases when replacing holders for

longs and doubles

- added accounting for frame offsets to cope with long and double local

variables and value holder members

- fixed a few minor bugs found with FindBugs

- stop using carrotlabs' hash map lget() method on shared constant data

- fixed an incorrect use of DUP2 on objectrefs when copying long or double

holder members into locals

- fixed a problem with redundant POP instructions left behind after replacement

- fixed a problem with incorrect DUPs in multiple assignment statements

- fixed a problem with DUP_X1 replacement when handling constants in multiple

assignment statements

- fixed a problem with non-replaced holder member post-decrements

- don't replace holders passed to static functions as "out" parameters

(common with Accessors on repeated value vectors)

- increased the maximum required stack size when transferring holder members to

locals

- changed the code generation block type mappings for constants for external

sorts

- fixed problems handling constant and non-constant member variables in

operator classes

- in general, if a holder is assigned to or from an operator member variable,

it can't be replaced (at least not until we replace those as well)

- Use a derived ASM Analyzer (MethodAnalyzer) and Frame

(AssignmentTrackingFrame) in order to establish relationships between

assignments of holders through chains of local variables. This effectively

back-propagates non-replaceability attributes so that if a holder variable

that can't be replaced is assigned to from another holder variable, that

second one cannot be replaced either, and so on through longer chains of

assignments.

- code for dumping generated source code

- MergeAdapter dumps before and after results of scalar replacement

(if it's on)

- fixed some problems in ReplacingBasicValue by replacing HashSet with

IdentityHashMap

- made loggers private

- added a retry strategy for scalar replacement

if a scalar replacement code rewriting fails, then this will try to

regenerate the bytecode again without the scalar replacement.

- bytecode verification is always on now (required for the retry strategy)

- use system option to determine whether scalar replacement should be used

- default option: if scalar replacement fails, retry without it

- force replacement on or off

- unit tests for the retry strategy are based on a single known failure case,

covered by DRILL-2326.

- add tests TestConvertFunctions to test the three scalar replacement options

for the failing test case (testVarCharReturnTripConvertLogical)

- made it possible to set a SYSTEM option as a java property in Drillbit

- added a command line argument to force scalar replacement to be on during

testing in the rootmost pom.xml

In the course of this, added increased checking of intermediate stages of code

rewriting, as well as logging of classes that cause failures.

- work around a bug in ASM's CheckClassAdapter that doesn't allow for checking

of inner classes

Added comments, tidied up formatting, and added "final" in a number of places.

Signed-off-by: vkorukanti <venki.korukanti@gmail.com>

  1. … 53 more files in changeset.
DRILL-2199: Fix major type returned for convert to nullable type functions

  1. … 4 more files in changeset.
DRILL-2049: NoClassDefFoundError: org/apache/commons/lang/StringEscapeUtils in JDBC Driver

+ Ban commons-lang v2 classes from being used.

  1. … 7 more files in changeset.
DRILL-1904 - Part 1: Package level docs for the common module and a few packages in exec.

Fix line wrapping on package level docs to standardize at 80 character wrap.

Patch updated to address review comments.

Patch updated again to address Sudheesh's review comments.

  1. … 28 more files in changeset.
DRILL-1402: Add check-style rules for trailing space, TABs and blocks without braces

  1. … 438 more files in changeset.
DRILL-634: Cleanup/organize Java imports and trailing whitespaces from Drill code

  1. … 765 more files in changeset.
DRILL-1281: Read into Direct Memory in Parquet Reader. Requires Hadoop 2.4 or above

  1. … 28 more files in changeset.
DRILL-1155: Add option to use JDK compiler and use it for runtime generated code over a certain threshold

+ Updating Janino to 2.7.4 (from 2.6.1).

+ Added session options `exec.java_compiler`:[DEFAULT, JDK, JANINO] and `exec.java_compiler_janino_maxsize`(256k).

+ Fix a bug in FragmentExecutor.java(125) where the root operator is not stopped in case of fragment failure.

+ Print Java stack trace of the allocator code if a TLA detects that any of the ChildAllocator is not closed.

  1. … 22 more files in changeset.
Switch to DrillBuf Add @Inject DrillBuf Move comparison functions to memory sensitive ones Add scalar replacement functionality for value holders Simplify date parsing function Add local compiled code caching

    • -0
    • +76
    ./CoreDecimalUtility.java
    • -1
    • +2
    ./DecimalScalePrecisionDivideFunction.java
    • -1
    • +2
    ./DecimalScalePrecisionModFunction.java
  1. … 210 more files in changeset.
DRILL-730: Initialize the buffer while creating sparse representation from BigDecimal

  1. … 4 more files in changeset.
DRILL-1110: Output precision of functions should always be greater or equal to input precision

    • -1
    • +4
    ./DecimalScalePrecisionDivideFunction.java
    • -0
    • +43
    ./DecimalScalePrecisionModFunction.java
  1. … 4 more files in changeset.
DRILL-861: Implement sum, avg for decimal data type.

  1. … 9 more files in changeset.
DRILL-1085: Round up while casting to int/bigint from float/double/decimal

  1. … 4 more files in changeset.
DRILL-794: Fix IOB while performing decimal subtraction

    • -0
    • +40
    ./DecimalScalePrecisionAddFunction.java
    • -12
    • +8
    ./DecimalScalePrecisionDivideFunction.java
    • -11
    • +6
    ./DecimalScalePrecisionMulFunction.java
    • -0
    • +37
    ./DrillBaseComputeScalePrecision.java
  1. … 6 more files in changeset.