common

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-5325: Unit tests for the managed sort

Uses the sub-operator test framework (DRILL-5318), including the test

row set abstraction (DRILL-5323) to enable unit testing of the

“managed” external sort. This PR allows early review of the code, but

cannot be pulled until the dependencies (mentioned above) are pulled.

Refactors the external sort code into small chunks that can be unit

tested, then “wraps” that code in tests for all interesting data types,

record batch sizes, and so on.

Refactors some of the operator definitions to more easily allow

programmatic setup in the unit tests.

Fixes a number of bugs discovered by the unit tests. The biggest

changes were in the new code: the code that computes spilling and

merging based on memory levels.

Otherwise, although GitHub will show many files change, most of the

changes are simply moving blocks of code around to create smaller units

that can be tested independently.

Includes a refactoring of the code that does spilling, along with a

complete set of low-level unit tests.

Excludes long-running sort tests.

Defines a test category for long-running tests.

First attempt to provide a way to run such tests from Maven.

closes #808

    • -0
    • +70
    ./src/main/java/org/apache/drill/test/SecondaryTest.java
  1. … 50 more files in changeset.
DRILL-5419: Calculate return string length for literals & some string functions

1. Revisited calculation logic for string literals and some string functions

(cast, upper, lower, initcap, reverse, concat, concat operator, rpad, lpad, case statement,

coalesce, first_value, last_value, lag, lead).

Synchronized return type length calculation logic between limit 0 and regular queries.

2. Deprecated width and changed it to precision for string types in MajorType.

3. Revisited FunctionScope and splitted it into FunctionScope and ReturnType.

FunctionScope will indicate only function usage in term of number of in / out rows, (n -> 1, 1 -> 1, 1->n).

New annotation in UDFs ReturnType will indicate which return type strategy should be used.

4. Changed MAX_VARCHAR_LENGTH from 65536 to 65535.

5. Updated calculation of precision and display size for INTERVALYEAR & INTERVALDAY.

6. Refactored part of function code-gen logic (ValueReference, WorkspaceReference, FunctionAttributes, DrillFuncHolder).

This closes #819

  1. … 78 more files in changeset.
Update version to 1.11.0-SNAPSHOT

  1. … 27 more files in changeset.
[maven-release-plugin] prepare release drill-1.10.0

  1. … 27 more files in changeset.
DRILL-5326: Unit tests failures related to the SERVER_METADTA

- adding of the sql type name for the "GENERIC_OBJECT";

- changing "NullCollation" in the "ServerMetaProvider" to the correct default value;

- changing RpcType to GET_SERVER_META in the appropriate ServerMethod

close #775

  1. … 3 more files in changeset.
DRILL-5260: Extend "Cluster Fixture" test framework

- Config option to suppress printing of CSV and other output. (Allows

printing for single tests, not printing when running from Maven.)

- Parsing of query profiles to extract plan and run time information.

- Fix bug in log fixture when enabling logging for a package.

- Improved ZK support.

- Set up the new CTTAS default temporary workspace for tests.

- Clean up persistent storage files on disk to avoid CTTAS startup

failures.

- Provides a set of examples for how to use the cluster fixture.

closes #753

  1. … 17 more files in changeset.
DRILL-4335: Apache Drill should support network encryption.

NOTE: This pull request provides support for on-wire encryption using SASL framework. The communication channel that are covered are:

1) Between Drill JDBC client and Drillbit.

2) Between Drillbit to Drillbit i.e. control/data channels.

3) It has UI change to view encryption is enabled on which network channel and number of encrypted/unencrypted connections for

user/control/data connections.

close apache/drill#773

  1. … 60 more files in changeset.
DRILL-4280: CORE (user to bit authentication, Java)

+ Add logic for authentication in UserClient and UserServer with

backward compatibility in both directions

+ Add abstract extension to ServerConnection and ClientConnection

+ Add concrete extensions to abstract connections:

BitToUserConnection and UserToBitConnection

+ Add ConnectionConfig interface with abstract and concrete

implementations to encapsulate configuration for server-side

connections

+ Encapsulate all requests handled by UserServer in

UserServerRequestHandler

+ Clear UserSession when connection is closed either by user or

bit

+ Add DrillProperties to encapsulate all connection properties

used during connection time

  1. … 15 more files in changeset.
DRILL-4280: HYGIENE

+ Do not recreate DrillConfig object in PamUserAuthenticator

+ Add new factory method to CaseInsensitiveMap

+ Clean documentation

  1. … 2 more files in changeset.
DRILL-4280: CORE (service login)

+ Support Drillbit login to KDC using Hadoop's

UserGroupInformation library

+ Set hostname in BootstrapContext

+ Use process user's short name in ImpersonationUtil

+ Add KerberosUtil class

    • -0
    • +93
    ./src/main/java/org/apache/drill/common/KerberosUtil.java
  1. … 4 more files in changeset.
Update version to 1.10.0-SNAPSHOT

  1. … 27 more files in changeset.
DRILL-5056: Fix UserException to write full message to log

A case occurred in which the External Sort failed during spilling.

All that was written to the log was:

2016-11-18 ... INFO o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred

Modified the logging code to provide more information to aid tracking down problems that occur in the field.

closes #665

[maven-release-plugin] prepare release drill-1.9.0

  1. … 27 more files in changeset.
DRILL-4994: Add back JDBC prepared statement for older servers

When the JDBC client is connected to an older Drill server, it always

attempted to use server-side prepared statement with no fallback.

With this change, client will check server version and will fallback to the

previous client-side prepared statement (which is still limited to only execute

queries and does not provide metadata).

close #613

    • -0
    • +157
    ./src/main/java/org/apache/drill/common/Version.java
    • -0
    • +103
    ./src/test/java/org/apache/drill/common/TestVersion.java
  1. … 22 more files in changeset.
DRILL-1950: Parquet rowgroup level filter pushdown in query planning time.

Implement Parquet rowgroup level filter pushdown. The filter pushdown is performed in

in Drill physical planning phase.

Only a local filter, which refers to columns in a single table, is qualified for filter pushdown.

A filter may be qualified if it is a simple comparison filter, or a compound "and/or" filter consists of

simple comparison filter. Data types allowed in comparison filter are int, bigint, float, double, date,

timestamp, time. Comparison operators are =, !=, <, <=, >, >=. Operands have to be a column of the above

data types, or an explicit cast or implicit cast function, or a constant expressions.

This closes #637

  1. … 43 more files in changeset.
DRILL-4969: Basic implementation of display size for column metadata

Add a simple implementation of display size, based on the ODBC table

available at: https://msdn.microsoft.com/en-us/library/ms713974(v=vs.85).aspx

closes #632

  1. … 5 more files in changeset.
DRILL-4369: Exchange name and version infos during handshake

There's no name and version exchanged between client and server over the User RPC

channel.

On client side, having access to the server name and version is useful to expose it

to the user (through JDBC or ODBC api like DatabaseMetadata#getDatabaseProductVersion()),

or to implement fallback strategy when some recent API are not available (like

metadata API).

On the server side, having access to the client version might be useful for audit

purposes and eventually to implement fallback strategy if it doesn't require a RPC

version change.

this closes #622

  1. … 21 more files in changeset.
DRILL-4945: Report INTERVAL exact type as column type name

closes #618

  1. … 1 more file in changeset.
DRILL-4800: Various fixes. Fix buffer underflow exception in BufferedDirectBufInputStream. Fix writer index for in64 dictionary encoded types. Added logging to help debug. Fix memory leaks. Work around issues with of InputStream.available() ( Do not use hasRemainder; Remove check for EOF in BufferedDirectBufInputStream.read() ). Finalize defaults. Remove commented code. Addressed review comments

This closes #611

  1. … 11 more files in changeset.
Update version to 1.9.0-SNAPSHOT.

  1. … 26 more files in changeset.
DRILL-4862: Fix binary_string to use an injected buffer as out buffer

+ Previously, binary_string used the input buffer as output buffer. So after calling binary_string, the original content was destroyed. Other expressions/ functions that need to access the original input buffer get wrong results.

+ This fix also sets readerIndex and writerIndex correctly for the output buffer, otherwise the consumer of the output buffer will hit issues.

closes #604

  1. … 3 more files in changeset.
[maven-release-plugin] prepare release drill-1.8.0

  1. … 26 more files in changeset.
DRILL-4800: Parallelize column reading. Read/Decode fixed width fields in parallel Decoding var length columns in parallel Use simplified decompress method for Gzip and Snappy decompression. Avoids concurrency issue with Parquet decompression. (It's also faster). Stress test Parquet read write Parallel column reader is disabled by default (may perform less well under higher concurrency)

  1. … 10 more files in changeset.
DRILL-4726: Dynamic UDF Support

1) Configuration / parsing / options / protos

2) Zookeeper integration

3) Registration / unregistration / lazy-init

4) Unit tests

This closes #574

  1. … 70 more files in changeset.
DRILL-4452: Uses Apache Calcite Avatica driver vs Optiq driver for Drill JDBC

Drill JDBC driver uses Optiq Avatica as its basis, but this dependency has

been moved over to Calcite, for quite some time without Drill code being

updated for it.

This patch updates Avatica version to the version from Calcite

(1.4.0-drill-r19). It also refactors Drill JDBC driver to comply with the

packages and API changes in Avatica. Finally it fixes the the SQL types for

lists and structs, since Drill doesn't support java.sql.Array and

java.sql.Struct interfaces.

this closes #395

Change-Id: Ia608adf900e8708d9e6f6f58ed41e104321a9914

  1. … 22 more files in changeset.
DRILL-4499: Remove 16 unused classes

This closes #426

  1. … 15 more files in changeset.
Update version to 1.8.0-SNAPSHOT.

  1. … 26 more files in changeset.
[maven-release-plugin] prepare release drill-1.7.0

  1. … 26 more files in changeset.
DRILL-4732: Update JDBC driver to use the new prepared statement APIs on DrillClient

this closes #532

  1. … 7 more files in changeset.
DRILL-4729: Add support for prepared statement implementation on server side

+ Add following APIs for Drill Java client

- DrillRpcFuture<CreatePreparedStatementResp> createPreparedStatement(final String query)

- void executePreparedStatement(final PreparedStatement preparedStatement, UserResultsListener resultsListener)

- List<QueryDataBatch> executePreparedStatement(final PreparedStatement preparedStatement) (for testing purpose)

+ Separated out the interface from UserClientConnection. It makes it easy to have wrappers which need to

tap the messages and data going to the actual client.

+ Implement CREATE_PREPARED_STATEMENT and handle RunQuery with PreparedStatement

+ Test changes to support prepared statement as query type

+ Add tests in TestPreparedStatementProvider

this closes #530

  1. … 37 more files in changeset.