Clone Tools
  • last updated 23 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7098: File Metadata Metastore Plugin closes #1754

    • -11
    • +6
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
  1. … 58 more files in changeset.
DRILL-7048: Implement JDBC Statement.setMaxRows() with System Option

This introduces support for JDBC's Statement.setMaxRows(int) API, which can help Drill execute a query much faster if it knows that not ALL the records in the resultset will be consumed upfront.

This Commit introduces the core changes to support the feature within Drill's execution engine

Protobuf Changes

1. RunQuery: Added "autolimit_rowcount"

2. QueryProfile: Added "autoLimit"

3. Regenerated Java and C++ client files

REST API support

1. Support for REST server to interpret a submitted query and also for rendering this information for an executed query

2. Updates to the Freemarker templates (for WebUI)

3. Safety check within Javascript (for WebUI)

JDBC API support

1. Introduces backend execution of 'ALTER SESSION' to apply the auto-limiting of resultset size

2. Added Unit Tests for PreparedStatement and Statement objects

3. Added getter setter methods to be skipped in testing for org.apache.drill.jdbc.test.Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight()

Updates based on review comments

Additional Updates

Test Cleanup

1. Revert Drill2489 hack

2. Formatting in *StatementTest

3. Removal f redundant `statement.close()`

4. Manage new Exception thrown when setting invalid maxRow values

Final updates

1. Test changes

2. Trim trailing spaces in auto-limit value (Javascript)

3. Before & After annotations to synchronize changes to system values for MaxRows(auto-limit)

Reorganized tests due to synchronized locking

Removed conflicting JsonCreator in QueryWrapper

Additional test cleanup

closes #1714

    • -1
    • +0
    ./apache/drill/jdbc/DrillPreparedStatement.java
    • -0
    • +3
    ./apache/drill/jdbc/impl/DrillCursor.java
  1. … 31 more files in changeset.
DRILL-6946: Implement java.sql.Connection setSchema and getSchema methods in DrillConnectionImpl closes #1596

  1. … 9 more files in changeset.
DRILL-6962: Function coalesce returns an Error when none of the columns in coalesce exist in a parquet file

- Updated UntypedNullVector to hold value count when vector is allocated and transfered to another one;

- Updated RecordBatchLoader and DrillCursor to handle case when only UntypedNull values are present in RecordBatch (special case when data buffer is null but actual values are present);

- Added functions to cast UntypedNull value to other types for use in UDFs;

- Moved UntypedReader, UntypedHolderReaderImpl and UntypedReaderImpl from org.apache.drill.exec.vector.complex.impl to org.apache.drill.exec.vector package.

closes #1614

    • -5
    • +4
    ./apache/drill/jdbc/impl/DrillCursor.java
  1. … 16 more files in changeset.
DRILL-6866: Upgrade to SqlLine 1.6.0

1. Changed SqlLine version to 1.6.0.

2. Overridden new getVersion method in DrillSqlLineApplication.

3. Set maxColumnWidth to 80 to avoid issue described in DRILL-6769.

4. Changed colorScheme to obsidian.

5. Output null value for varchar / char / boolean types as null instead of empty string.

6. Changed access modifier from package default to public for JDBC classes that implement external interfaces to avoid issues when calling methods from these classes using reflection.

closes #1556

    • -1
    • +1
    ./apache/drill/jdbc/DrillConnectionConfig.java
    • -1
    • +1
    ./apache/drill/jdbc/impl/DrillCursor.java
    • -2
    • +1
    ./apache/drill/jdbc/impl/DrillFactory.java
    • -4
    • +2
    ./apache/drill/jdbc/impl/DrillHandler.java
    • -3
    • +1
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
    • -1
    • +1
    ./apache/drill/jdbc/impl/WrappedAccessor.java
  1. … 4 more files in changeset.
DRILL-6834: Introduce option to disable result set for DDL queries for JDBC connection

- Added session-scoped option `drill.exec.fetch_resultset_for_ddl` to control whether update count or result set should be returned for JDBC connection session. By default the option is set to `true` which ensures that result set is returned;

- Updated Drill JDBC: `DrillCursor` and `DrillStatement` to achieve desired behaviour.

closes #1549

    • -0
    • +5
    ./apache/drill/jdbc/DrillStatement.java
    • -3
    • +11
    ./apache/drill/jdbc/impl/DrillCursor.java
  1. … 24 more files in changeset.
DRILL-6850: Force setting DRILL_LOGICAL Convention for DrillRelFactories and DrillFilterRel

- Fix workspace case insensitivity for JDBC storage plugin

  1. … 13 more files in changeset.
DRILL-6862: Update Calcite to 1.18.0

1. Moved Calcite dependency from profile hadoop-default to general dependency managment

2. Updated Calcite version to 1.18.0-drill-r0 and Avatica version to 1.13.0

3. Hook.REL_BUILDER_SIMPLIFY moved to static block, cause now it can't be removed (fixes DRILL-6830)

4. Removed WrappedAccessor, since it was workaround fixed in CALCITE-1408

5. Fixed setting of multiple options in TestBuilder

6. Timstampadd type inference aligned with CALCITE-2699

7. Dependency update caused 417 kB increase of jdb-all jar size, so the maxsize limit was

increased from 39.5 to 40 MB

8. Added test into TestDrillParquetReader to ensure that DRILL-6856 was

fixed by Calcite update

close apache/drill#1631

    • -447
    • +0
    ./apache/drill/jdbc/impl/WrappedAccessor.java
  1. … 10 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

    • -2
    • +2
    ./apache/drill/jdbc/impl/DrillCursor.java
    • -3
    • +3
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
  1. … 979 more files in changeset.
DRILL-6656: Disallow extra semicolons and multiple statements on the same line.

closes #1415

    • -2
    • +1
    ./apache/drill/jdbc/proxy/ProxiesManager.java
  1. … 143 more files in changeset.
DRILL-6647: Update Calcite version to 1.17.0

closes #1425

    • -299
    • +101
    ./apache/drill/jdbc/impl/DrillConnectionImpl.java
    • -1055
    • +439
    ./apache/drill/jdbc/impl/DrillResultSetImpl.java
    • -281
    • +46
    ./apache/drill/jdbc/impl/DrillStatementImpl.java
  1. … 10 more files in changeset.
DRILL-6386: Remove unused imports and star imports.

    • -1
    • +0
    ./apache/drill/jdbc/DrillDatabaseMetaData.java
    • -2
    • +0
    ./apache/drill/jdbc/DrillPreparedStatement.java
    • -7
    • +0
    ./apache/drill/jdbc/JdbcApiSqlException.java
    • -1
    • +0
    ./apache/drill/jdbc/impl/DrillAccessorList.java
    • -2
    • +0
    ./apache/drill/jdbc/impl/DrillCursor.java
  1. … 218 more files in changeset.
DRILL-6422: Update guava to 23.0 and shade it

- Fix compilation errors for new version of Guava.

- Remove usage of deprecated API

- Shade guava and add dependencies to the shaded version

- Ban unshaded package

- Introduce drill-shaded module and move guava-shaded under it

- Add methods to convert shaded guava lists to the unshaded ones

- Add instruction for publishing artifacts to the Apache repository

  1. … 81 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

    • -1
    • +1
    ./apache/drill/jdbc/DrillConnection.java
    • -1
    • +1
    ./apache/drill/jdbc/DrillConnectionConfig.java
    • -6
    • +7
    ./apache/drill/jdbc/DrillDatabaseMetaData.java
    • -7
    • +8
    ./apache/drill/jdbc/DrillPreparedStatement.java
    • -1
    • +1
    ./apache/drill/jdbc/DrillResultSet.java
    • -7
    • +8
    ./apache/drill/jdbc/DrillStatement.java
    • -1
    • +1
    ./apache/drill/jdbc/SQLConversionException.java
    • -1
    • +1
    ./apache/drill/jdbc/SchemaChangeListener.java
    • -1
    • +1
    ./apache/drill/jdbc/SqlTimeoutException.java
    • -1
    • +1
    ./apache/drill/jdbc/impl/BasicList.java
    • -1
    • +1
    ./apache/drill/jdbc/impl/DrillAccessorList.java
    • -1
    • +1
    ./apache/drill/jdbc/impl/DrillCursor.java
  1. … 2052 more files in changeset.
DRILL-6094: Decimal data type enhancements

Add ExprVisitors for VARDECIMAL

Modify writers/readers to support VARDECIMAL

- Added usage of VarDecimal for parquet, hive, maprdb, jdbc;

- Added options to store decimals as int32 and int64 or fixed_len_byte_array or binary;

Add UDFs for VARDECIMAL data type

- modify type inference rules

- remove UDFs for obsolete DECIMAL types

Enable DECIMAL data type by default

Add unit tests for DECIMAL data type

Fix mapping for NLJ when literal with non-primitive type is used in join conditions

Refresh protobuf C++ source files

Changes in C++ files

Add support for decimal logical type in Avro.

Add support for date, time and timestamp logical types.

Update Avro version to 1.8.2.

    • -0
    • +2
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
  1. … 200 more files in changeset.
DRILL-6202: Deprecate usage of IndexOutOfBoundsException to re-alloc vectors

closes #1144

  1. … 9 more files in changeset.
DRILL-6294: Changes to support Calcite 1.16.0 , and remove deprecated API usage

closes #1198

    • -0
    • +2
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
  1. … 38 more files in changeset.
DRILL-6090: While connecting to drill-bits using JDBC Driver through Zookeeper, a lot of "Curator-Framework-0" threads are created if connection to drill-bit is not successful(no drill-bits are up/reachable)

I am using Drill JDBC driver 1.12.0 to connect to MapR-DB. I am finding the available drill-bits using Zookeepers. When drill-bits are not up or not reachable, the connection is failed with exception: "Failure in connecting to Drill: oadd.org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client", which is expected, but number of threads created by ZKClusterCoordinator just keeps on increasing.

Steps to reproduce the issue

Setup a connection with a drill-bit using Apache Drill JDBC driver 1.12.0 through Zookeeper hosts(port 5181)

Now stop the drill-bit services or block the drill-bit IPs using iptable rules

Truncate catalina logs

Try to connect to the drill-bit/hit a code path that requires connection to drill-bits.

Take thread dump using kill -QUIT <java process id>

grep -c "Curator-Framework-0" catalina.out

Observe that the curator framework thread just keep on accumulating

RCA:

ZKClusterCoordinator creates curator threads in the constructor

ZKClusterCoordinator is instantiated by DrillClient.connect

DrillClient.connect is called in DrillConnectionImpl constructor

Fix:

Call DrillConnectionImpl .cleanup() from all the catch blocks in the DrillConnectionImpl constructor.

close apache/drill#1094

DRILL-3640: Support JDBC Statement.setQueryTimeout(int)

Allow for queries to be cancelled if they don't complete within the stipulated time.

This is done by having Drill[Prepared]StatementImpl create a Stopwatch timer to track elapsed time.

* DrillCursor uses this to detect timeouts.

* DrillResultSetImpl uses this to detech timeout from the client side (e.g. a slow client, when all batches have been processed by DrillCursor)

* Tests added to test these and other query timeout scenarios.

* Dependent on DRILL-5973 for enabling server-triggered timeout tests

NOTE: PreparedStatementTest.testServerTriggeredQueryTimeout is disabled

closes #1024

    • -15
    • +9
    ./apache/drill/jdbc/DrillStatement.java
    • -5
    • +9
    ./apache/drill/jdbc/SqlTimeoutException.java
    • -8
    • +53
    ./apache/drill/jdbc/impl/DrillCursor.java
  1. … 3 more files in changeset.
DRILL-5783, DRILL-5841, DRILL-5894: Rationalize test temp directories

This change includes:

DRILL-5783:

- A unit test is created for the priority queue in the TopN operator.

- The code generation classes passed around a completely unused function registry reference in some places so it is removed.

- The priority queue had unused parameters for some of its methods so it is removed.

DRILL-5841:

- Created standardized temp directory classes DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. And updated all unit tests to use them.

DRILL-5894:

- Removed the dfs_test storage plugin for tests and replaced it with the already existing dfs storage plugin.

Misc:

- General code cleanup.

- Removed unnecessary use of String.format in the tests.

This closes #984

  1. … 365 more files in changeset.
DRILL-3993: Fix unit test failures connected with support Calcite 1.13

- Use root schema as default for describe table statement.

Fix TestOpenTSDBPlugin.testDescribe() and TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema() unit tests.

- Modify expected results for tests:

TestPreparedStatementProvider.invalidQueryValidationError();

TestProjectPushDown.testTPCH1();

TestProjectPushDown.testTPCH3();

TestStorageBasedHiveAuthorization.selectUser1_db_u0_only();

TestStorageBasedHiveAuthorization.selectUser0_db_u1g1_only()

- Fix TestCTAS.whenTableQueryColumnHasStarAndTableFiledListIsSpecified(), TestViewSupport.createViewWhenViewQueryColumnHasStarAndViewFiledListIsSpecified(), TestInbuiltHiveUDFs.testIf(), testDisableUtf8SupportInQueryString unit tests.

- Fix UnsupportedOperationException and NPE for jdbc tests.

- Fix AssertionError: Conversion to relational algebra failed to preserve datatypes

*DrillCompoundIdentifier:

According to the changes, made in [CALCITE-546], star Identifier is replaced by empty string during parsing the query. Since Drill uses its own DrillCompoundIdentifier, it should also replace star by empty string before creating SqlIdentifier instance to avoid further errors connected with star column. see SqlIdentifier.isStar() method.

*SqlConverter:

In [CALCITE-1417] added simplification of expressions which should be projected every time when a new project rel node is created using RelBuilder. It causes assertion errors connected with types nullability. This hook was set to false to avoid project expressions simplification. See usage of this hook and RelBuilder.project() method.

In Drill the type nullability of the function depends on only the nullability of its arguments. In some cases, a function may return null value even if it had non-nullable arguments. When Calice simplifies expressions, it checks that the type of the result is the same as the type of the expression. Otherwise, makeCast() method is called. But when a function returns null literal, this cast does nothing, even when the function has a non-nullable type. So to avoid this issue, method makeCast() was overridden.

*DrillAvgVarianceConvertlet:

Problem with sum0 and specific changes in old Calcite (it is CALCITE-777). (see HistogramShuttle.visitCall method) Changes were made to avoid changes in Calcite.

*SqlConverter, DescribeTableHandler, ShowTablesHandler:

New Calcite tries to combine both default and specified workspaces during the query validation. In some cases, for example, when describe table statement is used, Calcite tries to find INFORMATION_SCHEMA in the schema used as default. When it does not find the schema, it tries to find a table with such name. For some storage plugins, such as opentsdb and hbase, when a table was not found, the error is thrown, and the query fails. To avoid this issue, default schema was changed to root schema for validation stage for describe table and show tables queries.

    • -5
    • +9
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
  1. … 16 more files in changeset.
DRILL-3993: Changes to support Calcite 1.13

- fixed all compiling errors (main changes were: Maven changes, chenges RelNode -> RelRoot, implementing some new methods from updated interfaces, chenges some literals, logger changes);

- fixed unexpected column errors, validation errors and assertion errors after Calcite update;

- fixed describe table/schema statement according to updated logic;

- added fixes with time-intervals;

- changed precision of BINARY to 65536 (was 1048576) according to updated logic (Calcite overrides bigger precision to own maxPrecision);

- ignored some incorrect tests with DRILL-3244;

- changed "Table not found" message to "Object not found within" according to new Calcite changes.

    • -10
    • +63
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
  1. … 66 more files in changeset.
DRILL-4264: Allow field names to include dots

  1. … 98 more files in changeset.
DRILL-5431: SSL Support (Java) - Java client server SSL implementation

Also enable OpenSSL support

Also fix exclusions and java-exec pom file to eliminate netty-tcnative as a transitive dependency on all projects

  1. … 30 more files in changeset.
DRILL-5568: Include hadoop-common jars inside drill-jdbc-all.jar 1) Introduce a new file inside exec/jdbc-all resource package which contains the property key/value pair for prefix 2) At build time based upon the profile choosen this property value is set inside the file 3) It is later consumed by SecurityConfiguration class to rename classpath for classes inside hadoop package.

DRILL-5568: Code review changes

close apache/drill#849

    • -3
    • +3
    ./apache/drill/jdbc/impl/DrillFactory.java
  1. … 5 more files in changeset.
DRILL-5413: DrillConnectionImpl.isReadOnly() throws NullPointerException

change is in CALCITE-843.

update drill's calcite version to 1.4.0-drill-r21

close #806

  1. … 2 more files in changeset.
DRILL-5326: Unit tests failures related to the SERVER_METADTA

- adding of the sql type name for the "GENERIC_OBJECT";

- changing "NullCollation" in the "ServerMetaProvider" to the correct default value;

- changing RpcType to GET_SERVER_META in the appropriate ServerMethod

close #775

  1. … 3 more files in changeset.
DRILL-5301: Server metadata API

Add a Server metadata API to the User protocol, to query server support

of various SQL features.

Add support to the client (DrillClient) to query this information.

Add support to the JDBC driver to query this information, if the server supports

the new API, or fallback to the previous behaviour (rely on Avatica defaults) otherwise.

close #764

    • -3
    • +2
    ./apache/drill/jdbc/DrillConnectionConfig.java
    • -4
    • +4
    ./apache/drill/jdbc/impl/DrillMetaImpl.java
  1. … 32 more files in changeset.
DRILL-4280: CORE (user to bit authentication, Java)

+ Add logic for authentication in UserClient and UserServer with

backward compatibility in both directions

+ Add abstract extension to ServerConnection and ClientConnection

+ Add concrete extensions to abstract connections:

BitToUserConnection and UserToBitConnection

+ Add ConnectionConfig interface with abstract and concrete

implementations to encapsulate configuration for server-side

connections

+ Encapsulate all requests handled by UserServer in

UserServerRequestHandler

+ Clear UserSession when connection is closed either by user or

bit

+ Add DrillProperties to encapsulate all connection properties

used during connection time

  1. … 15 more files in changeset.
DRILL-5241: JDBC proxy driver: Do not put null value in map

This closes #724

    • -1
    • +1
    ./apache/drill/jdbc/proxy/ProxiesManager.java