Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7048: Implement JDBC Statement.setMaxRows() with System Option

This introduces support for JDBC's Statement.setMaxRows(int) API, which can help Drill execute a query much faster if it knows that not ALL the records in the resultset will be consumed upfront.

This Commit introduces the core changes to support the feature within Drill's execution engine

Protobuf Changes

1. RunQuery: Added "autolimit_rowcount"

2. QueryProfile: Added "autoLimit"

3. Regenerated Java and C++ client files

REST API support

1. Support for REST server to interpret a submitted query and also for rendering this information for an executed query

2. Updates to the Freemarker templates (for WebUI)

3. Safety check within Javascript (for WebUI)

JDBC API support

1. Introduces backend execution of 'ALTER SESSION' to apply the auto-limiting of resultset size

2. Added Unit Tests for PreparedStatement and Statement objects

3. Added getter setter methods to be skipped in testing for org.apache.drill.jdbc.test.Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight()

Updates based on review comments

Additional Updates

Test Cleanup

1. Revert Drill2489 hack

2. Formatting in *StatementTest

3. Removal f redundant `statement.close()`

4. Manage new Exception thrown when setting invalid maxRow values

Final updates

1. Test changes

2. Trim trailing spaces in auto-limit value (Javascript)

3. Before & After annotations to synchronize changes to system values for MaxRows(auto-limit)

Reorganized tests due to synchronized locking

Removed conflicting JsonCreator in QueryWrapper

Additional test cleanup

closes #1714

  1. … 34 more files in changeset.
DRILL-6946: Implement java.sql.Connection setSchema and getSchema methods in DrillConnectionImpl closes #1596

  1. … 9 more files in changeset.
DRILL-5301: Server metadata API

Add a Server metadata API to the User protocol, to query server support

of various SQL features.

Add support to the client (DrillClient) to query this information.

Add support to the JDBC driver to query this information, if the server supports

the new API, or fallback to the previous behaviour (rely on Avatica defaults) otherwise.

close #764

  1. … 35 more files in changeset.
DRILL-4335: Apache Drill should support network encryption.

NOTE: This pull request provides support for on-wire encryption using SASL framework. The communication channel that are covered are:

1) Between Drill JDBC client and Drillbit.

2) Between Drillbit to Drillbit i.e. control/data channels.

3) It has UI change to view encryption is enabled on which network channel and number of encrypted/unencrypted connections for

user/control/data connections.

close apache/drill#773

  1. … 60 more files in changeset.
DRILL-4280: CORE (Java protocol)

+ Define SaslStatus and SaslMessage messages in protocol

+ Add "authenticationMechanisms" field to all handshakes

+ Add "saslSupport” field to UserToBitHandshake

  1. … 20 more files in changeset.
DRILL-4994: Add back JDBC prepared statement for older servers

When the JDBC client is connected to an older Drill server, it always

attempted to use server-side prepared statement with no fallback.

With this change, client will check server version and will fallback to the

previous client-side prepared statement (which is still limited to only execute

queries and does not provide metadata).

close #613

  1. … 23 more files in changeset.
DRILL-4968: Add column size to ColumnMetadata

Add a column size to ColumnMetadata so that JDBC and ODBC clients share

the implementation and don't have to recompute it client side.

this closes #631

  1. … 5 more files in changeset.
DRILL-4369: Exchange name and version infos during handshake

There's no name and version exchanged between client and server over the User RPC

channel.

On client side, having access to the server name and version is useful to expose it

to the user (through JDBC or ODBC api like DatabaseMetadata#getDatabaseProductVersion()),

or to implement fallback strategy when some recent API are not available (like

metadata API).

On the server side, having access to the client version might be useful for audit

purposes and eventually to implement fallback strategy if it doesn't require a RPC

version change.

this closes #622

  1. … 21 more files in changeset.
DRILL-4925: Add tableType filter to GetTables metadata query

- Adding tableType filter to GetTablesReq query (needed for JDBC and ODBC

drivers).

- Fix table type returned by sys and INFORMATION_SCHEMA tables

- Also fixes some protobuf typos to related classes.

this closes #612

Change-Id: If95246a312f6c6d64a88872936f516308874c2d2

  1. … 18 more files in changeset.
DRILL-4420: C++ API for metadata access and prepared statements

Add support to the C++ client for metadata querying and prepared

statement requests.

Part of the metadata API, add methods to query for server capabilities.

As of now, this interface is not backed up by any RPC exchange so

the information is pretty much static, and match Drill 1.8.0

current capabilities.

  1. … 33 more files in changeset.
DRILL-4729: Add support for prepared statement implementation on server side

+ Add following APIs for Drill Java client

- DrillRpcFuture<CreatePreparedStatementResp> createPreparedStatement(final String query)

- void executePreparedStatement(final PreparedStatement preparedStatement, UserResultsListener resultsListener)

- List<QueryDataBatch> executePreparedStatement(final PreparedStatement preparedStatement) (for testing purpose)

+ Separated out the interface from UserClientConnection. It makes it easy to have wrappers which need to

tap the messages and data going to the actual client.

+ Implement CREATE_PREPARED_STATEMENT and handle RunQuery with PreparedStatement

+ Test changes to support prepared statement as query type

+ Add tests in TestPreparedStatementProvider

this closes #530

  1. … 37 more files in changeset.
DRILL-4728: Add support for new metadata fetch APIs

+ Protobuf messages

- GetCatalogsReq -> GetCatalogsResp

- GetSchemasReq -> GetSchemasResp

- GetTablesReq -> GetTablesResp

- GetColumnsReq -> GetColumnsResp

+ Java Drill client changes

+ Server side changes to handle the metadata API calls

- Provide a self contained `Runnable` implementation for each metadata API

that process the requests and sends the response to client

- In `UserWorker` override the `handle` method that takes the `ResponseSender` and

send the response from the `handle` method instead of returning it.

- Add a method for each new API to UserWorker to submit the metadata work.

- Add a method `addNewWork(Runnable runnable)` to `WorkerBee` to submit a generic

`Runnable` to `ExecutorService`.

- Move out couple of methods from `QueryContext` into a separate interface

`SchemaConfigInfoProvider` to enable instantiating Schema trees without the

full `QueryContext`

+ New protobuf messages increased the `jdbc-all.jar` size. Up the limit to 21MB.

this closes #527

  1. … 31 more files in changeset.
DRILL-4132 Ability to submit simple type of physical plan directly to EndPoint DrillBit for execution. There are multiple changes to achieve this: 1. During physical planning split single plan into multiple based on the number of minor fragments of the Leaf Major fragment. a. Removing exchange operators during planning b. Producing just root fragments (that will be also leaf fragments) 2. Each fragment can be executed against Drillbit it is assigned to, so to keep locality Design document can be found in the JIRA: DRILL-4132

  1. … 27 more files in changeset.
DRILL-2971, DRILL-2886, DRILL-2778, DRILL-2545: Improve RPC connection detection failure. Add RPC timeout.

  1. … 31 more files in changeset.
DRILL-2697: Pauses sites wait indefinitely for a resume signal DrillClient sends a resume signal to UserServer. UserServer triggers a resume call in the correct Foreman. Foreman resumes all pauses related to the query through the Control layer.

+ Better error messages and more tests in TestDrillbitResilience and TestPauseInjection

+ Added execution controls to operator context

+ Removed ControlMessageHandler interface, renamed ControlHandlerImpl to ControlMessageHandler

+ Added CountDownLatchInjection, useful in cases like ParititionedSender that spawns multiple threads

  1. … 33 more files in changeset.
DRILL-2673: Update UserServer <==> UserClient RPC to better handle handshake response

  1. … 11 more files in changeset.
DRILL-2498: Separate QueryResult into two messages QueryResult and QueryData

  1. … 71 more files in changeset.
DRILL-836: Drill needs to return complex types (e.g., map and array) as a JSON string

  1. … 17 more files in changeset.
Use PStore for profiles, partial profiles on running queries

Full profiles through rpc layer

  1. … 29 more files in changeset.
status changes

  1. … 76 more files in changeset.
DRILL-399: Support USE SCHEMA. Also fixes bugs found in using default schema in queries.

  1. … 17 more files in changeset.
Move to Optiq 0.6 Also includes: -improve exception catching -move schema path parsing to Antlr -close zookeeper connection on if client created -enhance BaseTestQuery and have other query tests utilize it -Various test fixes for better memory release. still needs client allocator to be closed. -refactor DrillSqlWorker and create multiple SqlHandlers -Add PojoRecordReader and DirectPlan capabilities -Update Antlr to use same quoting rules as SQL: single quote for quoted strings, back ticks for identifiers -Move back to old Sorts until bugs are fixed -Refector SelectionVector management within Prels -Add support for NO_EXCHANGES option -Extract SchemaFactories to use Optiq's new Schema handling capabilities -Add basic handling of cancel in UserServer -Remove output requirement from Project -Add start of usercredentials to User communication

  1. … 181 more files in changeset.
Updates for latest Optiq version

  1. … 12 more files in changeset.
DRILL-257: Move SQL parsing to server side. Switch to Avatica based JDBC driver. Update QuerySubmitter to support SQL queries. Update SqlAccesors to support getObject() Remove ref, clean up SQL packages some. Various performance fixes. Updating result set so first set of results must be returned before control is return to client to allow metadata to populate for aggressive tools like sqlline Move timeout functionality to TestTools. Update Expression materializer so that it will return a nullable int if a field is not found. Update Project record batch to support simple wildcard queries. Updates to move JSON record reader test to expecting VarCharVector.getObject to return a String rather than a byte[].

  1. … 310 more files in changeset.
DRILL-334: Subdivide Drillbit control and data messages. Add support for socket backpressure. Add TopLevel and Child memory allocator with debug mode to capture memory leaks. Various memory leak fixes to get build to complete.

Also includes fixes from reviews by Tim.

  1. … 212 more files in changeset.
DRILL-233: Move protobuf IDLs to a separate module and check in the compile Java sources

  1. … 27 more files in changeset.