Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7730: Improve web query efficiency

Implements a direct transfer of batches from Screen to web client.

Cleans up web client query processing to avoid duplicate schema

info.

Much related code cleanup.

  1. … 38 more files in changeset.
DRILL-7011: Support schema in scan framework

* Adds schema support to the row set-based scan framework and to the "V3" text reader based on that framework.

* Adding the schema made clear that passing options as a long list of constructor arguments was not sustainable. Refactored code to use a builder pattern instead.

* Added support for default values in the "null column loader", which required adding a "setValue" method to the column accessors.

* Added unit tests for all new or changed functionality. See TestCsvWithSchema for the overall test of the entire integrated mechanism.

* Added tests for explicit projection with schema

* Better handling of date/time in column accessors

* Converted recent column metadata work from Java 8 date/time to Joda.

* Added more CSV-with-schema unit tests

* Removed the ID fields from "resolved columns", used "instanceof" instead.

* Added wildcard projection with an output schema. Handles both "lenient" and "strict" schemas.

* Tagged projection columns with their output schema, when available.

* Scan projection added modes for wildcard with an output schema. The reader projection added support for merging reader and output schemas.

* Includes refactoring of scan operator tests (the test file grew too large.)

* Renamed some classes to avoid confusing reader schemas with output schemas.

* Added unit tests for the new functionality.

* Added "lenient" wildcard with schema test for CSV

* Added more type conversions: string-to-bit, many-to-string

* Fixed bug in column writer for VarDecimal

* Added missing unit tests, and fixed bugs, in Bit column reader/writer

* Cleaned up a number of unneded "SuppressWarnings"

closes #1711

  1. … 224 more files in changeset.
DRILL-7068: Support memory adjustment framework for resource management with Queues. closes #1677

  1. … 37 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

  1. … 984 more files in changeset.
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.

This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes

Set the query ID string and display in WebUI

closes #1301

  1. … 9 more files in changeset.
DRILL-5305: Query Profile must display Query ID

Introduced change to the Protobuf to inject the text-equivalent of the QueryID into the profile. This way, the profile's file name can be changed, but restored back based on this new field.

The Profile UI also shows the Query ID, though this is not inferred from this new field, for sake of backward compatibility with older profiles.

closes #1265

  1. … 6 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2066 more files in changeset.
DRILL-5902: Queries encounter random failure due to RPC connection timed out

close apache/drill#1113

  1. … 4 more files in changeset.
DRILL-5922: - The QueryContext was never closed when the Foreman finished, so it's child allocator was never closed. Now it is. - The PlanSplitter created a QueryContext temporarily to construct an RPC message but never closed it. Now the temp QueryContext is closed. - The waitForExit method was error prone. Changed it to use the standard condition variable pattern. - Fixed timeouts in graceful shutdown tests

  1. … 6 more files in changeset.
DRILL-5716: Queue-driven memory allocation

* Creates new core resource management and query queue abstractions.

* Adds queue information to the Protobuf layer.

* Foreman and Planner changes

- Abstracts memory management out to the new resource management layer.

This means deferring generating the physical plan JSON to later in the

process after memory planning.

* Web UI changes

* Adds queue information to the main page and the profile page to each

query.

* Also sorts the list of options displayed in the Web UI.

- Added memory reserve

A new config parameter, exec.queue.memory_reserve_ratio, sets aside a

slice of total memory for operators that do not participate in the

memory assignment process. The default is 20% testing will tell us if

that value should be larger or smaller.

* Additional minor fixes

- Code cleanup.

- Added mechanism to abandon lease release during shutdown.

- Log queue configuration only when the config changes, rather than on

every query.

- Apply Boaz’ option to enforce a minimum memory allocation per

operator.

- Additional logging to help testers see what is happening.

closes #928

  1. … 57 more files in changeset.
DRILL-5457: Spill implementation for Hash Aggregate

closes #822

  1. … 35 more files in changeset.
DRILL-5485: Remove WebServer dependency on DrillClient

1. Added WebUserConnection/AnonWebUserConnection and their providers for Authenticated and Anonymous web users.

2. Updated to store the UserSession, BufferAllocator and other session states inside the HttpSession of Jetty instead

of storing in DrillUserPrincipal. For each request now a new instance of WebUserConnection will be created. However

for authenticated users the UserSession and other states will be re-used whereas for Anonymous Users it will created

for each request and later re-cycled after query execution.

close #829

  1. … 45 more files in changeset.
DRILL-5301: Server metadata API

Add a Server metadata API to the User protocol, to query server support

of various SQL features.

Add support to the client (DrillClient) to query this information.

Add support to the JDBC driver to query this information, if the server supports

the new API, or fallback to the previous behaviour (rely on Avatica defaults) otherwise.

close #764

  1. … 35 more files in changeset.
DRILL-4729: Add support for prepared statement implementation on server side

+ Add following APIs for Drill Java client

- DrillRpcFuture<CreatePreparedStatementResp> createPreparedStatement(final String query)

- void executePreparedStatement(final PreparedStatement preparedStatement, UserResultsListener resultsListener)

- List<QueryDataBatch> executePreparedStatement(final PreparedStatement preparedStatement) (for testing purpose)

+ Separated out the interface from UserClientConnection. It makes it easy to have wrappers which need to

tap the messages and data going to the actual client.

+ Implement CREATE_PREPARED_STATEMENT and handle RunQuery with PreparedStatement

+ Test changes to support prepared statement as query type

+ Add tests in TestPreparedStatementProvider

this closes #530

  1. … 37 more files in changeset.
DRILL-4728: Add support for new metadata fetch APIs

+ Protobuf messages

- GetCatalogsReq -> GetCatalogsResp

- GetSchemasReq -> GetSchemasResp

- GetTablesReq -> GetTablesResp

- GetColumnsReq -> GetColumnsResp

+ Java Drill client changes

+ Server side changes to handle the metadata API calls

- Provide a self contained `Runnable` implementation for each metadata API

that process the requests and sends the response to client

- In `UserWorker` override the `handle` method that takes the `ResponseSender` and

send the response from the `handle` method instead of returning it.

- Add a method for each new API to UserWorker to submit the metadata work.

- Add a method `addNewWork(Runnable runnable)` to `WorkerBee` to submit a generic

`Runnable` to `ExecutorService`.

- Move out couple of methods from `QueryContext` into a separate interface

`SchemaConfigInfoProvider` to enable instantiating Schema trees without the

full `QueryContext`

+ New protobuf messages increased the `jdbc-all.jar` size. Up the limit to 21MB.

this closes #527

  1. … 31 more files in changeset.
DRILL-4132 Ability to submit simple type of physical plan directly to EndPoint DrillBit for execution. There are multiple changes to achieve this: 1. During physical planning split single plan into multiple based on the number of minor fragments of the Leaf Major fragment. a. Removing exchange operators during planning b. Producing just root fragments (that will be also leaf fragments) 2. Each fragment can be executed against Drillbit it is assigned to, so to keep locality Design document can be found in the JIRA: DRILL-4132

    • -0
    • +133
    ./PlanSplitter.java
  1. … 26 more files in changeset.
DRILL-2697: Pauses sites wait indefinitely for a resume signal DrillClient sends a resume signal to UserServer. UserServer triggers a resume call in the correct Foreman. Foreman resumes all pauses related to the query through the Control layer.

+ Better error messages and more tests in TestDrillbitResilience and TestPauseInjection

+ Added execution controls to operator context

+ Removed ControlMessageHandler interface, renamed ControlHandlerImpl to ControlMessageHandler

+ Added CountDownLatchInjection, useful in cases like ParititionedSender that spawns multiple threads

  1. … 33 more files in changeset.
DRILL-2383: Support to inject exceptions and pauses in various components of Drill + Controls are fired only if assertions are enabled + Controls can be introduced in any class that has access to FragmentContext/QueryContext + Controls can be fired by altering the DRILLBIT_CONTROL_INJECTIONS session option + Renames: SimulatedExceptions => ExecutionControls, ExceptionInjector => ExecutionControlsInjector + Added injection sites in Foreman, DrillSqlWorker, FragmentExecutor + Unit tests in TestDrillbitResilience, TestExceptionInjection and TestPauseInjection

Other commits included:

+ DRILL-2437: Moved ExecutionControls from DrillbitContext to FragmentContext/QueryContext

+ DRILL-2382: Added address and port to Injection to specify drillbit

+ DRILL-2384: Added QueryState to SingleRowListener and assert that state is COMPLETED while testing

Other edits:

+ Support for short lived session options in SessionOptionManager (using TTL in OptionValidator)

+ Introduced query count in UserSession

+ Added QueryState to queryCompleted() in UserResultsListener to check if COMPLETED/CANCELED

+ Added JSONStringValidator to TypeValidators

+ Log query id as string in DrillClient, WorkEventBus, QueryResultHandler

+ Use try..catch block only around else clause for OptionList in FragmentContext

+ Fixed drillbitContext spelling error in QueryContext

+ Fixed state transition when cancel() before run() in FragmentExecutor

+ Do not call setLocalOption twice in FallbackOptionManager

+ Show explicitly that submitWork() returns queryId in UserServer

+ Updated protocol/readme.txt to include an alternative way to generate sources

  1. … 45 more files in changeset.
DRILL-1517: Update Foreman to improve state management.

  1. … 55 more files in changeset.
DRILL-1684, DRILL-1517, DRILL-1350: Profile and cancellation updates - Remove any storage of persisted profiles. - Store a separate query info object for active queries. - Update cancellation and running profile loading to query foreman server. - Make file store support HDFS APIs - Update PStoreProvider to use configuration to decide if you want PERSISTENT, EPHEMERAL, or BLOB storage rather than separate interfaces. - Update ZkPStore's persistent mode to leverage a cache and respond to changes rather than actively probing values. - Update ZkPStore's cache to be effectively write-through. - Automatically delete deprecated or default value options from PStore.

  1. … 42 more files in changeset.
DRILL-1402: Add check-style rules for trailing space, TABs and blocks without braces

  1. … 441 more files in changeset.
Use PStore for profiles, partial profiles on running queries

Full profiles through rpc layer

  1. … 29 more files in changeset.
Move to Optiq 0.6 Also includes: -improve exception catching -move schema path parsing to Antlr -close zookeeper connection on if client created -enhance BaseTestQuery and have other query tests utilize it -Various test fixes for better memory release. still needs client allocator to be closed. -refactor DrillSqlWorker and create multiple SqlHandlers -Add PojoRecordReader and DirectPlan capabilities -Update Antlr to use same quoting rules as SQL: single quote for quoted strings, back ticks for identifiers -Move back to old Sorts until bugs are fixed -Refector SelectionVector management within Prels -Add support for NO_EXCHANGES option -Extract SchemaFactories to use Optiq's new Schema handling capabilities -Add basic handling of cancel in UserServer -Remove output requirement from Project -Add start of usercredentials to User communication

  1. … 181 more files in changeset.
DRILL-381: Implement SYSTEM and SESSION options.

  1. … 69 more files in changeset.
DRILL-334: Subdivide Drillbit control and data messages. Add support for socket backpressure. Add TopLevel and Child memory allocator with debug mode to capture memory leaks. Various memory leak fixes to get build to complete.

Also includes fixes from reviews by Tim.

  1. … 212 more files in changeset.
DRILL-221 Add license header to all files

  1. … 829 more files in changeset.
DRILL-165: Reorganize directories (moves only)

  1. … 1739 more files in changeset.