UserBitShared.proto

Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-6635: PartitionLimit for Lateral/Unnest Protobuf changes to add new operator PartitionLimit

  1. … 4 more files in changeset.
DRILL-6627: Adding REGEX_SUB_SCAN operator to protobuf file

- The operator is added to Java based UserBitShared.proto and C++ based UserBitShared.pb.h

- Java and C++ protobuf files are regenerated

  1. … 4 more files in changeset.
DRILL-6179: Added pcapng-format support

  1. … 22 more files in changeset.
DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.

1. Added protobuf files (incl generated C++ and Java)

2. Updated JdbcSubScan's getOperatorType API

closes #1297

  1. … 5 more files in changeset.
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.

This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes

Set the query ID string and display in WebUI

closes #1301

  1. … 9 more files in changeset.
DRILL-6385: Support JPPD feature

  1. … 63 more files in changeset.
DRILL-5305: Query Profile must display Query ID

Introduced change to the Protobuf to inject the text-equivalent of the QueryID into the profile. This way, the profile's file name can be changed, but restored back based on this new field.

The Profile UI also shows the Query ID, though this is not inferred from this new field, for sake of backward compatibility with older profiles.

closes #1265

  1. … 6 more files in changeset.
DRILL-6284: Add operator metrics for batch sizing for flatten

  1. … 6 more files in changeset.
DRILL-6331: Revisit Hive Drill native parquet implementation to be exposed to Drill optimizations (filter / limit push down, count to direct scan)

1. Factored out common logic for Drill parquet reader and Hive Drill native parquet readers: AbstractParquetGroupScan, AbstractParquetRowGroupScan, AbstractParquetScanBatchCreator.

2. Rules that worked previously only with ParquetGroupScan, now can be applied for any class that extends AbstractParquetGroupScan: DrillFilterItemStarReWriterRule, ParquetPruneScanRule, PruneScanRule.

3. Hive populated partition values based on information returned from Hive metastore. Drill populates partition values based on path difference between selection root and actual file path.

Before ColumnExplorer populated partition values based on Drill approach. Since now ColumnExplorer populates values for parquet files from Hive tables,

`populateImplicitColumns` method logic was changed to populated partition columns only based on given partition values.

4. Refactored ParquetPartitionDescriptor to be responsible for populating partition values rather than storing this logic in parquet group scan class.

5. Metadata class was moved to separate metadata package (org.apache.drill.exec.store.parquet.metadata). Factored out several inner classed to improve code readability.

6. Collected all Drill native parquet reader unit tests into one class TestHiveDrillNativeParquetReader, also added new tests to cover new functionality.

7. Reduced excessive logging when parquet files metadata is read

closes #1214

  1. … 64 more files in changeset.
DRILL-6381: (Part 1) Secondary Index framework

  1. Secondary Index planning interfaces and abstract classes like DBGroupScan, DbSubScan, IndexDecriptor etc.

  2. Statistics and Cost model interfaces/classes: PluginCost, Statistics, StatisticsPayload, AbstractIndexStatistics

  3. ScanBatch and RecordReader to support repeatable scan

  4. Secondary Index execution related interfaces: RangePartitionSender, RowKeyJoin, PartitionFunction

5. MD-3979: Query using cast index plan fails with NPE

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java

protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java

protocol/src/main/java/org/apache/drill/exec/proto/beans/CoreOperatorType.java

protocol/src/main/protobuf/UserBitShared.proto

  1. … 44 more files in changeset.
DRILL-6322: Lateral Join: Common changes - Add new iterOutcome, Operatortypes, MockRecordBatch for testing

Added new Iterator State EMIT, added operatos LATERA_JOIN & UNNEST in CoreOperatorType and added LateralContract interface

Implementation of MockRecordBatch to test operator behavior for different IterOutcomes. a) Creates new output container for schema change cases. b) Doesn't create new container for each next() call without schema change, since the operator in test expects the ValueVector object in it's incoming batch to be same unless a OK_NEW_SCHEMA case is hit. Since setup() method of operator in test will store the reference to value vector received in first batch

This closes #1211

  1. … 6 more files in changeset.
DRILL-6130: Fix NPE during physical plan submission for various storage plugins

1. Fixed ser / de issues for Hive, Kafka, Hbase plugins.

2. Added physical plan submission unit test for all storage plugins in contrib module.

3. Refactoring.

closes #1108

  1. … 26 more files in changeset.
DRILL-6049: Misc. hygiene and code cleanup changes

close apache/drill#1085

  1. … 123 more files in changeset.
DRILL-4779: Kafka storage plugin (Kamesh Bhallamudi & Anil Kumar Batchu)

closes #1052

  1. … 36 more files in changeset.
DRILL-5963: Query state process improvements

1. Added two new query states: PREPARING (when foreman is initialized) and PLANNING (includes logical and / or physical planning).

2. Ability to cancel query during planning and enqueued states was added.

3. Logic for submitting fragments was moved from Foreman to new class FragmentsRunner.

4. Logic for moving query from to new state and incrementing / decrementing query counters was moved into QueryStateProcessor class.

5. Major type in DrillFuncHolderExpr was cached for better performance.

closes #1051

  1. … 12 more files in changeset.
DRILL-5716: Queue-driven memory allocation

* Creates new core resource management and query queue abstractions.

* Adds queue information to the Protobuf layer.

* Foreman and Planner changes

- Abstracts memory management out to the new resource management layer.

This means deferring generating the physical plan JSON to later in the

process after memory planning.

* Web UI changes

* Adds queue information to the main page and the profile page to each

query.

* Also sorts the list of options displayed in the Web UI.

- Added memory reserve

A new config parameter, exec.queue.memory_reserve_ratio, sets aside a

slice of total memory for operators that do not participate in the

memory assignment process. The default is 20% testing will tell us if

that value should be larger or smaller.

* Additional minor fixes

- Code cleanup.

- Added mechanism to abandon lease release during shutdown.

- Log queue configuration only when the config changes, rather than on

every query.

- Apply Boaz’ option to enforce a minimum memory allocation per

operator.

- Additional logging to help testers see what is happening.

closes #928

  1. … 57 more files in changeset.
DRILL-5512: Standardize error handling in ScanBatch

Standardizes error handling to throw a UserException. Prior code threw

various exceptions, called the fail() method, or returned a variety of

status codes.

closes #838

  1. … 5 more files in changeset.
DRILL-5190: Display planning and queued time for a query's profile page

Modified UserSharedBit protobuf for marking planning and wait-in-queue end times. This will allow for accurately reporting the planning, queued and actual execution times of a query.

Planning Time:

In the absence of the planning time's end, for older profiles, the root fragment's (i.e. SCREEN operator) start time is taken as the estimated end of planning time, and as the estimated start time of the execution phase.

QueueWait Time:

We do not estimate the queue time if the planning end time is not available.

Execution Time:

We calculate the execution time based on the availability of these 2 planning time. The computation is done the following way, and reflects a decreasing level of accuracy

1. Execution time = [end(QueueWait) - endTime(Query)]

2. Execution time = [end(Planning) - endTime(Query)]

3. Execution time = [start(rootFragment) - endTime(Query)] - {Estimated}

closes #738

  1. … 7 more files in changeset.
DRILL-4280: CORE (Java protocol)

+ Define SaslStatus and SaslMessage messages in protocol

+ Add "authenticationMechanisms" field to all handshakes

+ Add "saslSupport” field to UserToBitHandshake

  1. … 20 more files in changeset.
DRILL-4726: Dynamic UDF Support

1) Configuration / parsing / options / protos

2) Zookeeper integration

3) Registration / unregistration / lazy-init

4) Unit tests

This closes #574

  1. … 71 more files in changeset.
DRILL-4792: Include session options used for a query as part of the profile

closes #551

  1. … 8 more files in changeset.
DRILL-4729: Add support for prepared statement implementation on server side

+ Add following APIs for Drill Java client

- DrillRpcFuture<CreatePreparedStatementResp> createPreparedStatement(final String query)

- void executePreparedStatement(final PreparedStatement preparedStatement, UserResultsListener resultsListener)

- List<QueryDataBatch> executePreparedStatement(final PreparedStatement preparedStatement) (for testing purpose)

+ Separated out the interface from UserClientConnection. It makes it easy to have wrappers which need to

tap the messages and data going to the actual client.

+ Implement CREATE_PREPARED_STATEMENT and handle RunQuery with PreparedStatement

+ Test changes to support prepared statement as query type

+ Add tests in TestPreparedStatementProvider

this closes #530

  1. … 37 more files in changeset.
DRILL-4132 Ability to submit simple type of physical plan directly to EndPoint DrillBit for execution. There are multiple changes to achieve this: 1. During physical planning split single plan into multiple based on the number of minor fragments of the Leaf Major fragment. a. Removing exchange operators during planning b. Producing just root fragments (that will be also leaf fragments) 2. Each fragment can be executed against Drillbit it is assigned to, so to keep locality Design document can be found in the JIRA: DRILL-4132

  1. … 27 more files in changeset.
DRILL-4187: introduce a new query state ENQUEUED and rename the state PENDING to STARTING

  1. … 7 more files in changeset.
DRILL-3340: Part 2: Reverting 1a589ab and committing latest patch

Add operator metrics registry for metric definitions

+ Display metrics as a table within an operator profile panel

+ Rename FragmentStats#getOperatorStats to newOperatorStats

  1. … 14 more files in changeset.
DRILL-3340: Added operator names and metric names to query profile before writing it to store

+ Rename: FragmentStats#getOperatorStats => newOperatorStats

+ Documentation

this closes #216

  1. … 12 more files in changeset.
DRILL-2997: Remove references to groupCount from SerializedField

  1. … 17 more files in changeset.
DRILL-2981: Add queries log. Update profile to store normal and verbose exception as well as node and errorid.

  1. … 20 more files in changeset.
DRILL-3010: Convert bad command error messages into UserExceptions in SqlHandlers

  1. … 22 more files in changeset.
DRILL-2675 (PART-2): Implement a subset of User Exceptions to improve how errors are reported to the user

Added missing changes from committed patch

  1. … 35 more files in changeset.