Clone Tools
  • last updated 12 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7294: Regenerate protobufs

  1. … 94 more files in changeset.
DRILL-7048: Implement JDBC Statement.setMaxRows() with System Option

This introduces support for JDBC's Statement.setMaxRows(int) API, which can help Drill execute a query much faster if it knows that not ALL the records in the resultset will be consumed upfront.

This Commit introduces the core changes to support the feature within Drill's execution engine

Protobuf Changes

1. RunQuery: Added "autolimit_rowcount"

2. QueryProfile: Added "autoLimit"

3. Regenerated Java and C++ client files

REST API support

1. Support for REST server to interpret a submitted query and also for rendering this information for an executed query

2. Updates to the Freemarker templates (for WebUI)

3. Safety check within Javascript (for WebUI)

JDBC API support

1. Introduces backend execution of 'ALTER SESSION' to apply the auto-limiting of resultset size

2. Added Unit Tests for PreparedStatement and Statement objects

3. Added getter setter methods to be skipped in testing for org.apache.drill.jdbc.test.Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight()

Updates based on review comments

Additional Updates

Test Cleanup

1. Revert Drill2489 hack

2. Formatting in *StatementTest

3. Removal f redundant `statement.close()`

4. Manage new Exception thrown when setting invalid maxRow values

Final updates

1. Test changes

2. Trim trailing spaces in auto-limit value (Javascript)

3. Before & After annotations to synchronize changes to system values for MaxRows(auto-limit)

Reorganized tests due to synchronized locking

Removed conflicting JsonCreator in QueryWrapper

Additional test cleanup

closes #1714

  1. … 33 more files in changeset.
DRILL-7014: Format plugin for LTSV files closes #1627

  1. … 16 more files in changeset.
DRILL-6582: SYSLOG (RFC-5424) Format Plugin closes #1530

  1. … 20 more files in changeset.
DRILL-6946: Implement java.sql.Connection setSchema and getSchema methods in DrillConnectionImpl closes #1596

  1. … 9 more files in changeset.
DRILL-6834: Introduce option to disable result set for DDL queries for JDBC connection

- Added session-scoped option `drill.exec.fetch_resultset_for_ddl` to control whether update count or result set should be returned for JDBC connection session. By default the option is set to `true` which ensures that result set is returned;

- Updated Drill JDBC: `DrillCursor` and `DrillStatement` to achieve desired behaviour.

closes #1549

  1. … 25 more files in changeset.
DRILL-1328: Support table statistics

  1. … 52 more files in changeset.
DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf reference count bugs & tune the execution flow & support left deep tree

closes #1504

  1. … 28 more files in changeset.
DRILL-6731: Move the BFs aggregating work from the Foreman to the RuntimeFilter

  1. … 17 more files in changeset.
DRILL-6639: Exception happens while displaying operator profiles for some queries

  1. … 17 more files in changeset.
DRILL-6639: Exception happens while displaying operator profiles for some queries

closes #1404

  1. … 17 more files in changeset.
DRILL-6635: PartitionLimit for Lateral/Unnest Protobuf changes to add new operator PartitionLimit

  1. … 4 more files in changeset.
DRILL-6627: Adding REGEX_SUB_SCAN operator to protobuf file

- The operator is added to Java based UserBitShared.proto and C++ based UserBitShared.pb.h

- Java and C++ protobuf files are regenerated

  1. … 4 more files in changeset.
DRILL-6179: Added pcapng-format support

  1. … 22 more files in changeset.
DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.

1. Added protobuf files (incl generated C++ and Java)

2. Updated JdbcSubScan's getOperatorType API

closes #1297

  1. … 5 more files in changeset.
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.

This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes

Set the query ID string and display in WebUI

closes #1301

  1. … 8 more files in changeset.
DRILL-6385: Support JPPD feature

    • -0
    • +293
    ./RuntimeFilterBDef.java
  1. … 62 more files in changeset.
DRILL-5305: Query Profile must display Query ID

Introduced change to the Protobuf to inject the text-equivalent of the QueryID into the profile. This way, the profile's file name can be changed, but restored back based on this new field.

The Profile UI also shows the Query ID, though this is not inferred from this new field, for sake of backward compatibility with older profiles.

closes #1265

  1. … 6 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2052 more files in changeset.
DRILL-6289: Cluster view should show more relevant information

Protobuf change to carry HTTP port info

Allow CORS for access to remote Drillbit metrics

Cross-origin resource sharing (CORS) is required to ensure that the WebServer is able serve REST calls for status pages.

Materialize relevant metrics

1. Heap memory (incl usage)

2. Heap memory (incl usage)

3. Average System Load (last 1 min)

4. Option to view from other nodes (pop out)

5. Added Glyphicons

Update System Table and related tests

1. Updated System Table to show HTTP port

2. Updated unit tests

Skip updating remote bit info when HTTPS (SSL) or Authentication is enabled.

Default CpuGaugeSet is public; Added Gauges

* CPU Utiization by Drill

* Uptime

Show ALL Buttons, but do HTTPS Check

Reduce power button to icon

Allowing CORS for /status/metrics only

Accounting for situations when JVM does not report Process CPU Load

i.e. returned value is negative.

See https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad()

Addressed shutdown security conditions

Added C++ Client Protobuf

Added steps for Protobuf generation to protocol/readme.txt

This closes #1203

  1. … 16 more files in changeset.
DRILL-6284: Add operator metrics for batch sizing for flatten

  1. … 6 more files in changeset.
DRILL-6331: Revisit Hive Drill native parquet implementation to be exposed to Drill optimizations (filter / limit push down, count to direct scan)

1. Factored out common logic for Drill parquet reader and Hive Drill native parquet readers: AbstractParquetGroupScan, AbstractParquetRowGroupScan, AbstractParquetScanBatchCreator.

2. Rules that worked previously only with ParquetGroupScan, now can be applied for any class that extends AbstractParquetGroupScan: DrillFilterItemStarReWriterRule, ParquetPruneScanRule, PruneScanRule.

3. Hive populated partition values based on information returned from Hive metastore. Drill populates partition values based on path difference between selection root and actual file path.

Before ColumnExplorer populated partition values based on Drill approach. Since now ColumnExplorer populates values for parquet files from Hive tables,

`populateImplicitColumns` method logic was changed to populated partition columns only based on given partition values.

4. Refactored ParquetPartitionDescriptor to be responsible for populating partition values rather than storing this logic in parquet group scan class.

5. Metadata class was moved to separate metadata package (org.apache.drill.exec.store.parquet.metadata). Factored out several inner classed to improve code readability.

6. Collected all Drill native parquet reader unit tests into one class TestHiveDrillNativeParquetReader, also added new tests to cover new functionality.

7. Reduced excessive logging when parquet files metadata is read

closes #1214

  1. … 64 more files in changeset.
DRILL-6381: (Part 1) Secondary Index framework

  1. Secondary Index planning interfaces and abstract classes like DBGroupScan, DbSubScan, IndexDecriptor etc.

  2. Statistics and Cost model interfaces/classes: PluginCost, Statistics, StatisticsPayload, AbstractIndexStatistics

  3. ScanBatch and RecordReader to support repeatable scan

  4. Secondary Index execution related interfaces: RangePartitionSender, RowKeyJoin, PartitionFunction

5. MD-3979: Query using cast index plan fails with NPE

Co-authored-by: Aman Sinha <asinha@maprtech.com>

Co-authored-by: chunhui-shi <cshi@maprtech.com>

Co-authored-by: Gautam Parai <gparai@maprtech.com>

Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>

Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>

Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java

exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java

protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java

protocol/src/main/java/org/apache/drill/exec/proto/beans/CoreOperatorType.java

protocol/src/main/protobuf/UserBitShared.proto

  1. … 44 more files in changeset.
DRILL-6322: Lateral Join: Common changes - Add new iterOutcome, Operatortypes, MockRecordBatch for testing

Added new Iterator State EMIT, added operatos LATERA_JOIN & UNNEST in CoreOperatorType and added LateralContract interface

Implementation of MockRecordBatch to test operator behavior for different IterOutcomes. a) Creates new output container for schema change cases. b) Doesn't create new container for each next() call without schema change, since the operator in test expects the ValueVector object in it's incoming batch to be same unless a OK_NEW_SCHEMA case is hit. Since setup() method of operator in test will store the reference to value vector received in first batch

This closes #1211

  1. … 6 more files in changeset.
DRILL-6130: Fix NPE during physical plan submission for various storage plugins

1. Fixed ser / de issues for Hive, Kafka, Hbase plugins.

2. Added physical plan submission unit test for all storage plugins in contrib module.

3. Refactoring.

closes #1108

  1. … 26 more files in changeset.
DRILL-6049: Misc. hygiene and code cleanup changes

close apache/drill#1085

  1. … 123 more files in changeset.
DRILL-4779: Kafka storage plugin (Kamesh Bhallamudi & Anil Kumar Batchu)

closes #1052

  1. … 36 more files in changeset.
DRILL-5963: Query state process improvements

1. Added two new query states: PREPARING (when foreman is initialized) and PLANNING (includes logical and / or physical planning).

2. Ability to cancel query during planning and enqueued states was added.

3. Logic for submitting fragments was moved from Foreman to new class FragmentsRunner.

4. Logic for moving query from to new state and incrementing / decrementing query counters was moved into QueryStateProcessor class.

5. Major type in DrillFuncHolderExpr was cached for better performance.

closes #1051

  1. … 12 more files in changeset.
DRILL-5716: Queue-driven memory allocation

* Creates new core resource management and query queue abstractions.

* Adds queue information to the Protobuf layer.

* Foreman and Planner changes

- Abstracts memory management out to the new resource management layer.

This means deferring generating the physical plan JSON to later in the

process after memory planning.

* Web UI changes

* Adds queue information to the main page and the profile page to each

query.

* Also sorts the list of options displayed in the Web UI.

- Added memory reserve

A new config parameter, exec.queue.memory_reserve_ratio, sets aside a

slice of total memory for operators that do not participate in the

memory assignment process. The default is 20% testing will tell us if

that value should be larger or smaller.

* Additional minor fixes

- Code cleanup.

- Added mechanism to abandon lease release during shutdown.

- Log queue configuration only when the config changes, rather than on

every query.

- Apply Boaz’ option to enforce a minimum memory allocation per

operator.

- Additional logging to help testers see what is happening.

closes #928

  1. … 56 more files in changeset.
DRILL-4286: Graceful shutdown of drillbit

closes #921

  1. … 26 more files in changeset.