Clone Tools
  • last updated 27 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7368: Fix Iceberg Metastore failure when filter column contains nulls

  1. … 7 more files in changeset.
DRILL-7273: Introduce operators for handling metadata

closes #1886

  1. … 156 more files in changeset.
DRILL-7272: Drill Metastore Read / Write API and Drill Iceberg Metastore implementation

1. Drill Metastore Read / Write API.

2. Drill Iceberg Metastore implementation in iceberg-metastore module.

3. Patches Guava Preconditions class for Apache Iceberg.

4. General refactoring.

5. Unit tests.

6. Documentation.

  1. … 97 more files in changeset.
DRILL-7048: Implement JDBC Statement.setMaxRows() with System Option

This introduces support for JDBC's Statement.setMaxRows(int) API, which can help Drill execute a query much faster if it knows that not ALL the records in the resultset will be consumed upfront.

This Commit introduces the core changes to support the feature within Drill's execution engine

Protobuf Changes

1. RunQuery: Added "autolimit_rowcount"

2. QueryProfile: Added "autoLimit"

3. Regenerated Java and C++ client files

REST API support

1. Support for REST server to interpret a submitted query and also for rendering this information for an executed query

2. Updates to the Freemarker templates (for WebUI)

3. Safety check within Javascript (for WebUI)

JDBC API support

1. Introduces backend execution of 'ALTER SESSION' to apply the auto-limiting of resultset size

2. Added Unit Tests for PreparedStatement and Statement objects

3. Added getter setter methods to be skipped in testing for org.apache.drill.jdbc.test.Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight()

Updates based on review comments

Additional Updates

Test Cleanup

1. Revert Drill2489 hack

2. Formatting in *StatementTest

3. Removal f redundant `statement.close()`

4. Manage new Exception thrown when setting invalid maxRow values

Final updates

1. Test changes

2. Trim trailing spaces in auto-limit value (Javascript)

3. Before & After annotations to synchronize changes to system values for MaxRows(auto-limit)

Reorganized tests due to synchronized locking

Removed conflicting JsonCreator in QueryWrapper

Additional test cleanup

closes #1714

  1. … 34 more files in changeset.
DRILL-7110: Skip writing profile when an ALTER SESSION is executed (#1703)

Allows (by default) for `ALTER SESSION SET <option>=<value>` queries to NOT be writen to the profile store. This would avoid the risk of potentially adding up to a lot of profiles being written unnecessarily, since those changes are also reflected on the queries that follow.

  1. … 5 more files in changeset.
DRILL-1328: Support table statistics

  1. … 52 more files in changeset.
DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf reference count bugs & tune the execution flow & support left deep tree

closes #1504

  1. … 26 more files in changeset.
DRILL-3988: Expose Drill built-in functions & UDFs in a system table (#1483)

This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill.

An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill.

```

0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source;

+-----------------------------------------+----------------+

| source | functionCount |

+-----------------------------------------+----------------+

| built-in | 2704 |

| simple-drill-function-1.0-SNAPSHOT.jar | 12 |

| drill-url-tools-1.0.jar | 1 |

+-----------------------------------------+----------------+

3 rows selected (0.209 seconds)

```

The system table exposes information as shown. The UDF is initialized, making the `returnType` available.

The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a UDF that has overloaded arguments (see `signature`).

The `url_parse(VARCHAR-REQUIRED)` function is another example of an initialized UDF.

Rest are built-in functions that meet the query's filter criteria.

```

0: jdbc:drill:schema=sys> select * from sys.functions where name like 'random' or name like '%url%';

+-------------+----------------------------------+-------------+-----------------------------------------+

| name | signature | returnType | source |

+-------------+----------------------------------+-------------+-----------------------------------------+

| parse_url | VARCHAR-REQUIRED | LATE | built-in |

| random | | FLOAT8 | built-in |

| random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | FLOAT8 | simple-drill-function-1.0-SNAPSHOT.jar |

| url_decode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_encode | VARCHAR-REQUIRED | VARCHAR | built-in |

| url_parse | VARCHAR-REQUIRED | LATE | drill-url-tools-1.0.jar |

+-------------+----------------------------------+-------------+-----------------------------------------+

6 rows selected (0.619 seconds)

```

  1. … 9 more files in changeset.
DRILL-6731: use thread pool to run the runtime filter aggregating work closes #1459

  1. … 3 more files in changeset.
DRILL-6763: Codegen optimization of SQL functions with constant values(#1481)

closes #1481

  1. … 19 more files in changeset.
DRILL-6731: Move the BFs aggregating work from the Foreman to the RuntimeFilter

  1. … 16 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

  1. … 976 more files in changeset.
DRILL-6696: IOBE in Operator Metric Registry

  1. … 1 more file in changeset.
DRILL-6385: Support JPPD feature

  1. … 59 more files in changeset.
DRILL-6356: batch sizing for union all

closes #1255

  1. … 6 more files in changeset.
DRILL-6386: Remove unused imports and star imports.

  1. … 228 more files in changeset.
DRILL-6389: Fixed building javadocs - Added documentation about how to build javadocs - Fixed some of the javadoc warnings

closes #1276

  1. … 65 more files in changeset.
DRILL-6348: Received batches are now owned by the receive operators instead of the parent

closes #1237

  1. … 7 more files in changeset.
DRILL-6334: Minor code cleanup

This closes #1213

  1. … 5 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

  1. … 2055 more files in changeset.
DRILL-6296: Add operator metrics for batch sizing for merge join

closes #1181

  1. … 5 more files in changeset.
DRILL-6284: Add operator metrics for batch sizing for flatten

  1. … 6 more files in changeset.
DRILL-6331: Revisit Hive Drill native parquet implementation to be exposed to Drill optimizations (filter / limit push down, count to direct scan)

1. Factored out common logic for Drill parquet reader and Hive Drill native parquet readers: AbstractParquetGroupScan, AbstractParquetRowGroupScan, AbstractParquetScanBatchCreator.

2. Rules that worked previously only with ParquetGroupScan, now can be applied for any class that extends AbstractParquetGroupScan: DrillFilterItemStarReWriterRule, ParquetPruneScanRule, PruneScanRule.

3. Hive populated partition values based on information returned from Hive metastore. Drill populates partition values based on path difference between selection root and actual file path.

Before ColumnExplorer populated partition values based on Drill approach. Since now ColumnExplorer populates values for parquet files from Hive tables,

`populateImplicitColumns` method logic was changed to populated partition columns only based on given partition values.

4. Refactored ParquetPartitionDescriptor to be responsible for populating partition values rather than storing this logic in parquet group scan class.

5. Metadata class was moved to separate metadata package (org.apache.drill.exec.store.parquet.metadata). Factored out several inner classed to improve code readability.

6. Collected all Drill native parquet reader unit tests into one class TestHiveDrillNativeParquetReader, also added new tests to cover new functionality.

7. Reduced excessive logging when parquet files metadata is read

closes #1214

  1. … 64 more files in changeset.
DRILL-6275: Fixed direct memory reporting in sys.memory.

closes #1176

  1. … 2 more files in changeset.
DRILL-6323: Lateral Join - Lateral Join Batch Memory manager support using the record batch sizer

  1. … 2 more files in changeset.
DRILL-6249: Adding more unit testing documentation.

close apache/drill#1251

  1. … 25 more files in changeset.
DRILL-6323: Lateral Join - Refactor BatchMemorySize to put outputBatchSize in abstract class. Created a new JoinBatchMemoryManager to be shared across join record batches. Changed merge join to use AbstractBinaryRecordBatch instead of AbstractRecordBatch, and use JoinBatchMemoryManager

  1. … 7 more files in changeset.
DRILL-6324: Unnest - Add tests with real Unnest and real Lateral.

Code cleanup, more comments, fix license headers, and more logging.

Refactor Unnest to allow setting in incoming batch after construction

fix compilation after rebase

This closes #1223

  1. … 13 more files in changeset.
DRILL-6197: Skip duplicate entry for OperatorStats

org.apache.drill.exec.ops.FragmentStats should skip injecting the org.apache.drill.exec.ops.OperatorStats instance for these operators:

org.apache.drill.exec.proto.beans.CoreOperatorType.SCREEN

org.apache.drill.exec.proto.beans.CoreOperatorType.SINGLE_SENDER

org.apache.drill.exec.proto.beans.CoreOperatorType.BROADCAST_SENDER

org.apache.drill.exec.proto.beans.CoreOperatorType.HASH_PARTITION_SENDER

closes #1141

  1. … 1 more file in changeset.
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

    • -0
    • +34
    ./ExchangeFragmentContext.java
    • -0
    • +56
    ./ExecutorFragmentContext.java
    • -0
    • +491
    ./FragmentContextImpl.java
    • -0
    • +24
    ./RootFragmentContext.java
  1. … 213 more files in changeset.