Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-4303: ESRI Shapefile (shp) Format Plugin

  1. … 20 more files in changeset.
DRILL-7177: Format Plugin for Excel Files

closes #1749

  1. … 16 more files in changeset.
DRILL-7273: Introduce operators for handling metadata

closes #1886

  1. … 155 more files in changeset.
DRILL-7188: Revert DRILL-6642: Update protocol-buffers version

1. Updated protobuf to version 3.6.1

2. Added protobuf to the root pom dependency management

3. Added classes BoundedByteString and LiteralByteString for compatibility with HBase

4. Added ProtobufPatcher to provide compatibility with MapR-DB and HBase

  1. … 26 more files in changeset.
DRILL-7048: Implement JDBC Statement.setMaxRows() with System Option

This introduces support for JDBC's Statement.setMaxRows(int) API, which can help Drill execute a query much faster if it knows that not ALL the records in the resultset will be consumed upfront.

This Commit introduces the core changes to support the feature within Drill's execution engine

Protobuf Changes

1. RunQuery: Added "autolimit_rowcount"

2. QueryProfile: Added "autoLimit"

3. Regenerated Java and C++ client files

REST API support

1. Support for REST server to interpret a submitted query and also for rendering this information for an executed query

2. Updates to the Freemarker templates (for WebUI)

3. Safety check within Javascript (for WebUI)

JDBC API support

1. Introduces backend execution of 'ALTER SESSION' to apply the auto-limiting of resultset size

2. Added Unit Tests for PreparedStatement and Statement objects

3. Added getter setter methods to be skipped in testing for org.apache.drill.jdbc.test.Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight()

Updates based on review comments

Additional Updates

Test Cleanup

1. Revert Drill2489 hack

2. Formatting in *StatementTest

3. Removal f redundant `statement.close()`

4. Manage new Exception thrown when setting invalid maxRow values

Final updates

1. Test changes

2. Trim trailing spaces in auto-limit value (Javascript)

3. Before & After annotations to synchronize changes to system values for MaxRows(auto-limit)

Reorganized tests due to synchronized locking

Removed conflicting JsonCreator in QueryWrapper

Additional test cleanup

closes #1714

  1. … 31 more files in changeset.
DRILL-7096: Develop vector for canonical Map<K,V>

- Added new type DICT;

- Created value vectors for the type for single and repeated modes;

- Implemented corresponding FieldReaders and FieldWriters;

- Made changes in EvaluationVisitor to be able to read values from the map by key;

- Made changes to DrillParquetGroupConverter to be able to read Parquet's MAP type;

- Added an option `store.parquet.reader.enable_map_support` to disable reading MAP type as DICT from Parquet files;

- Updated AvroRecordReader to use new DICT type for Avro's MAP;

- Added support of the new type to ParquetRecordWriter.

  1. … 108 more files in changeset.
DRILL-7014: Format plugin for LTSV files closes #1627

  1. … 15 more files in changeset.
DRILL-6582: SYSLOG (RFC-5424) Format Plugin closes #1530

  1. … 19 more files in changeset.
DRILL-6946: Implement java.sql.Connection setSchema and getSchema methods in DrillConnectionImpl closes #1596

  1. … 8 more files in changeset.
DRILL-6925: Unable to generate Protobuf

- use ${maven.multiModuleProjectDirectory}/header to find header file from any submodule

- suppress UnresolvedMavenProperty, since IDE expects that property should be set explicitly

- update "kr.motd.maven:os-maven-plugin" github.com/trustin/os-maven-plugin to the latest 1.6.1 version

- correction of ${user.name} propery for "maven-jar-plugin" <Built-By>

- update "apache-rat-plugin" to solve undefined "excludeSubprojects" in IDE

- regenerate Java and C++ protobuf files

closes #1585

  1. … 5 more files in changeset.
DRILL-6834: Introduce option to disable result set for DDL queries for JDBC connection

- Added session-scoped option `drill.exec.fetch_resultset_for_ddl` to control whether update count or result set should be returned for JDBC connection session. By default the option is set to `true` which ensures that result set is returned;

- Updated Drill JDBC: `DrillCursor` and `DrillStatement` to achieve desired behaviour.

closes #1549

  1. … 25 more files in changeset.
DRILL-6642: Update protocol-buffers version

1. Updated protobuf to version 3.6.1

2. Added protobuf to the root pom dependency management

3. Added classes BoundedByteString and LiteralByteString for compatibility with HBase

4. Added ProtobufPatcher to provide compatibility with MapR-DB and HBase

closes #1639

  1. … 26 more files in changeset.
DRILL-6381: Address code review comments (part 3).

DRILL-6381: Add missing joinControl logic for INTERSECT_DISTINCT.

- Modified HashJoin's probe phase to process INTERSECT_DISTINCT.

- NOTE: For build phase, the functionality will be same as for SemiJoin when it is added later.

DRILL-6381: Address code review comment for intersect_distinct.

DRILL-6381: Rebase on latest master and fix compilation issues.

DRILL-6381: Generate protobuf files for C++ native client.

DRILL-6381: Use shaded Guava classes. Add more comments and Javadoc.

  1. … 31 more files in changeset.
DRILL-6710: Disallow negative scale for decimal data type

  1. … 11 more files in changeset.
DRILL-6639: Exception happens while displaying operator profiles for some queries

  1. … 16 more files in changeset.
DRILL-6639: Exception happens while displaying operator profiles for some queries

closes #1404

  1. … 16 more files in changeset.
DRILL-6635: PartitionLimit for Lateral/Unnest Protobuf changes to add new operator PartitionLimit

  1. … 3 more files in changeset.
DRILL-6627: Adding REGEX_SUB_SCAN operator to protobuf file

- The operator is added to Java based UserBitShared.proto and C++ based UserBitShared.pb.h

- Java and C++ protobuf files are regenerated

  1. … 3 more files in changeset.
DRILL-6179: Added pcapng-format support

  1. … 21 more files in changeset.
DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.

1. Added protobuf files (incl generated C++ and Java)

2. Updated JdbcSubScan's getOperatorType API

closes #1297

  1. … 4 more files in changeset.
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.

This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes

Set the query ID string and display in WebUI

closes #1301

  1. … 8 more files in changeset.
DRILL-6385: Support JPPD feature

  1. … 60 more files in changeset.
DRILL-6094: Decimal data type enhancements

Add ExprVisitors for VARDECIMAL

Modify writers/readers to support VARDECIMAL

- Added usage of VarDecimal for parquet, hive, maprdb, jdbc;

- Added options to store decimals as int32 and int64 or fixed_len_byte_array or binary;

Add UDFs for VARDECIMAL data type

- modify type inference rules

- remove UDFs for obsolete DECIMAL types

Enable DECIMAL data type by default

Add unit tests for DECIMAL data type

Fix mapping for NLJ when literal with non-primitive type is used in join conditions

Refresh protobuf C++ source files

Changes in C++ files

Add support for decimal logical type in Avro.

Add support for date, time and timestamp logical types.

Update Avro version to 1.8.2.

  1. … 201 more files in changeset.
DRILL-6300: Refresh protobuf C++ source files

close apache/drill#1194

  1. … 1 more file in changeset.
DRILL-6289: Cluster view should show more relevant information

Protobuf change to carry HTTP port info

Allow CORS for access to remote Drillbit metrics

Cross-origin resource sharing (CORS) is required to ensure that the WebServer is able serve REST calls for status pages.

Materialize relevant metrics

1. Heap memory (incl usage)

2. Heap memory (incl usage)

3. Average System Load (last 1 min)

4. Option to view from other nodes (pop out)

5. Added Glyphicons

Update System Table and related tests

1. Updated System Table to show HTTP port

2. Updated unit tests

Skip updating remote bit info when HTTPS (SSL) or Authentication is enabled.

Default CpuGaugeSet is public; Added Gauges

* CPU Utiization by Drill

* Uptime

Show ALL Buttons, but do HTTPS Check

Reduce power button to icon

Allowing CORS for /status/metrics only

Accounting for situations when JVM does not report Process CPU Load

i.e. returned value is negative.

See https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad()

Addressed shutdown security conditions

Added C++ Client Protobuf

Added steps for Protobuf generation to protocol/readme.txt

This closes #1203

  1. … 13 more files in changeset.
DRILL-6331: Revisit Hive Drill native parquet implementation to be exposed to Drill optimizations (filter / limit push down, count to direct scan)

1. Factored out common logic for Drill parquet reader and Hive Drill native parquet readers: AbstractParquetGroupScan, AbstractParquetRowGroupScan, AbstractParquetScanBatchCreator.

2. Rules that worked previously only with ParquetGroupScan, now can be applied for any class that extends AbstractParquetGroupScan: DrillFilterItemStarReWriterRule, ParquetPruneScanRule, PruneScanRule.

3. Hive populated partition values based on information returned from Hive metastore. Drill populates partition values based on path difference between selection root and actual file path.

Before ColumnExplorer populated partition values based on Drill approach. Since now ColumnExplorer populates values for parquet files from Hive tables,

`populateImplicitColumns` method logic was changed to populated partition columns only based on given partition values.

4. Refactored ParquetPartitionDescriptor to be responsible for populating partition values rather than storing this logic in parquet group scan class.

5. Metadata class was moved to separate metadata package (org.apache.drill.exec.store.parquet.metadata). Factored out several inner classed to improve code readability.

6. Collected all Drill native parquet reader unit tests into one class TestHiveDrillNativeParquetReader, also added new tests to cover new functionality.

7. Reduced excessive logging when parquet files metadata is read

closes #1214

  1. … 63 more files in changeset.
DRILL-5924: native-client: Support user-specified CXX_FLAGS

This closes #1022

  1. … 3 more files in changeset.
DRILL-5431: Regenerate protobuf files

This closes #950

DRILL-5799: native-client: Support alternative build directories

closes #946

DRILL-4335: Apache Drill should support network encryption.

NOTE: This pull request provides support for on-wire encryption using SASL framework. Communication channel covered is:

1) C++ Drill Client and Drillbit channel.

close apache/drill#809

  1. … 12 more files in changeset.