Clone Tools
  • last updated 26 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7388: Kafka improvements

1. Upgraded Kafka libraries to 2.3.1 (DRILL-6739).

2. Added new options to support the same features as native JSON reader:

a. store.kafka.reader.skip_invalid_records, default: false (DRILL-6723);

b. store.kafka.reader.allow_nan_inf, default: true;

c. store.kafka.reader.allow_escape_any_char, default: false.

3. Fixed issue when Kafka topic contains only one message (DRILL-7388).

4. Replaced Gson parser with Jackson to parse JSON in the same manner as Drill native Json reader.

5. Performance improvements: Kafka consumers will be closed async, fixed issue with resource leak (DRILL-7290), moved to debug unnecessary info logging.

6. Updated bootstrap-storage-plugins.json to reflect actual Kafka connection properties.

7. Added unit tests.

8. Refactoring and code clean up.

closes #1901

  1. … 33 more files in changeset.
DRILL-7401: Upgrade to SqlLine 1.9.0

closes #1875

  1. … 3 more files in changeset.
DRILL-7402: Suppress batch dumps for expected failures in tests

Drill provides a way to dump the last few batches when an error

occurs. However, in tests, we often deliberately cause something

to fail. In this case, the batch dump is unnecessary.

This enhancement adds a config property, disabled in tests, that

controls the dump activity. The option is enabled in the one test

that needs it enabled.

closes #1872

  1. … 4 more files in changeset.
DRILL-6096: Provide mechanism to configure text writer configuration

1. Usage of format plugin configuration allows to specify line and field delimiters, quotes and escape characters.

2. Usage of system / session options allows to specify if writer should add headers, force quotes.

closes #1873

  1. … 19 more files in changeset.
DRILL-7351: Added tokens to Web forms to prevent CSRF attacks

  1. … 21 more files in changeset.
DRILL-7222: Visualize estimated and actual row counts for a query

With statistics in place, it is useful to have the estimated rowcount along side the actual rowcount query profile's operator overview. A toggle button allows this with the estimated rows hidden by default

We can extract this from the Physical Plan section of the profile.

Added a toggle-ready table-column header

closes #1779

  1. … 4 more files in changeset.
DRILL-7356: Introduce session options for the Drill Metastore

closes #1846

  1. … 2 more files in changeset.
DRILL-7338: REST API calls to Drill fail due to insufficient heap memory

This PR allows for the 85% threshold to be customizable with a value of 0 meant for disabling.

closes #1837

  1. … 2 more files in changeset.
DRILL-7313: Use Hive schema for MaprDB native reader when field was empty

- Added all_text_mode option for hive maprDB Json

- Improved logic to convert Hive's schema into Drill's one

- Added unit tests for schema conversion

  1. … 27 more files in changeset.
DRILL-7273: Introduce operators for handling metadata

closes #1886

  1. … 156 more files in changeset.
DRILL-7276: Fixed an XSS vulnerability in Drill Web-UI query profile page

DRILL-7204: Add proper validation when creating plugin

- Added validation for an empty plugin name.

- Added an URL encoding for pluing name, so plugins with special characters can be accessed without issues.

- Replaced alerts with modal windows.

- Added a confirmation dialog when disabling a plugin on Update page.

    • -0
    • +54
    ./rest/confirmationModals.ftl
  1. … 1 more file in changeset.
DRILL-7196: Queries are still runnable on disabled plugins

- Storage client is not created anymore for disabled plugins

- GET "/storage/{name}.json" endpoint now working with

plugin configuration directly, without client instantination.

It have increased UI responsitivity.

- Hbase and mongo base test classes refactored to honor enabled

plugin attribute

- Fixed path contructor for mongo test datasets:

Now it is cross-platform

- Fixed test json files format which using plugin definitions

- Code cleanup

    • -103
    • +91
    ./bootstrap-storage-plugins.json
  1. … 106 more files in changeset.
DRILL-7201: Strange symbols in error window (Windows)

Looks like some unicode characters creeped in, causing the rendering to be messed up ONLY for Windows OS. This PR patches that issue and ensures that the Alert symbol also appears correctly.

DRILL-7202: Failed query shows warning that fragments made no progress

This bug arises because of the `toggleWarning()` javascript function that shows the warning if the number of slow major fragments matches the number of major fragments.

For failed queries, the number of major fragments = 0; and that matches the number of tags (which is also zero for a non-running query). Hence the warning shows.

DRILL-7190: Missing backward compatibility for REST API with DRILL-6562

  1. … 1 more file in changeset.
DRILL-7048: Implement JDBC Statement.setMaxRows() with System Option

This introduces support for JDBC's Statement.setMaxRows(int) API, which can help Drill execute a query much faster if it knows that not ALL the records in the resultset will be consumed upfront.

This Commit introduces the core changes to support the feature within Drill's execution engine

Protobuf Changes

1. RunQuery: Added "autolimit_rowcount"

2. QueryProfile: Added "autoLimit"

3. Regenerated Java and C++ client files

REST API support

1. Support for REST server to interpret a submitted query and also for rendering this information for an executed query

2. Updates to the Freemarker templates (for WebUI)

3. Safety check within Javascript (for WebUI)

JDBC API support

1. Introduces backend execution of 'ALTER SESSION' to apply the auto-limiting of resultset size

2. Added Unit Tests for PreparedStatement and Statement objects

3. Added getter setter methods to be skipped in testing for org.apache.drill.jdbc.test.Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight()

Updates based on review comments

Additional Updates

Test Cleanup

1. Revert Drill2489 hack

2. Formatting in *StatementTest

3. Removal f redundant `statement.close()`

4. Manage new Exception thrown when setting invalid maxRow values

Final updates

1. Test changes

2. Trim trailing spaces in auto-limit value (Javascript)

3. Before & After annotations to synchronize changes to system values for MaxRows(auto-limit)

Reorganized tests due to synchronized locking

Removed conflicting JsonCreator in QueryWrapper

Additional test cleanup

closes #1714

  1. … 30 more files in changeset.
DRILL-7062: Initial implementation of run-time rowgroup pruning closes #1738

  1. … 24 more files in changeset.
DRILL-7096: Develop vector for canonical Map<K,V>

- Added new type DICT;

- Created value vectors for the type for single and repeated modes;

- Implemented corresponding FieldReaders and FieldWriters;

- Made changes in EvaluationVisitor to be able to read values from the map by key;

- Made changes to DrillParquetGroupConverter to be able to read Parquet's MAP type;

- Added an option `store.parquet.reader.enable_map_support` to disable reading MAP type as DICT from Parquet files;

- Updated AvroRecordReader to use new DICT type for Avro's MAP;

- Added support of the new type to ParquetRecordWriter.

  1. … 108 more files in changeset.
DRILL-7110: Skip writing profile when an ALTER SESSION is executed (#1703)

Allows (by default) for `ALTER SESSION SET <option>=<value>` queries to NOT be writen to the profile store. This would avoid the risk of potentially adding up to a lot of profiles being written unnecessarily, since those changes are also reflected on the queries that follow.

  1. … 5 more files in changeset.
DRILL-7148: Use improved join cardinality and ndv estimation with statistics

closes #1744

  1. … 11 more files in changeset.
DRILL-7095: Expose table schema (TupleMetadata) to physical operator (EasySubScan)

1. Add system / session option store.table.use_schema_file to control if file schema can be used during query execution. False by default.

2. Added methods in StoragePlugin interface which allow to create Group Scan with provided table schema.

3. EasyGroupScan and EasySubScan now contain table schema, also they are able to serialize / deserialize it along with other scan properties.

4. DrillTable which is the main entry point for schema provisioning, has method to store schema and later uses it to create physical scan.

5. WorkspaceSchema when returning Drill table instance will get table schema from table root if available and if store.table.use_schema_file is set to true.

This PR is the next step for Schema Provisioning project which currently exposes schema only for text reader.

closes #1696

  1. … 15 more files in changeset.
DRILL-7061: Disable LIMIT Rows Option

Freemarker by default introduces a comma in numeric values greater than 999. This corrects that by removing the ',' in the default limit size.

However, since a Server-side implementation is in progress (DRILL-6960 and DRILL-7048), it is best to disable this for now. The latest commit in this will hide those capabilities in the WebUI until the server-side feature goes in.

X

closes #1689

DRILL-7021: HTTPD Throws NPE and Doesn't Recognize Timeformat

    • -89
    • +89
    ./bootstrap-storage-plugins.json
  1. … 9 more files in changeset.
DRILL-7060: Support JsonParser Feature 'ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER' (#1663)

  1. … 7 more files in changeset.
DRILL-6952: Host compliant text reader on the row set framework

The result set loader allows controlling batch sizes. The new scan framework

built on top of that framework handles projection, implicit columns, null

columns and more. This commit converts the "new" ("compliant") text reader

to use the new framework. Options select the use of the V2 ("new") or V3

(row-set based) versions. Unit tests demonstrate V3 functionality.

closes #1683

  1. … 58 more files in changeset.
DRILL-7052: Relative path for URL redirection

  1. … 5 more files in changeset.
DRILL-7036: Improve UI for alert and error messages closes #1644 This PR standardizes error and alert messages to a cleaner interface by leveraging Bootstraps UX elements for publishing the messages in a presentable format. Exceptions reported back to the browser and rendered in a neat tabular format (using Panels) All errors can be redirected to errorMessage.ftl which will render it in a neat format. Alerts are replaced with modals. Interactions (pages) affected by Alert modals 1. Missing Query submission 2. profile Query Rerun 3. invalid Profile Listing Fetch 4. invalid Option Value for update 5. Missing username/password submission

The errorMessage.ftl has been moved to root dir, and unused `error.ftl` was removed

    • -0
    • +104
    ./rest/alertModals.ftl
    • -0
    • +37
    ./rest/errorMessage.ftl
    • -27
    • +27
    ./rest/static/js/querySubmission.js
  1. … 3 more files in changeset.
DRILL-7117: Support creation of equi-depth histogram for selected data types.

Support int/bigint/float4/float8, time/timestamp/date and boolean.

Build the histogram from the t-digest byte array and serialize as JSON string.

More changes for serialization/deserialization.

Add code-gen stubs (empty) for VarChar/VarBinary types.

Address review comments (part 1). Add unit test.

Address review comments (part 2) for sampling.

close apache/drill#1715

  1. … 15 more files in changeset.
DRILL-7046: Support for loading and parsing new RM config file closes #1652

    • -0
    • +34
    ./drill-rm-default.conf
  1. … 62 more files in changeset.