Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
DRILL-7357: Expose Drill Metastore data through information_schema

1. Add additional columns to TABLES and COLUMNS tables.

2. Add PARTITIONS table.

3. General refactoring to adjust information_schema data retrieval from multiple sources.

closes #1860

  1. … 33 more files in changeset.
DRILL-7339: Iceberg commit upgrade and Metastore tests categorization

1. Upgraded Iceberg commit to fix issue with deletes in transaction

2. Categorize Metastore tests

closes #1842

    • -0
    • +24
    ./drill/categories/MetastoreTest.java
  1. … 7 more files in changeset.
DRILL-7046: Support for loading and parsing new RM config file closes #1652

    • -0
    • +21
    ./drill/categories/ResourceManagerTest.java
  1. … 63 more files in changeset.
DRILL-7024: Refactor ColumnWriter to simplify type-conversion shim

DRILL-7006 added a type conversion "shim" within the row set framework. Basically, we insert a "shim" column writer that takes data in one form (String, say), and does reader-specific conversions to a target format (INT, say).

The code works fine, but the shim class ends up needing to override a bunch of methods which it then passes along to the base writer. This PR refactors the code so that the conversion shim is simpler.

closes #1633

    • -0
    • +26
    ./drill/categories/RowSetTests.java
  1. … 66 more files in changeset.
DRILL-6889: Exclude Drill unit tests to avoid Travis timing out

closes #1567

  1. … 11 more files in changeset.
DRILL-6762: Fix dynamic UDFs versioning issue

1. Added UndefinedVersionDelegatingStore to serve as versioned wrapper for those stores that do not support versioning.

2. Aligned remote and local function registries version type. Type will be represented as int since ZK version is returned as int.

3. Added NOT_AVAILABLE and UNDEFINED versions to DataChangeVersion holder to indicate proper registry state.

4. Added additional trace logging.

5. Minor refactoring and clean up.

closes #1484

    • -0
    • +53
    ./drill/common/util/function/TestCheckedFunction.java
  1. … 21 more files in changeset.
DRILL-6422: Replace guava imports with shaded ones

  1. … 984 more files in changeset.
DRILL-6492: Ensure schema / workspace case insensitivity in Drill

1. StoragePluginsRegistryImpl was updated:

a. for backward compatibility at init to convert all existing storage plugins names to lower case, in case of duplicates, to log warning and skip the duplicate.

b. to wrap persistent plugins registry into case insensitive store wrapper (CaseInsensitivePersistentStore) to ensure all given keys are converted into lower case when performing insert, update, delete, search operations.

c. to load system storage plugins dynamically by @SystemStorage annotation.

2. StoragePlugins class was updated to stored storage plugins configs by name in case insensitive map.

3. SchemaUtilities.searchSchemaTree method was updated to convert all schema names into lower case to ensure that are they are matched case insensitively (all schemas are stored in Drill in lower case).

4. FileSystemConfig was updated to store workspaces by name in case insensitive hash map.

5. All plugins schema factories are now extend AbstractSchemaFactory to ensure that given schema name is converted to lower case.

6. New method areTableNamesAreCaseInsensitive was added to AbstractSchema to indicate if schema tables names are case insensitive. By default, false. Schema implementation is responsible for table names case insensitive search in case it supports one. Currently, information_schema, sys and hive do so.

7. System storage plugins (information_schema, sys) were refactored to ensure their schema, table names are case insensitive, also the annotation @SystemPlugin and additional constructor were added to allow dynamically load system plugins at storage plugin registry during init phase.

8. MetadataProvider was updated to concert all schema filter conditions into lower case to ensure schema would be matched case insensitively.

9. ShowSchemasHandler, ShowTablesHandler, DescribeTableHandler were updated to ensure schema / tables names (this depends if schema supports case insensitive table names) would be found case insensitively.

git closes #1439

    • -0
    • +15
    ./drill/common/map/TestCaseInsensitiveMap.java
  1. … 55 more files in changeset.
DRILL-6656: Disallow extra semicolons and multiple statements on the same line.

closes #1415

  1. … 144 more files in changeset.
DRILL-6438: Remove excess logging form the tests. - Removed usages of System.out and System.err from the test and replaced with loggers

closes #1284

    • -2
    • +2
    ./drill/test/Drill2130CommonHamcrestConfigurationTest.java
  1. … 89 more files in changeset.
DRILL-6363: Upgrade jmockit and mockito libs

  1. … 6 more files in changeset.
DRILL-6320: Fixed license headers.

closes #1207

    • -2
    • +1
    ./drill/categories/HbaseStorageTest.java
    • -2
    • +1
    ./drill/categories/HiveStorageTest.java
    • -2
    • +1
    ./drill/categories/JdbcStorageTest.java
    • -1
    • +0
    ./drill/categories/KafkaStorageTest.java
    • -2
    • +1
    ./drill/categories/KuduStorageTest.java
    • -2
    • +1
    ./drill/categories/MongoStorageTest.java
    • -2
    • +1
    ./drill/categories/SqlFunctionTest.java
  1. … 2052 more files in changeset.
DRILL-6249: Adding more unit testing documentation.

close apache/drill#1251

  1. … 28 more files in changeset.
DRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet

  1. … 30 more files in changeset.
DRILL-6049: Misc. hygiene and code cleanup changes

close apache/drill#1085

  1. … 123 more files in changeset.
DRILL-5989: Categories some tests to speed up smoke tests. Made travis run tests.

closes #1053

    • -0
    • +25
    ./drill/categories/KafkaStorageTest.java
  1. … 31 more files in changeset.
DRILL-5922: - The QueryContext was never closed when the Foreman finished, so it's child allocator was never closed. Now it is. - The PlanSplitter created a QueryContext temporarily to construct an RPC message but never closed it. Now the temp QueryContext is closed. - The waitForExit method was error prone. Changed it to use the standard condition variable pattern. - Fixed timeouts in graceful shutdown tests

  1. … 6 more files in changeset.
DRILL-5783, DRILL-5841, DRILL-5894: Rationalize test temp directories

This change includes:

DRILL-5783:

- A unit test is created for the priority queue in the TopN operator.

- The code generation classes passed around a completely unused function registry reference in some places so it is removed.

- The priority queue had unused parameters for some of its methods so it is removed.

DRILL-5841:

- Created standardized temp directory classes DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. And updated all unit tests to use them.

DRILL-5894:

- Removed the dfs_test storage plugin for tests and replaced it with the already existing dfs storage plugin.

Misc:

- General code cleanup.

- Removed unnecessary use of String.format in the tests.

This closes #984

    • -0
    • +167
    ./drill/test/DirTestWatcher.java
    • -0
    • +193
    ./drill/test/SubDirTestWatcher.java
    • -0
    • +104
    ./drill/test/TestTools.java
    • -61
    • +0
    ./drill/testutils/DirTestWatcher.java
  1. … 361 more files in changeset.
DRILL-5809 made the storage format of system options backwards compatible, and avoided storing unnecessary option info.

- Fixed forward compatability issue

- Added error message for debugging

- Fix flakey test

- Cleaned up bad logging

- Applied comments

closes #957

    • -0
    • +61
    ./drill/testutils/DirTestWatcher.java
  1. … 19 more files in changeset.
DRILL-5752 this change includes:

1. Increased test parallelism and fixed associated bugs

2. Added test categories and categorized tests appropriately

- Don't exclude anything by default

- Increase test timeout

- Fixed flakey test

closes #940

    • -0
    • +25
    ./drill/categories/HbaseStorageTest.java
    • -0
    • +25
    ./drill/categories/HiveStorageTest.java
    • -0
    • +25
    ./drill/categories/JdbcStorageTest.java
    • -0
    • +25
    ./drill/categories/JdbcTest.java
    • -0
    • +25
    ./drill/categories/KuduStorageTest.java
    • -0
    • +25
    ./drill/categories/MemoryTest.java
    • -0
    • +25
    ./drill/categories/MongoStorageTest.java
    • -0
    • +25
    ./drill/categories/OperatorTest.java
    • -0
    • +25
    ./drill/categories/OptionsTest.java
    • -0
    • +25
    ./drill/categories/ParquetTest.java
    • -0
    • +26
    ./drill/categories/PlannerTest.java
    • -0
    • +25
    ./drill/categories/SecurityTest.java
    • -0
    • +25
    ./drill/categories/SlowTest.java
    • -0
    • +25
    ./drill/categories/SqlFunctionTest.java
    • -0
    • +25
    ./drill/categories/SqlTest.java
  1. … 251 more files in changeset.
DRILL-5723: Added System Internal Options That can be Modified at Runtime Changes include:

1. Addition of internal options.

2. Refactoring of OptionManagers and OptionValidators.

3. Fixed ambiguity in the meaning of an option type, and changed its name to accessibleScopes.

4. Updated javadocs in the Option System classes.

5. Added RestClientFixture for testing the Rest API.

6. Fixed flakey test in TestExceptionInjection caused by race condition.

7. Fixed various tests which started zookeeper but failed to shut it down at the end of tests.

8. Added port hunting to the Drill Webserver for testing

9. Fixed various flaky tests

10. Fix compile issue

closes #923

  1. … 85 more files in changeset.
DRILL-5260: Extend "Cluster Fixture" test framework

- Config option to suppress printing of CSV and other output. (Allows

printing for single tests, not printing when running from Maven.)

- Parsing of query profiles to extract plan and run time information.

- Fix bug in log fixture when enabling logging for a package.

- Improved ZK support.

- Set up the new CTTAS default temporary workspace for tests.

- Clean up persistent storage files on disk to avoid CTTAS startup

failures.

- Provides a set of examples for how to use the cluster fixture.

closes #753

  1. … 18 more files in changeset.
DRILL-4994: Add back JDBC prepared statement for older servers

When the JDBC client is connected to an older Drill server, it always

attempted to use server-side prepared statement with no fallback.

With this change, client will check server version and will fallback to the

previous client-side prepared statement (which is still limited to only execute

queries and does not provide metadata).

close #613

    • -0
    • +103
    ./drill/common/TestVersion.java
  1. … 23 more files in changeset.
DRILL-4800: Various fixes. Fix buffer underflow exception in BufferedDirectBufInputStream. Fix writer index for in64 dictionary encoded types. Added logging to help debug. Fix memory leaks. Work around issues with of InputStream.available() ( Do not use hasRemainder; Remove check for EOF in BufferedDirectBufInputStream.read() ). Finalize defaults. Remove commented code. Addressed review comments

This closes #611

  1. … 11 more files in changeset.
DRILL-4800: Parallelize column reading. Read/Decode fixed width fields in parallel Decoding var length columns in parallel Use simplified decompress method for Gzip and Snappy decompression. Avoids concurrency issue with Parquet decompression. (It's also faster). Stress test Parquet read write Parallel column reader is disabled by default (may perform less well under higher concurrency)

  1. … 10 more files in changeset.
DRILL-3987: (CLEANUP) Final cleanups to get complete working build/distribution

- small cleanups

- move Hook to drill-adbc

- update distribution assembly to include new modules

This closes #250

  1. … 31 more files in changeset.
DRILL-3987: (MOVE) Move logical expressions and operators out of common. Move to new drill-logical model.

    • -93
    • +0
    ./drill/common/logical/data/OrderTest.java
    • -51
    • +0
    ./drill/storage/CheckStorageConfig.java
  1. … 210 more files in changeset.
DRILL-3742: Classpath scanning and build improvement

Makes the classpath scanning a build time class discovery

Makes the fmpp generation incremental

Removes some slowness in DrillBit closing

Reduces the build time by 30%

This closes #148

    • -5
    • +9
    ./drill/storage/CheckStorageConfig.java
  1. … 143 more files in changeset.
DRILL-2304: Manage options with lowercase names

This closes #90

Major refactoring:

+ Removed unnecessary methods from OptionManagers, and refactored implementations

+ Removed OptionAdmin interface, which was a delegate to validators

Other fixes:

+ Added CaseInsensitiveMap

+ Moved OptionValue creation from TypeValidators to SetOptionHandler

+ Fixed error message in BaseOptionManager

+ Added documentation for all OptionManagers, and OptionValidator

DRILL-2304: Addressing Jason's review comments (1)

+ Use generic Map rather than CaseInsensitiveMap for member variables and parameters

+ SessionOptionManager should use a ConcurrentHashMap

    • -0
    • +69
    ./drill/common/map/TestCaseInsensitiveMap.java
  1. … 14 more files in changeset.
DRILL-3497: Throw UserException#validationError instead of SetOptionException and ExpressionParsingException for options

This closes #98

+ Fixed bug in PositiveLongValidator (0 is not positive)

+ Added UserExceptionMatcher for unit tests

DRILL-3497: Addressing Jason's review comments (1)

    • -0
    • +71
    ./drill/test/UserExceptionMatcher.java
  1. … 11 more files in changeset.