Clone Tools
  • last updated 29 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][COMP] Refactor drop dataverse and drop function

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactor feed handling when processing drop dataverse

and drop function

- Move feed dependency checking from QueryTranslator

to MetadataNode to align with other entities

Change-Id: I493dfffd77c596bb3485eccb00b417d40f47b647

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/7066

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

    • -1
    • +46
    ./asterix/metadata/MetadataNode.java
  1. … 4 more files in changeset.
[NO ISSUE][COMP][RT] New library deployment model

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Deploy external libraries using Hyracks jobs

- Library manager is no longer available on CC

- Add IServlet.init() which is invoked during web server startup

- External libraries can no longer provide implementations of

IExternalDataSourceFactory because external libraries are

not available on CC

- Added testcase for an external adapter

Change-Id: If64f99f6a15b81b1e426239bde63360f5ef57059

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6863

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -0
    • +12
    ./asterix/metadata/MetadataManager.java
    • -55
    • +108
    ./asterix/metadata/MetadataNode.java
    • -0
    • +10
    ./asterix/metadata/api/IMetadataManager.java
    • -0
    • +11
    ./asterix/metadata/api/IMetadataNode.java
    • -1
    • +13
    ./asterix/metadata/entities/Library.java
    • -19
    • +21
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 83 more files in changeset.
[ASTERIXDB-2752][MTD] Add metadata for synonyms on upgrade

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- On version upgrade insert synonyms dataset entry

into Metadata.Dataset and corresponding type entry

into Metadata.Datatype

Change-Id: I32b3bafab3c130666f13063d8959f5b6e5fb7261

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6944

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Michael Blow <mblow@apache.org>

Merge commit 'c25c36d1b0' from 'gerrit/mad-hatter'

Change-Id: I8a78950cb81f0a29ceae94fa439eec700c17d06d

    • -5
    • +7
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 8 more files in changeset.
[ASTERIXDB-2751][EXT] Issue a warning if external dataset configuration returns no files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Issue a warning when creating an external dataset if the dataset

configuration returns no files.

- Issue a warning when querying an external dataset if no files

are returned.

Change-Id: I0b2be49bf3c69c3e055d942a5c3f0106b3069d52

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6843

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -5
    • +7
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 48 more files in changeset.
[NO ISSUE] Improve handling of udf parameter/return types

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Prohibit parameter/return type specification for inline udfs

- Prohibit parameter/return type quantifier (?) for external udfs

- Allow external udfs to refer to types from other dataverses

- Do not store type quantifiers in udf metadata

- Align names of anonymous types automatically created by

QueryTranslator for datasets and udfs

- Reserve user-defined type names starting from '$' for system use

- Add testcases

- Prohibit UdfServlet operations if cluster is not active

- Return correct path from IOManager.getWorkspacePath()

- Test framework: support multiple commands in .lib.sqlpp files

- Remove CC files when deleting old instance data in

AsterixHyracksIntegrationUtil

Change-Id: I09bdfd8c7b26750d12339034d2143a16102d1212

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6545

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -9
    • +1
    ./asterix/metadata/MetadataManager.java
    • -20
    • +19
    ./asterix/metadata/entities/Function.java
    • -10
    • +0
    ./asterix/metadata/utils/DatasetUtil.java
    • -0
    • +37
    ./asterix/metadata/utils/TypeUtil.java
  1. … 27 more files in changeset.
Merge commit '6904d36' from 'gerrit/mad-hatter'

Change-Id: Ib118bb2d1b18a93412015b1b652684c3fbb11e52

    • -10
    • +20
    ./asterix/metadata/utils/MetadataLockUtil.java
  1. … 2 more files in changeset.
Merge commit '3a62802' from 'gerrit/mad-hatter'

Change-Id: Ia70a791f6d9fd05854017ef7692adc14a6a5f37f

  1. … 8 more files in changeset.
[NO ISSUE][COMP] Refactor locking in CREATE DATASET

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactor locking in CREATE DATASET to make it extensible

by products (introduce QueryTranslator.createDatasetBegin())

- Refactor MetadataLockUtil.createDatasetBegin() into two methods

Change-Id: I1b430a5e8f4a3cdf1a360397e3d160f7b8c95d26

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6323

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -6
    • +27
    ./asterix/metadata/utils/MetadataLockUtil.java
  1. … 1 more file in changeset.
Merge commit 'f1cd417' from 'gerrit/mad-hatter'

Change-Id: I0f30399a6485708f4b154af09ea36377fe4d3dbf

    • -5
    • +5
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 9 more files in changeset.
Merge commit '4bf7048' from 'gerrit/mad-hatter'

Change-Id: I06432e8daa1669862d1f23a04efdfec5b1e6d7b5

  1. … 2 more files in changeset.
[NO ISSUE][COMP] Rename NOT NULL to NOT UNKNOWN

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Rename NOT NULL type quantifier to NOT UNKNOWN

in CREATE DATASET type specification

- Add IsMissable field to datatype metadata to support

storing all possible NULL/MISSING combinations

Change-Id: Ia7f22ba5d212e3b1824fafb571e1724f2fd25ea9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6306

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +28
    ./asterix/metadata/utils/TypeUtil.java
  1. … 35 more files in changeset.
Merge commit '3e2623cf64827e891ecd8c6dea13d03f5599c6a5' into HEAD

Change-Id: I89091d8bf8575c73d26f497d5fbffe780adbedf9

    • -3
    • +17
    ./asterix/metadata/utils/DatasetUtil.java
    • -2
    • +4
    ./asterix/metadata/utils/MetadataLockUtil.java
  1. … 7 more files in changeset.
Merge commit 'bc0f7e0' from branch 'gerrit/mad-hatter'

Change-Id: I629850e0f6ac7e1afae3fb5c6e1f1f859b2ead72

    • -11
    • +11
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 13 more files in changeset.
[ASTERIXDB-2176] Python UDFs

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Allow UDFs to be shiv-packaged python modules with their

dependencies

- Use pyro for python RPC

- Maven build for pyro package into server

- Remove JObject spatial types

- Simpler conversion from java primitive/standard types and

collections

to JObject equivalents

Change-Id: Ibea23a2e9308132f343d80eff04ede9a235aa021

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -4
    • +14
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 106 more files in changeset.
[ASTERIXDB-2715][STO] Dynamic Memory Component Architecture

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Introduce a dynamic memory component architecture that uses a global

virtual buffer cache to manage the write memory for all LSM-trees.

- When the overall write memory is nearly full, we flush a dataset

partition at a time using a round-robin way. Additionally, we allow

users to configure the maximum size of filtered memory components

to provide better pruning capability.

- Clean up legacy code for statically allocating write memory to each

dataset.

- Remove the following parameters:

storage.metadata.memorycomponent.numpages

storage.max.active.writable.datasets

- Add the following parameters:

storage.memorycomponent.flush.threshold (default: 0.9)

storage.filtered.memorycomponent.max.size (default 0)

Change-Id: Ia6a0f4de020acd7af89ef630322526c4be5076e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5824

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 44 more files in changeset.
[NO ISSUE] Fix UDF Metadata Transactions

- Move UDF Metadata Transactions to be initiated from the CC

directly

- Remove old UDF initialization code

- Make DROP DATAVERSE remove libraries

- Make NCs properly read libraries on init with multipart dataverse

names

Change-Id: Ibda23a2e8308937f343d80eff04ede9a235da0d2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4923

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 45 more files in changeset.
[ASTERIXDB-2713][EXT] CSV & TSV support for external dataset p2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- validate adapter configuration

- ignore empty lines in CSV/TSV files

- require "header" parameter for CSV/TSV formats

- make some parameters case-insensitive

- few fixes and clean-ups

Change-Id: I2f523de0d482a358ada0c27236ff24616ad0d7da

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5848

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -5
    • +5
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 37 more files in changeset.
[NO ISSUE][MTD] Customizable dataset compaction policy in metadata

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Allow product extensions to customize how dataset's

compaction policy is stored in the metadata

Change-Id: I0216af5eabdf5ff269ba2d3feccf1371d273315b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5224

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[ASTERIXDB-2708] Introduce batch and stateful point cursors

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a stateful btree point search cursor that uses previous search history

and exponential search algorithm to optimize point search performance

- Add a batching LSM btree point search cursor to perform point searches for

a batch of keys. Search states are cleared after each batch.

Change-Id: I0b0ade723895bcd71463df7a9703fe78a238e6c7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5484

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 24 more files in changeset.
[NO ISSUE][COMP] Prohibit anonymous type use by CREATE DATASET

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- CREATE DATASET should not be able to use anonymous types

Change-Id: I0dcdd2684180b054ff48f0f0a8ccd6041e8bd00c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5664

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

  1. … 7 more files in changeset.
[NO ISSUE][MTD] Restrict Metadata Catalog Update to Synonym Dataset

- Only allow Metadata.Synonym datasets to be added to

an existing cluster. This is done to detect the case

when another metadata dataset that is expected to be

there but is not found.

Change-Id: I142d2537816cc302ef9e02e7faf18af03cb72350

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5604

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[NO ISSUE][COMP] CREATE DATASET with inline type definition

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Support inline type specification in CREATE DATASET statement:

CREATE DATASET customer(cid INTEGER NOT NULL, first_name STRING) ...

- Add testcases

Change-Id: Ic88ccacd016a6144f1b05a2f79a07b2a980d9c9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5524

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

    • -3
    • +18
    ./asterix/metadata/utils/DatasetUtil.java
    • -0
    • +1
    ./asterix/metadata/utils/MetadataUtil.java
  1. … 14 more files in changeset.
[ASTERIXDB-2697]: Implementing AWS s3 as external data source

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Added an external reader for AWS S3.

- Updated query translator to include the WITH

parameters into the dataset details when creating

an external dataset.

- Added test case for AWS S3 using an S3 mocking server

to avoid using real credentials.

Change-Id: I71d89116c0bb404c9621b16f21a6a31cbf7bb7f6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5025

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -11
    • +11
    ./asterix/metadata/feeds/FeedMetadataUtil.java
  1. … 45 more files in changeset.
[NO ISSUE][COMP][RT] External function improvements

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Support multipart external identifiers in CREATE FUNCTION

- Add preliminary code for Python UDFs

- Some refactoring in external function framework

- Add method to get IServiceContext from IEvaluatorContext

Change-Id: I7ec91f5be2efa8409cda3a3c13f5e8b4de3e75e8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5244

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -11
    • +7
    ./asterix/metadata/entities/Function.java
  1. … 30 more files in changeset.
[NO ISSUE][COMP] Non-deterministic external functions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add compiler support for non-deterministic external functions

- Constant fold only those external functions that are

declared as deterministic and are implemented in Java

(grammar default is non-deterministic)

- Add testcase

Change-Id: Iae7839cb7f6c21f8980867e31de78887a8e9e801

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5205

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -25
    • +5
    ./asterix/metadata/entities/Function.java
  1. … 23 more files in changeset.
[NO ISSUE] Cleanup function metadata

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Introduce AbstractDatatypeTupleTranslator to be used

as a base class by metadata entity translators that need

to write data types

2. Align type information stored in user defined function

metadata (return and parameter types) with how types are

stored for user defined types

3. Remove 'deterministic' and 'nullcall' modifiers from metadata

for inline functions because 'nullcall' is not applicable and

'deterministic' is supposed to be inferred by the compiler

4. Introduce FunctionLanguage enum to replace string constants

Change-Id: I2054dc7b13f31f33e57dca6da6d44ab95ac9bb4f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4924

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -53
    • +61
    ./asterix/metadata/entities/Function.java
  1. … 26 more files in changeset.
[ASTERIXDB-2693] CREATE FUNCTION/ADAPTER ddl

- don't map UDFs on upload with descriptor

- UDFs get mapped via CREATE FUNCTION/ADAPTER after UDF payload is

uploaded

- add Library metadata catalog

Change-Id: Ic3c1e98c183cd214eea3e4fee24b2b7c46166b32

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3609

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -4
    • +31
    ./asterix/metadata/MetadataNode.java
    • -0
    • +13
    ./asterix/metadata/entities/DatasourceAdapter.java
    • -10
    • +55
    ./asterix/metadata/entities/Function.java
    • -0
    • +10
    ./asterix/metadata/lock/MetadataLockKey.java
    • -1
    • +35
    ./asterix/metadata/utils/MetadataLockUtil.java
  1. … 187 more files in changeset.
[NO ISSUE][COMP] Drop synonyms in MetadataNode

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Move code that drops synonyms when their dataverse

is deleted from QueryTranslator to MetadataNode

Change-Id: I97233a6c264d058166d5029d396bcfc539b4d35b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4724

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 1 more file in changeset.
[NO ISSUE][COMP] Minor MetadataProvider refactoring

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add MetadataProvider.findSynonym() method

Change-Id: Idc7e0fa83da05b2892acbd786154acea7f2f4cd0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4703

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>