Clone Tools
  • last updated 12 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][COMP][RT] New library deployment model

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Deploy external libraries using Hyracks jobs

- Library manager is no longer available on CC

- Add IServlet.init() which is invoked during web server startup

- External libraries can no longer provide implementations of

IExternalDataSourceFactory because external libraries are

not available on CC

- Added testcase for an external adapter

Change-Id: If64f99f6a15b81b1e426239bde63360f5ef57059

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6863

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 93 more files in changeset.
[NO ISSUE][EXT] Copy JSON object data to buffer only when reading JSON object has started

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Copy JSON object data to buffer only when reading JSON object has started since the JSON

object could be a nested object (i.e. leading comma character and spaces should not be copied over)

Change-Id: Iddac6d5c3926367770ffd31714c54361d99b4268

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/7043

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +1
    ./external-dataset/aws/s3/json/json/external_dataset.007.adm
  1. … 7 more files in changeset.
[ASTERIXDB-2751][EXT] Issue a warning if external dataset configuration returns no files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Issue a warning when creating an external dataset if the dataset

configuration returns no files.

- Issue a warning when querying an external dataset if no files

are returned.

Change-Id: I0b2be49bf3c69c3e055d942a5c3f0106b3069d52

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6843

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 51 more files in changeset.
[ASTERIXDB-2748][RT] CASE fails if default value is TRUE

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix ArrayIndexOutOfBoundsException in CASE expression

when the default value is TRUE and none of its WHEN

clauses match at runtime

Change-Id: I8d16c200fa4b65008c241aeb8a0ebb86f7b4a6c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6823

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +1
    ./misc/case_09/case_09.1.adm
  1. … 3 more files in changeset.
[ASTERIXDB-2750][COMP] Incorrect result with unnest and join

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix incorrect result when FROM clause contains

simple unnest (",") followed by JOIN

Change-Id: Icfd174ad099aaf24f9553349b3d417d64f9c70b5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6824

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 4 more files in changeset.
[ASTERIXDB-2744][COMP] Fix compiler error during index selection

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fix NullPointerException in BTreeAccessMethod.createKeyVarsAndExprs()

when there is a composite key.

Change-Id: Ie49a5940afa35c3c2af26a7e7f029a1f73e469f3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6743

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 7 more files in changeset.
[ASTERIXDB-2741][EXT] Support include/exclude filters for S3 external datasets

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added support for include/exclude filters.

- Added test cases for include/exclude.

Change-Id: I21dbc5c08456f95a2cad1a8fd9147472fad15376

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6604

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 75 more files in changeset.
[NO ISSUE] Improve handling of udf parameter/return types

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Prohibit parameter/return type specification for inline udfs

- Prohibit parameter/return type quantifier (?) for external udfs

- Allow external udfs to refer to types from other dataverses

- Do not store type quantifiers in udf metadata

- Align names of anonymous types automatically created by

QueryTranslator for datasets and udfs

- Reserve user-defined type names starting from '$' for system use

- Add testcases

- Prohibit UdfServlet operations if cluster is not active

- Return correct path from IOManager.getWorkspacePath()

- Test framework: support multiple commands in .lib.sqlpp files

- Remove CC files when deleting old instance data in

AsterixHyracksIntegrationUtil

Change-Id: I09bdfd8c7b26750d12339034d2143a16102d1212

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6545

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 34 more files in changeset.
[NO ISSUE][COMP] Implement FILTER clause in aggregates

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Implement FILTER subclause in SQL aggregate function calls

- Add testcases and documentation

Change-Id: Ie346a09db51907757df78575d2145085b1d89782

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6424

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +1
    ./aggregate-subclause/agg_filter_01/agg_filter_01.3.adm
    • -0
    • +2
    ./aggregate-subclause/agg_filter_01/agg_filter_01.4.adm
    • -0
    • +3
    ./aggregate-subclause/agg_filter_01/agg_filter_01.5.adm
    • -0
    • +10
    ./aggregate-subclause/agg_filter_01/agg_filter_01.6.adm
    • -0
    • +20
    ./aggregate-subclause/agg_filter_01/agg_filter_01.7.adm
    • -0
    • +2
    ./aggregate-subclause/agg_filter_01/agg_filter_01.8.adm
  1. … 72 more files in changeset.
[NO ISSUE] Added date/time/datetime support to RangeMapBuilder

- user model changes: no

- storage format changes: no

- interface changes: no

details:

-Added date, time, and datetime support to RangeMapBuilder

-Refactored date and datetime parsing

-Added runtime tests for RangeMapBuilder

Change-Id: If1a16a1a5dc4d72ba02ac5717d7e30a62a30bc18

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6403

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +9
    ./sorting/range_hint/range_hint.10.adm
    • -0
    • +9
    ./sorting/range_hint/range_hint.3.adm
    • -0
    • +9
    ./sorting/range_hint/range_hint.4.adm
    • -0
    • +9
    ./sorting/range_hint/range_hint.5.adm
    • -0
    • +9
    ./sorting/range_hint/range_hint.6.adm
    • -0
    • +9
    ./sorting/range_hint/range_hint.7.adm
    • -0
    • +9
    ./sorting/range_hint/range_hint.8.adm
    • -0
    • +9
    ./sorting/range_hint/range_hint.9.adm
  1. … 15 more files in changeset.
[ASTERIXDB-2722][EXT] Require minimum permissions + improve error reporting

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Creating an external dataset requires S3:ListBucket permission.

- Querying an external dataset requires S3:ListBucket and

S3:GetObject permissions.

- Improved error reporting.

- Added test cases for the above mentioned items (where applicable).

Change-Id: Idc266cf63b8f92a07af7341118d2636673913160

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6463

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 18 more files in changeset.
[ASTERIXDB-2732][FUN] Support missing and union type in get_object_field_value

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I2aeacfa2469c8245ce2fd9532434a0a0ecb6eafb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6104

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +3
    ./null-missing/field-access/field-access.2.adm
  1. … 6 more files in changeset.
Merge commit '3a62802' from 'gerrit/mad-hatter'

Change-Id: Ia70a791f6d9fd05854017ef7692adc14a6a5f37f

  1. … 12 more files in changeset.
[ASTERIXDB-2725][COMP] Check if dataverse exists before continuing the DDL execution

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added dataverse check at multiple places before proceeding

with the execution of the DDL.

- Added a test cases for each newly added check.

Change-Id: I8d54b0b10741d65df91cbb4b7596d363f7ddbd97

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6203

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 14 more files in changeset.
[ASTERIXDB-2736][EXT] Ensure retrieving all objects if an S3 bucket has more than 1000 objects

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fixed the AWS reader to handle reading more than 1000

objects coming in a single request.

- Added a test case for the above mentioned item.

Change-Id: Ic7891aa86852e07dfad9ce41de908b34f86bdb42

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6344

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

  1. … 6 more files in changeset.
[ASTERIXDB-2731][API] GET is not supported for the query service servlet

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Remove the restriction of only supporting POST requests in the query service servlet

while maintaining that GET requests cannot change the state of the system

(i.e. no modifying statements can be evaluated when using the GET method).

Change-Id: I989ff489d863697808cdc26898a52b867a9621d6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6303

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +1
    ./api/get-query/get-query.1.adm
    • -0
    • +1
    ./api/get-query/get-query.2.adm
    • -0
    • +1
    ./api/get-query/get-query.3.adm
    • -0
    • +1
    ./api/post-non-query/post-non-query.2.adm
  1. … 13 more files in changeset.
[NO ISSUE][COMP] Rename NOT NULL to NOT UNKNOWN

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Rename NOT NULL type quantifier to NOT UNKNOWN

in CREATE DATASET type specification

- Add IsMissable field to datatype metadata to support

storing all possible NULL/MISSING combinations

Change-Id: Ia7f22ba5d212e3b1824fafb571e1724f2fd25ea9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6306

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 36 more files in changeset.
[ASTERIXDB-2726][EXT] Report line number instead of record number in messages of parsers

- user model changes: no

- storage format changes: no

- interface changes: yes

IRecordReader:

added getLineNumber() to provide line number for parsers and converters.

IRecordConverter:

added configure() to pass the line number supplier to the record converter.

IRecordDataParser:

pass line number supplier from the Reader to the Parser.

Details:

Report line number instead of record number in messages of parsers.

- added getPreviousStreamName() to allow readers to report errors happening on

the previous stream when the underlying stream has already switched to a new one.

- changed the test executor to compare actual warnings issues by a test case with

the expected warnigns properly.

Change-Id: I00508d8eeca4d9bae95f55ab51ecfb0ce2ced6b0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6245

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

    • -0
    • +1
    ./external-dataset/aws/s3/csv-warnings/external_dataset.006.adm
    • -0
    • +2
    ./external-dataset/aws/s3/csv-warnings/external_dataset.007.adm
  1. … 38 more files in changeset.
[NO ISSUE][RT] Collect Tuple Stats in External Scan

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Report number of procssed tuples in external scan operator.

- Add test case.

Change-Id: I5dda25f1fc53581dcc5663f2516e79b9b66fd0a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6224

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 12 more files in changeset.
[ASTERIXDB-2724][EXT] Handle passing empty defintion to external datasets

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When an empty string is passed as an external dataset's definition, then

no prefix is supplied to the AWS client (nothing to filter).

- Added a test case for the above mentioned item.

Change-Id: I500e7afb97aa076b690ef3b98ee83c8f5934a88f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6183

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2720][EXT] Support arrays of objects for external dataset with json format

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Currently, the json format that the external dataset support is

the extended version of jsonl. The format is consecutive json objects

(normally one object per line). Arrays of objects should

be supported as valid values as well.

Change-Id: I59462c274ec3186cf35ba4ce222a9ffc4056e00a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6063

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +50
    ./external-dataset/aws/s3/jsonl/external_dataset.002.adm
  1. … 15 more files in changeset.
[NO ISSUE][COMP] Range scan on a prefix of a composite key

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Fixed an issue with range scan on a prefix of a composite

key not using both bounds to probe the index

Change-Id: I35fb979191b254857211da5c0ec28bae4a370ba8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6124

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 7 more files in changeset.
[ASTERIXDB-2719][EXT] Add .gz compressed test cases for CSV and TSV

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added test cases for compressed data for CSV and TSV.

- Addressed a comment to ensure closing the S3 stream in

case of a failure during streaming the compressed data.

Change-Id: I6160ee5f5aefa4ca60e5a88c3922bde40f4c65ec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6123

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +30
    ./external-dataset/aws/s3/csv/csv/external_dataset.001.adm
    • -0
    • +30
    ./external-dataset/aws/s3/csv/gz/external_dataset.001.adm
    • -0
    • +60
    ./external-dataset/aws/s3/csv/mixed/external_dataset.001.adm
    • -0
    • +30
    ./external-dataset/aws/s3/tsv/gz/external_dataset.001.adm
    • -0
    • +60
    ./external-dataset/aws/s3/tsv/mixed/external_dataset.001.adm
    • -0
    • +30
    ./external-dataset/aws/s3/tsv/tsv/external_dataset.001.adm
  1. … 27 more files in changeset.
[ASTERIXDB-2176] Python UDFs

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Allow UDFs to be shiv-packaged python modules with their

dependencies

- Use pyro for python RPC

- Maven build for pyro package into server

- Remove JObject spatial types

- Simpler conversion from java primitive/standard types and

collections

to JObject equivalents

Change-Id: Ibea23a2e9308132f343d80eff04ede9a235aa021

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +1
    ./external-library/mysentiment/mysentiment.1.adm
    • -0
    • +1
    ./external-library/mysentiment/mysentiment.2.adm
    • -0
    • +1
    ./external-library/mysentiment/mysentiment.3.adm
  1. … 101 more files in changeset.
[ASTERIXDB-2719][EXT] Default external datasets to read .gz and .gzip files as well

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added support to properly stream and read .gz files.

- Added test cases for .json.gz files, and a mix of .json and .gz files.

Change-Id: Ic16044966400954d0cb7c36b99839ad91267ff84

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6043

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +1
    ./external-dataset/aws/s3/json/gz/external_dataset.001.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/gz/external_dataset.002.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/gz/external_dataset.003.adm
    • -0
    • +25
    ./external-dataset/aws/s3/json/gz/external_dataset.004.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/gz/external_dataset.005.adm
    • -0
    • +25
    ./external-dataset/aws/s3/json/gz/external_dataset.006.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/json/external_dataset.001.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/json/external_dataset.002.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/json/external_dataset.003.adm
  1. … 45 more files in changeset.
[NO ISSUE][COMP] Extract SQL aggregates from CASE expressions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Extracts SQL-92 aggregate functions from CASE/IF expressions

into LET clauses, so they can be pushed into GROUPBY subplans

by the optimizer

- Refactor AbstractSqlppExpressionExtractionVisitor to improve

its extensibility

Change-Id: Ia1ae879e845bac5656749966ca57054cbfce6df6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6044

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +2
    ./group-by/gby-case-01/gby-case-01.3.adm
    • -0
    • +2
    ./group-by/gby-case-01/gby-case-01.4.adm
  1. … 15 more files in changeset.
[ASTERIXDB-2715][STO] Dynamic Memory Component Architecture

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Introduce a dynamic memory component architecture that uses a global

virtual buffer cache to manage the write memory for all LSM-trees.

- When the overall write memory is nearly full, we flush a dataset

partition at a time using a round-robin way. Additionally, we allow

users to configure the maximum size of filtered memory components

to provide better pruning capability.

- Clean up legacy code for statically allocating write memory to each

dataset.

- Remove the following parameters:

storage.metadata.memorycomponent.numpages

storage.max.active.writable.datasets

- Add the following parameters:

storage.memorycomponent.flush.threshold (default: 0.9)

storage.filtered.memorycomponent.max.size (default 0)

Change-Id: Ia6a0f4de020acd7af89ef630322526c4be5076e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5824

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 42 more files in changeset.
[ASTERIXDB-2713][EXT] CSV & TSV support for external dataset p4

- user model changes: no

- storage format changes: no

- interface changes: yes

- IValueParser

Details:

- added parameter "null" to allow user to specifiy what string represents

a null value. Only nullable fields whose values match this string will

produce NULL. Otherwise, no NULLs will be produced.

- empty fields:

- for string fields, produce empty string "".

- for non-string fields, issue a warning an ignore the record.

- changed IValueParser to return boolean to allow some implementations

to return true or false instread of throwing an exception.

- added parameter "redact-warnings" to allow user to specifiy if parser

warnings should exclude information like file name.

- changed the Integer and Long parsers to consider spaces properly and

also to handle overflow and underflow.

- changed the boolean parser to align with other parsers that ignore

leading and trailing spaces.

- added test cases.

Change-Id: Ib6aed0095a472510b69cc29a3e444e7de5a2c1ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5963

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +1
    ./external-dataset/aws/s3/csv-warnings/external_dataset.003.adm
    • -0
    • +3
    ./external-dataset/aws/s3/csv-warnings/external_dataset.004.adm
    • -0
    • +3
    ./external-dataset/aws/s3/csv-warnings/external_dataset.005.adm
  1. … 72 more files in changeset.
[ASTERIXDB-2718][EXT] Add more test cases to cover complex JSON formats

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added test cases for JSON single line, multiple lines, arrays, nested

objects, ... etc.

Change-Id: I3cdcd2a3f417b9a045067c2fb50249c3bdf02211

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5923

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +6
    ./external-dataset/aws/s3/csv/000/external_dataset.001.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/000/external_dataset.002.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/000/external_dataset.003.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/000/external_dataset.004.adm
    • -0
    • +10
    ./external-dataset/aws/s3/json/000/external_dataset.005.adm
    • -0
    • +1
    ./external-dataset/aws/s3/json/000/external_dataset.006.adm
    • -0
    • +10
    ./external-dataset/aws/s3/json/000/external_dataset.007.adm
    • -0
    • +6
    ./external-dataset/aws/s3/tsv/000/external_dataset.001.adm
  1. … 30 more files in changeset.
[NO ISSUE][COMP] Fix AbstractIntroduceGroupByCombinerRule

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- AbstractIntroduceGroupByCombinerRule did not always

restore the plan to its original state when failing

Change-Id: I9cada7f324a271ff017c64b69326b9d4a1a32a85

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5925

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +2
    ./group-by/gby-nested-01/gby-nested-01.3.adm
  1. … 5 more files in changeset.