Clone Tools
  • last updated 16 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2741][EXT] Support include/exclude filters for S3 external datasets

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added support for include/exclude filters.

- Added test cases for include/exclude.

Change-Id: I21dbc5c08456f95a2cad1a8fd9147472fad15376

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6604

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 89 more files in changeset.
[NO ISSUE][API][SQL] retrieve request and statement information

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Get user agent and remote address from request reference.

- Extract statement properties that are determined during parsing.

Change-Id: I26dab3ccee21293963496f38abe44b9a653a35e4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5783

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -2
    • +5
    ./message/ExecuteStatementRequestMessage.java
    • -0
    • +10
    ./message/ExecuteStatementResponseMessage.java
    • -9
    • +18
    ./translator/RequestParameters.java
  1. … 12 more files in changeset.
[NO ISSUE] Improve handling of udf parameter/return types

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Prohibit parameter/return type specification for inline udfs

- Prohibit parameter/return type quantifier (?) for external udfs

- Allow external udfs to refer to types from other dataverses

- Do not store type quantifiers in udf metadata

- Align names of anonymous types automatically created by

QueryTranslator for datasets and udfs

- Reserve user-defined type names starting from '$' for system use

- Add testcases

- Prohibit UdfServlet operations if cluster is not active

- Return correct path from IOManager.getWorkspacePath()

- Test framework: support multiple commands in .lib.sqlpp files

- Remove CC files when deleting old instance data in

AsterixHyracksIntegrationUtil

Change-Id: I09bdfd8c7b26750d12339034d2143a16102d1212

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6545

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -66
    • +90
    ./translator/QueryTranslator.java
  1. … 35 more files in changeset.
Merge commit 'a548307b9f' from 'gerrit/mad-hatter'

Change-Id: I57eb5d96af8f227e5a9550381418f9468eb400d8

  1. … 8 more files in changeset.
[ASTERIXDB-2722][EXT] Require minimum permissions + improve error reporting

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Creating an external dataset requires S3:ListBucket permission.

- Querying an external dataset requires S3:ListBucket and

S3:GetObject permissions.

- Improved error reporting.

- Added test cases for the above mentioned items (where applicable).

Change-Id: Idc266cf63b8f92a07af7341118d2636673913160

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6463

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 18 more files in changeset.
[ASTERIXDB-2738][STO] Create Mask File Before Merge Operations

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before starting a merge operation, create a mask file

(.mask_C_startSeq_endSeq) for the merged component to

indicate that this component isn't valid yet.

- On the merge operation successful completion, delete the

merged component mask file.

- In the case of any unexpected failure during the merge

operation, all files of the failed merged component will

be deleted on node startup/shutdown, including the mask

file.

- Halt on any IO opeartion failure.

- Add a test case that ensures only masked merged components

are deleted but not the original components that were

supposed to be merged.

Change-Id: I476dd3be5e75468e83044b3aaf0f6c2d8beadf1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6425

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Luo Chen <cluo8@uci.edu>

  1. … 6 more files in changeset.
Merge commit '0f407d9224' from 'gerrit/mad-hatter'

Change-Id: I89eaf1bace87130d8d35ec0ce8e0c263ac87b59a

  1. … 1 more file in changeset.
Merge commit '594e7eca21' from 'gerrit/mad-hatter'

Change-Id: I975228925366adc94f136bbbb014cb6817bf180e

  1. … 5 more files in changeset.
Merge commit '6904d36' from 'gerrit/mad-hatter'

Change-Id: Ib118bb2d1b18a93412015b1b652684c3fbb11e52

    • -18
    • +24
    ./translator/QueryTranslator.java
  1. … 2 more files in changeset.
[ASTERIXDB-2725][COMP] Check if dataverse exists before continuing the DDL execution

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added dataverse check at multiple places before proceeding

with the execution of the DDL.

- Added a test cases for each newly added check.

Change-Id: I8d54b0b10741d65df91cbb4b7596d363f7ddbd97

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6203

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 14 more files in changeset.
[NO ISSUE][EXT] Minor refactoring of external datasets details validation

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Minor refactoring of external datasets details validation.

- Added a missing error message in Hyracks.

Change-Id: I975228925366adc94f136bbbb014cb6817bf180e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6325

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 3 more files in changeset.
Merge commit 'aa7ff7b' from 'gerrit/mad-hatter'

Change-Id: Ibfc874ca45692420fb86038537d10e172894730a

    • -15
    • +23
    ./translator/QueryTranslator.java
  1. … 19 more files in changeset.
[NO ISSUE][COMP] Refactor locking in CREATE DATASET

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactor locking in CREATE DATASET to make it extensible

by products (introduce QueryTranslator.createDatasetBegin())

- Refactor MetadataLockUtil.createDatasetBegin() into two methods

Change-Id: I1b430a5e8f4a3cdf1a360397e3d160f7b8c95d26

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6323

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -22
    • +38
    ./translator/QueryTranslator.java
  1. … 1 more file in changeset.
Merge commit 'f1cd417' from 'gerrit/mad-hatter'

Change-Id: I0f30399a6485708f4b154af09ea36377fe4d3dbf

    • -16
    • +38
    ./translator/QueryTranslator.java
  1. … 10 more files in changeset.
Merge commit '4bf7048' from 'gerrit/mad-hatter'

Change-Id: I06432e8daa1669862d1f23a04efdfec5b1e6d7b5

  1. … 3 more files in changeset.
Merge commit '0c91bd7c564e1bd5cb5d54b137f41a4d9b9db076' into HEAD

Change-Id: Iebc71d2b9797e5f645580ea8ae2b7e149e749a03

    • -27
    • +10
    ./translator/QueryTranslator.java
Merge commit '3e2623cf64827e891ecd8c6dea13d03f5599c6a5' into HEAD

Change-Id: I89091d8bf8575c73d26f497d5fbffe780adbedf9

    • -23
    • +120
    ./translator/QueryTranslator.java
  1. … 8 more files in changeset.
[NO ISSUE][COMP] Introduce QueryTranslator.validateIndexType()

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce QueryTranslator.validateIndexType() that

can be overriden by product extensions to customize

supported index types.

Change-Id: I000ba078579b8e0a96d0219cefad137f41abf791

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6204

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -17
    • +25
    ./translator/QueryTranslator.java
Merge commit 'bc0f7e0' from branch 'gerrit/mad-hatter'

Change-Id: I629850e0f6ac7e1afae3fb5c6e1f1f859b2ead72

  1. … 19 more files in changeset.
[ASTERIXDB-2176] Python UDFs

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Allow UDFs to be shiv-packaged python modules with their

dependencies

- Use pyro for python RPC

- Maven build for pyro package into server

- Remove JObject spatial types

- Simpler conversion from java primitive/standard types and

collections

to JObject equivalents

Change-Id: Ibea23a2e9308132f343d80eff04ede9a235aa021

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -57
    • +0
    ./external/ExternalLibraryClassLoader.java
    • -134
    • +6
    ./external/ExternalLibraryUtils.java
    • -3
    • +1
    ./nc/task/ExternalLibrarySetupTask.java
  1. … 101 more files in changeset.
[ASTERIXDB-2715][STO] Dynamic Memory Component Architecture

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Introduce a dynamic memory component architecture that uses a global

virtual buffer cache to manage the write memory for all LSM-trees.

- When the overall write memory is nearly full, we flush a dataset

partition at a time using a round-robin way. Additionally, we allow

users to configure the maximum size of filtered memory components

to provide better pruning capability.

- Clean up legacy code for statically allocating write memory to each

dataset.

- Remove the following parameters:

storage.metadata.memorycomponent.numpages

storage.max.active.writable.datasets

- Add the following parameters:

storage.memorycomponent.flush.threshold (default: 0.9)

storage.filtered.memorycomponent.max.size (default 0)

Change-Id: Ia6a0f4de020acd7af89ef630322526c4be5076e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5824

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 44 more files in changeset.
[NO ISSUE] Fix UDF Metadata Transactions

- Move UDF Metadata Transactions to be initiated from the CC

directly

- Remove old UDF initialization code

- Make DROP DATAVERSE remove libraries

- Make NCs properly read libraries on init with multipart dataverse

names

Change-Id: Ibda23a2e8308937f343d80eff04ede9a235da0d2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4923

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -198
    • +27
    ./external/ExternalLibraryUtils.java
    • -1
    • +1
    ./nc/task/ExternalLibrarySetupTask.java
  1. … 40 more files in changeset.
[ASTERIXDB-2713][EXT] CSV & TSV support for external dataset p2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- validate adapter configuration

- ignore empty lines in CSV/TSV files

- require "header" parameter for CSV/TSV formats

- make some parameters case-insensitive

- few fixes and clean-ups

Change-Id: I2f523de0d482a358ada0c27236ff24616ad0d7da

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5848

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 37 more files in changeset.
[NO ISSUE][COMP] Extensible item type check in CREATE DATASET

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce QueryTranslator.validateDatasetItemType().

By overriding this method product extensions can further

restrict types that are allowed for dataset creation.

Change-Id: I426430707766b7b6049abd2687bee69c011335fb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5785

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

    • -12
    • +15
    ./translator/QueryTranslator.java
[ASTERIXDB-2713][EXT] Add CSV & TSV support for external dataset

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Add CSV support for external dataset.

- support S3

- add boolean parser to Hyracks

Change-Id: Id1790fa73461e9f4a5fb443c51c1905ac588cee6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5743

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 48 more files in changeset.
[NO ISSUE][COMP] Prohibit anonymous type use by CREATE DATASET

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- CREATE DATASET should not be able to use anonymous types

Change-Id: I0dcdd2684180b054ff48f0f0a8ccd6041e8bd00c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5664

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

  1. … 8 more files in changeset.
[NO ISSUE][COMP] CREATE INDEX with Geometry datatype

Details:

- Support geometry datatype for RTree Index:

CREATE INDEX geomIndex ON geoDataset(geometry) TYPE rtree;

- Enable ST functions for rtree with geometry datatype

- Add testcases

Change-Id: I8f4a82c43b950fc3573cae5aa7c0782b475f962d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5584

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Mehnaz Tabassum Mahin <mmahi004@ucr.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 48 more files in changeset.
[NO ISSUE][COMP] Minor refactoring in QueryTranslator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace QueryTranslator method

validateExternalDatasetRequirements()

with createExternalDatasetProperties()

Change-Id: Ifbf54dbcd560a3c987f6c42a387d1fa0f0e41e0a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5566

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

    • -27
    • +10
    ./translator/QueryTranslator.java
[NO ISSUE][COMP] CREATE DATASET with inline type definition

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Support inline type specification in CREATE DATASET statement:

CREATE DATASET customer(cid INTEGER NOT NULL, first_name STRING) ...

- Add testcases

Change-Id: Ic88ccacd016a6144f1b05a2f79a07b2a980d9c9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5524

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

    • -22
    • +125
    ./translator/QueryTranslator.java
  1. … 17 more files in changeset.
[ASTERIXDB-2697]: Implementing AWS s3 as external data source

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Added an external reader for AWS S3.

- Updated query translator to include the WITH

parameters into the dataset details when creating

an external dataset.

- Added test case for AWS S3 using an S3 mocking server

to avoid using real credentials.

Change-Id: I71d89116c0bb404c9621b16f21a6a31cbf7bb7f6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5025

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 51 more files in changeset.