Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][COMP][RT] New library deployment model

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Deploy external libraries using Hyracks jobs

- Library manager is no longer available on CC

- Add IServlet.init() which is invoked during web server startup

- External libraries can no longer provide implementations of

IExternalDataSourceFactory because external libraries are

not available on CC

- Added testcase for an external adapter

Change-Id: If64f99f6a15b81b1e426239bde63360f5ef57059

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6863

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -0
    • +140
    ./external/ExternalLibraryUtil.java
    • -55
    • +0
    ./external/ExternalLibraryUtils.java
    • -50
    • +0
    ./nc/task/ExternalLibrarySetupTask.java
    • -2
    • +0
    ./replication/NcLifecycleCoordinator.java
    • -14
    • +217
    ./translator/QueryTranslator.java
  1. … 85 more files in changeset.
Merge commit 'c25c36d1b0' from 'gerrit/mad-hatter'

Change-Id: I8a78950cb81f0a29ceae94fa439eec700c17d06d

  1. … 11 more files in changeset.
[NO ISSUE] Pass item type when creating external dataset properties

Change-Id: I8b6693d229a36358eda584ddfb95d1819f76ce5c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6963

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[ASTERIXDB-2751][EXT] Issue a warning if external dataset configuration returns no files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Issue a warning when creating an external dataset if the dataset

configuration returns no files.

- Issue a warning when querying an external dataset if no files

are returned.

Change-Id: I0b2be49bf3c69c3e055d942a5c3f0106b3069d52

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6843

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 53 more files in changeset.
Merge branch 'gerrit/mad-hatter'

Change-Id: I5cd06cc3cd69068faecced6a2b1cc558eecd0784

  1. … 4 more files in changeset.
[NO ISSUE][MISC] IRetryPolicy requires non-null throwable

Change-Id: I4835266435fc31f5973f4312e6ddcbdb85d1e839

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6923

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 1 more file in changeset.
[NO ISSUE][COMP] Improve error reporting in TypeTranslator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add error code and source location to exceptions

raised by TypeTranslator

- Pass dataset item type when calling

QueryTranslator.validateExternalDatasetProperties()

Change-Id: I2b62c162f37e72245cdbf70fe9f8307bbf9b2bf1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6885

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
[ASTERIXDB-2730][STO] Optimize flush in GVBC

- user model changes: no

- storage format changes: no

- interface changes: yes. Introduce a new storage property:

storage.memorycomponent.max.concurrent.flushes (default 0)

Details:

- Introduce a new storage property to allow concurrent flushes

by GVBC. The default value is 0, which means that the flush concurency

will be the same as the number of NC partitions.

- Move cleaning up of a memory component out of the synchronization block

on op tracker because this may take a relatively long time (a full scan over

all GVBC pages).

- Introduce a minor fix to make sure the memory component is unwritable

before requesting flushing it by GVBC

Change-Id: Id8867fa3ac65da319723b804cc1e39dc8eb6bde5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6624

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 11 more files in changeset.
Merge commit '0163af68d0' from 'gerrit/mad-hatter'

Change-Id: I5b53384961e42fe070d2e1b885e762428f6f8c45

  1. … 4 more files in changeset.
Merge branch 'gerrit/mad-hatter'

Change-Id: I85867c451c84aaa94f008c6b718ab7805983c503

  1. … 5 more files in changeset.
[ASTERIXDB-2741][EXT] Support include/exclude filters for S3 external datasets

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added support for include/exclude filters.

- Added test cases for include/exclude.

Change-Id: I21dbc5c08456f95a2cad1a8fd9147472fad15376

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6604

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 89 more files in changeset.
[NO ISSUE][API][SQL] retrieve request and statement information

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Get user agent and remote address from request reference.

- Extract statement properties that are determined during parsing.

Change-Id: I26dab3ccee21293963496f38abe44b9a653a35e4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5783

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -2
    • +5
    ./message/ExecuteStatementRequestMessage.java
    • -0
    • +10
    ./message/ExecuteStatementResponseMessage.java
    • -9
    • +18
    ./translator/RequestParameters.java
  1. … 12 more files in changeset.
[NO ISSUE] Improve handling of udf parameter/return types

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Prohibit parameter/return type specification for inline udfs

- Prohibit parameter/return type quantifier (?) for external udfs

- Allow external udfs to refer to types from other dataverses

- Do not store type quantifiers in udf metadata

- Align names of anonymous types automatically created by

QueryTranslator for datasets and udfs

- Reserve user-defined type names starting from '$' for system use

- Add testcases

- Prohibit UdfServlet operations if cluster is not active

- Return correct path from IOManager.getWorkspacePath()

- Test framework: support multiple commands in .lib.sqlpp files

- Remove CC files when deleting old instance data in

AsterixHyracksIntegrationUtil

Change-Id: I09bdfd8c7b26750d12339034d2143a16102d1212

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6545

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -66
    • +90
    ./translator/QueryTranslator.java
  1. … 35 more files in changeset.
Merge commit 'a548307b9f' from 'gerrit/mad-hatter'

Change-Id: I57eb5d96af8f227e5a9550381418f9468eb400d8

  1. … 8 more files in changeset.
[ASTERIXDB-2722][EXT] Require minimum permissions + improve error reporting

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Creating an external dataset requires S3:ListBucket permission.

- Querying an external dataset requires S3:ListBucket and

S3:GetObject permissions.

- Improved error reporting.

- Added test cases for the above mentioned items (where applicable).

Change-Id: Idc266cf63b8f92a07af7341118d2636673913160

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6463

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 18 more files in changeset.
[ASTERIXDB-2738][STO] Create Mask File Before Merge Operations

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before starting a merge operation, create a mask file

(.mask_C_startSeq_endSeq) for the merged component to

indicate that this component isn't valid yet.

- On the merge operation successful completion, delete the

merged component mask file.

- In the case of any unexpected failure during the merge

operation, all files of the failed merged component will

be deleted on node startup/shutdown, including the mask

file.

- Halt on any IO opeartion failure.

- Add a test case that ensures only masked merged components

are deleted but not the original components that were

supposed to be merged.

Change-Id: I476dd3be5e75468e83044b3aaf0f6c2d8beadf1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6425

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Luo Chen <cluo8@uci.edu>

  1. … 6 more files in changeset.
Merge commit '0f407d9224' from 'gerrit/mad-hatter'

Change-Id: I89eaf1bace87130d8d35ec0ce8e0c263ac87b59a

  1. … 1 more file in changeset.
Merge commit '594e7eca21' from 'gerrit/mad-hatter'

Change-Id: I975228925366adc94f136bbbb014cb6817bf180e

  1. … 5 more files in changeset.
Merge commit '6904d36' from 'gerrit/mad-hatter'

Change-Id: Ib118bb2d1b18a93412015b1b652684c3fbb11e52

    • -18
    • +24
    ./translator/QueryTranslator.java
  1. … 2 more files in changeset.
[ASTERIXDB-2725][COMP] Check if dataverse exists before continuing the DDL execution

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added dataverse check at multiple places before proceeding

with the execution of the DDL.

- Added a test cases for each newly added check.

Change-Id: I8d54b0b10741d65df91cbb4b7596d363f7ddbd97

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6203

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 14 more files in changeset.
[NO ISSUE][EXT] Minor refactoring of external datasets details validation

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Minor refactoring of external datasets details validation.

- Added a missing error message in Hyracks.

Change-Id: I975228925366adc94f136bbbb014cb6817bf180e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6325

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 3 more files in changeset.
Merge commit 'aa7ff7b' from 'gerrit/mad-hatter'

Change-Id: Ibfc874ca45692420fb86038537d10e172894730a

    • -15
    • +23
    ./translator/QueryTranslator.java
  1. … 19 more files in changeset.
[NO ISSUE][COMP] Refactor locking in CREATE DATASET

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactor locking in CREATE DATASET to make it extensible

by products (introduce QueryTranslator.createDatasetBegin())

- Refactor MetadataLockUtil.createDatasetBegin() into two methods

Change-Id: I1b430a5e8f4a3cdf1a360397e3d160f7b8c95d26

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6323

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -22
    • +38
    ./translator/QueryTranslator.java
  1. … 1 more file in changeset.
Merge commit 'f1cd417' from 'gerrit/mad-hatter'

Change-Id: I0f30399a6485708f4b154af09ea36377fe4d3dbf

    • -16
    • +38
    ./translator/QueryTranslator.java
  1. … 10 more files in changeset.
Merge commit '4bf7048' from 'gerrit/mad-hatter'

Change-Id: I06432e8daa1669862d1f23a04efdfec5b1e6d7b5

  1. … 3 more files in changeset.
Merge commit '0c91bd7c564e1bd5cb5d54b137f41a4d9b9db076' into HEAD

Change-Id: Iebc71d2b9797e5f645580ea8ae2b7e149e749a03

    • -27
    • +10
    ./translator/QueryTranslator.java
Merge commit '3e2623cf64827e891ecd8c6dea13d03f5599c6a5' into HEAD

Change-Id: I89091d8bf8575c73d26f497d5fbffe780adbedf9

    • -23
    • +120
    ./translator/QueryTranslator.java
  1. … 8 more files in changeset.
[NO ISSUE][COMP] Introduce QueryTranslator.validateIndexType()

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce QueryTranslator.validateIndexType() that

can be overriden by product extensions to customize

supported index types.

Change-Id: I000ba078579b8e0a96d0219cefad137f41abf791

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6204

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -17
    • +25
    ./translator/QueryTranslator.java
Merge commit 'bc0f7e0' from branch 'gerrit/mad-hatter'

Change-Id: I629850e0f6ac7e1afae3fb5c6e1f1f859b2ead72

  1. … 19 more files in changeset.
[ASTERIXDB-2176] Python UDFs

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Allow UDFs to be shiv-packaged python modules with their

dependencies

- Use pyro for python RPC

- Maven build for pyro package into server

- Remove JObject spatial types

- Simpler conversion from java primitive/standard types and

collections

to JObject equivalents

Change-Id: Ibea23a2e9308132f343d80eff04ede9a235aa021

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -57
    • +0
    ./external/ExternalLibraryClassLoader.java
    • -134
    • +6
    ./external/ExternalLibraryUtils.java
    • -3
    • +1
    ./nc/task/ExternalLibrarySetupTask.java
  1. … 101 more files in changeset.