Clone Tools
  • last updated 22 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2176][RT] Improved Python IPC

- use msgpack serialization over tcp loopback

- convert directly from ADM binary format to msg

Change-Id: I5cbbc367944b489aee651ea050e74990dcf65521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6883

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +1
    ./asterix/common/exceptions/ErrorCode.java
    • -0
    • +127
    ./asterix/external/ipc/ExternalFunctionResultRouter.java
  1. … 57 more files in changeset.
[NO ISSUE][COMP] Improve error message when function not found

Details:

- Remove arity from the error message raised when

the function is not found

- Also remove "null." prefix from that message if

the function call did not specify a dataverse

Change-Id: I384ca2af87078e72aab23dccd3651a8edf02a3b4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/7307

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 5 more files in changeset.
[ASTERIXDB-2760][COMP] Improve function call resolution

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactor function call resolution and perform it earlier,

before GROUP BY expressions are substituted

- Consolidate gby-expr testcases into a single one

- Introduce QueryTranslator.handleDeclareFunctionStatement()

- Rename existing SqlppCaseExpressionVisitor to

SqlppCaseAggregateExtractionVisitor

- Move CASE expression rewriting code from

SqlppBuiltinFunctionRewriteVisitor into a separate visitor:

SqlppCaseExpressionVisitor

- Remove function name normalization code from SqlppAstPrintVisitor

and updated existing reference results

Change-Id: I527ffbff487b8534f66ca27d48788e96c65aec9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/7264

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +1
    ./asterix/common/exceptions/ErrorCode.java
  1. … 1162 more files in changeset.
[NO ISSUE][COMP] Refactor drop dataverse and drop function

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Refactor feed handling when processing drop dataverse

and drop function

- Move feed dependency checking from QueryTranslator

to MetadataNode to align with other entities

Change-Id: I493dfffd77c596bb3485eccb00b417d40f47b647

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/7066

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

    • -1
    • +0
    ./asterix/common/exceptions/ErrorCode.java
  1. … 4 more files in changeset.
[NO ISSUE][COMP][RT] New library deployment model

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Deploy external libraries using Hyracks jobs

- Library manager is no longer available on CC

- Add IServlet.init() which is invoked during web server startup

- External libraries can no longer provide implementations of

IExternalDataSourceFactory because external libraries are

not available on CC

- Added testcase for an external adapter

Change-Id: If64f99f6a15b81b1e426239bde63360f5ef57059

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6863

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -6
    • +0
    ./asterix/common/api/IApplicationContext.java
    • -1
    • +0
    ./asterix/common/exceptions/ErrorCode.java
    • -1
    • +2
    ./asterix/common/library/ILibrary.java
    • -8
    • +13
    ./asterix/common/library/LibraryDescriptor.java
  1. … 88 more files in changeset.
Merge commit 'c25c36d1b0' from 'gerrit/mad-hatter'

Change-Id: I8a78950cb81f0a29ceae94fa439eec700c17d06d

    • -7
    • +8
    ./asterix/common/exceptions/ErrorCode.java
  1. … 11 more files in changeset.
[ASTERIXDB-2751][EXT] Issue a warning if external dataset configuration returns no files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Issue a warning when creating an external dataset if the dataset

configuration returns no files.

- Issue a warning when querying an external dataset if no files

are returned.

Change-Id: I0b2be49bf3c69c3e055d942a5c3f0106b3069d52

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6843

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +1
    ./asterix/common/exceptions/ErrorCode.java
    • -0
    • +46
    ./asterix/common/exceptions/NoOpWarningCollector.java
  1. … 51 more files in changeset.
[ASTERIXDB-2730][STO] Optimize flush in GVBC

- user model changes: no

- storage format changes: no

- interface changes: yes. Introduce a new storage property:

storage.memorycomponent.max.concurrent.flushes (default 0)

Details:

- Introduce a new storage property to allow concurrent flushes

by GVBC. The default value is 0, which means that the flush concurency

will be the same as the number of NC partitions.

- Move cleaning up of a memory component out of the synchronization block

on op tracker because this may take a relatively long time (a full scan over

all GVBC pages).

- Introduce a minor fix to make sure the memory component is unwritable

before requesting flushing it by GVBC

Change-Id: Id8867fa3ac65da319723b804cc1e39dc8eb6bde5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6624

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 10 more files in changeset.
Merge commit '0163af68d0' from 'gerrit/mad-hatter'

Change-Id: I5b53384961e42fe070d2e1b885e762428f6f8c45

    • -7
    • +11
    ./asterix/common/exceptions/ErrorCode.java
  1. … 4 more files in changeset.
[ASTERIXDB-2741][EXT] Support include/exclude filters for S3 external datasets

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Added support for include/exclude filters.

- Added test cases for include/exclude.

Change-Id: I21dbc5c08456f95a2cad1a8fd9147472fad15376

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6604

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +4
    ./asterix/common/exceptions/ErrorCode.java
  1. … 89 more files in changeset.
[NO ISSUE][API][SQL] retrieve request and statement information

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Get user agent and remote address from request reference.

- Extract statement properties that are determined during parsing.

Change-Id: I26dab3ccee21293963496f38abe44b9a653a35e4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5783

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +14
    ./asterix/common/api/IRequestReference.java
  1. … 15 more files in changeset.
[NO ISSUE][COMP] Implement FILTER clause in aggregates

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Implement FILTER subclause in SQL aggregate function calls

- Add testcases and documentation

Change-Id: Ie346a09db51907757df78575d2145085b1d89782

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6424

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -0
    • +1
    ./asterix/common/exceptions/ErrorCode.java
  1. … 77 more files in changeset.
Merge commit 'a548307b9f' from 'gerrit/mad-hatter'

Change-Id: I57eb5d96af8f227e5a9550381418f9468eb400d8

    • -6
    • +8
    ./asterix/common/exceptions/ErrorCode.java
  1. … 8 more files in changeset.
[ASTERIXDB-2722][EXT] Require minimum permissions + improve error reporting

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Creating an external dataset requires S3:ListBucket permission.

- Querying an external dataset requires S3:ListBucket and

S3:GetObject permissions.

- Improved error reporting.

- Added test cases for the above mentioned items (where applicable).

Change-Id: Idc266cf63b8f92a07af7341118d2636673913160

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6463

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -0
    • +2
    ./asterix/common/exceptions/ErrorCode.java
  1. … 18 more files in changeset.
Merge branch 'gerrit/mad-hatter'

Change-Id: I1ed2ad00b1fb6ef5fe70e7a1d6a753d8da59e269

    • -0
    • +14
    ./asterix/common/storage/ResourceReference.java
[ASTERIXDB-2738][STO] Create Mask File Before Merge Operations

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before starting a merge operation, create a mask file

(.mask_C_startSeq_endSeq) for the merged component to

indicate that this component isn't valid yet.

- On the merge operation successful completion, delete the

merged component mask file.

- In the case of any unexpected failure during the merge

operation, all files of the failed merged component will

be deleted on node startup/shutdown, including the mask

file.

- Halt on any IO opeartion failure.

- Add a test case that ensures only masked merged components

are deleted but not the original components that were

supposed to be merged.

Change-Id: I476dd3be5e75468e83044b3aaf0f6c2d8beadf1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6425

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Luo Chen <cluo8@uci.edu>

    • -0
    • +14
    ./asterix/common/storage/ResourceReference.java
  1. … 5 more files in changeset.
Merge commit '594e7eca21' from 'gerrit/mad-hatter'

Change-Id: I975228925366adc94f136bbbb014cb6817bf180e

    • -4
    • +4
    ./asterix/common/exceptions/ErrorCode.java
  1. … 5 more files in changeset.
[ASTERIXDB-2730][STO] Do not wait on flushing metadata indexes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure GVBC does not wait on flushing metadata indexes because

metadata indexes support full ACID transactions and waiting on them

can lead to deadlocks.

- Add more log messages to GVBC.

Change-Id: Iba4020d8584768940176023ac4ab9135cff7b7e6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6345

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Merge commit '6904d36' from 'gerrit/mad-hatter'

Change-Id: Ib118bb2d1b18a93412015b1b652684c3fbb11e52

  1. … 2 more files in changeset.
[NO ISSUE][EXT] Minor refactoring of external datasets details validation

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Minor refactoring of external datasets details validation.

- Added a missing error message in Hyracks.

Change-Id: I975228925366adc94f136bbbb014cb6817bf180e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6325

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -1
    • +2
    ./asterix/common/exceptions/ErrorCode.java
  1. … 3 more files in changeset.
Merge commit 'aa7ff7b' from 'gerrit/mad-hatter'

Change-Id: Ibfc874ca45692420fb86038537d10e172894730a

    • -3
    • +3
    ./asterix/common/exceptions/ErrorCode.java
  1. … 19 more files in changeset.
Merge commit 'f1cd417' from 'gerrit/mad-hatter'

Change-Id: I0f30399a6485708f4b154af09ea36377fe4d3dbf

    • -2
    • +4
    ./asterix/common/exceptions/ErrorCode.java
  1. … 10 more files in changeset.
[ASTERIXDB-2730][STO] Avoid Double Flushes in GVBC

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure GVBC only flushes a primary index if it has a modified

memory component and there is no pending flush request.

Change-Id: Ib4c3c632c43d83c5e60960c2cdcce54f1216b851

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6305

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2730][STO] Safely Update Flush Pointer in GVBC

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure at least one primary index is registered before

updating the flush pointer in GlobalVirtualBufferCache.

Otherwise, reset the pointer to zero.

Change-Id: Ib0b3c9c7cfcf7b6a3c2f404f42d1d4802cf12b28

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6285

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

[NO ISSUE][RT] Collect Tuple Stats in External Scan

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Report number of procssed tuples in external scan operator.

- Add test case.

Change-Id: I5dda25f1fc53581dcc5663f2516e79b9b66fd0a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6224

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 12 more files in changeset.
Merge commit 'bc0f7e0' from branch 'gerrit/mad-hatter'

Change-Id: I629850e0f6ac7e1afae3fb5c6e1f1f859b2ead72

  1. … 19 more files in changeset.
[ASTERIXDB-2720][EXT] Support arrays of objects for external dataset with json format

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Currently, the json format that the external dataset support is

the extended version of jsonl. The format is consecutive json objects

(normally one object per line). Arrays of objects should

be supported as valid values as well.

Change-Id: I59462c274ec3186cf35ba4ce222a9ffc4056e00a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6063

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -3
    • +3
    ./asterix/common/exceptions/ErrorCode.java
  1. … 15 more files in changeset.
[ASTERIXDB-2176] Python UDFs

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Allow UDFs to be shiv-packaged python modules with their

dependencies

- Use pyro for python RPC

- Maven build for pyro package into server

- Remove JObject spatial types

- Simpler conversion from java primitive/standard types and

collections

to JObject equivalents

Change-Id: Ibea23a2e9308132f343d80eff04ede9a235aa021

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -1
    • +1
    ./asterix/common/config/ExternalProperties.java
    • -0
    • +27
    ./asterix/common/functions/ExternalFunctionLanguage.java
    • -0
    • +30
    ./asterix/common/library/ILibrary.java
    • -36
    • +11
    ./asterix/common/library/ILibraryManager.java
    • -0
    • +61
    ./asterix/common/library/LibraryDescriptor.java
  1. … 104 more files in changeset.
[ASTERIXDB-2715][STO] Dynamic Memory Component Architecture

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Introduce a dynamic memory component architecture that uses a global

virtual buffer cache to manage the write memory for all LSM-trees.

- When the overall write memory is nearly full, we flush a dataset

partition at a time using a round-robin way. Additionally, we allow

users to configure the maximum size of filtered memory components

to provide better pruning capability.

- Clean up legacy code for statically allocating write memory to each

dataset.

- Remove the following parameters:

storage.metadata.memorycomponent.numpages

storage.max.active.writable.datasets

- Add the following parameters:

storage.memorycomponent.flush.threshold (default: 0.9)

storage.filtered.memorycomponent.max.size (default 0)

Change-Id: Ia6a0f4de020acd7af89ef630322526c4be5076e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5824

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -28
    • +12
    ./asterix/common/config/StorageProperties.java
    • -12
    • +2
    ./asterix/common/context/DatasetInfo.java
    • -154
    • +0
    ./asterix/common/context/DatasetMemoryManager.java
    • -0
    • +485
    ./asterix/common/context/GlobalVirtualBufferCache.java
  1. … 36 more files in changeset.
[NO ISSUE] Fix UDF Metadata Transactions

- Move UDF Metadata Transactions to be initiated from the CC

directly

- Remove old UDF initialization code

- Make DROP DATAVERSE remove libraries

- Make NCs properly read libraries on init with multipart dataverse

names

Change-Id: Ibda23a2e8308937f343d80eff04ede9a235da0d2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4923

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +1
    ./asterix/common/exceptions/ErrorCode.java
  1. … 45 more files in changeset.