Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2176] Python UDFs

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Allow UDFs to be shiv-packaged python modules with their

dependencies

- Use pyro for python RPC

- Maven build for pyro package into server

- Remove JObject spatial types

- Simpler conversion from java primitive/standard types and

collections

to JObject equivalents

Change-Id: Ibea23a2e9308132f343d80eff04ede9a235aa021

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5526

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 107 more files in changeset.
[NO ISSUE] Fix UDF Metadata Transactions

- Move UDF Metadata Transactions to be initiated from the CC

directly

- Remove old UDF initialization code

- Make DROP DATAVERSE remove libraries

- Make NCs properly read libraries on init with multipart dataverse

names

Change-Id: Ibda23a2e8308937f343d80eff04ede9a235da0d2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4923

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 45 more files in changeset.
[ASTERIXDB-2697]: Implementing AWS s3 as external data source

- user model changes: yes

- storage format changes: no

- interface changes: no

Details:

- Added an external reader for AWS S3.

- Updated query translator to include the WITH

parameters into the dataset details when creating

an external dataset.

- Added test case for AWS S3 using an S3 mocking server

to avoid using real credentials.

Change-Id: I71d89116c0bb404c9621b16f21a6a31cbf7bb7f6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5025

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 52 more files in changeset.
[NO ISSUE] HTTP Basic Auth servlet wrapper

- Basic auth servlet that checks a /etc/passwd style file on the CC

- Only enabled for UDF API servlet for now

- Associated code in client helper for adding file to CC path

- Config variable to manually set path if desired

Change-Id: Ibea23a2e8308937f343d80eff04ede9a235aa0d1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5204

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 61 more files in changeset.
[ASTERIXDB-2693] CREATE FUNCTION/ADAPTER ddl

- don't map UDFs on upload with descriptor

- UDFs get mapped via CREATE FUNCTION/ADAPTER after UDF payload is

uploaded

- add Library metadata catalog

Change-Id: Ic3c1e98c183cd214eea3e4fee24b2b7c46166b32

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3609

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 200 more files in changeset.
[NO ISSUE] Support multipart dataverse names

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Support dataverse names that consist of multiple parts

- Introduce DataverseName class and adopt it throughout the system

- Add testcases for multipart dataverse names

Change-Id: I1313b2dc0e8df6a9b0ded48dea122afc3ba5dbe7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4004

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 284 more files in changeset.
[ASTERIXDB-2597] Load UDFs via HTTP

- POST existing UDF format to /admin/udf/$DATAVERSE/$LIBNAME

- DELETE against that URL to remove UDFs

Change-Id: I6be9fef54c010bdb32f5c78af9b973f9843f442f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3386

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 35 more files in changeset.
[NO ISSUE][FUN] Add more data types for external library

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Added more types for external library.

2. Added test case for data types.

3. Fixed minor bug where the parameter type is not trimmed in UDF.

Change-Id: I6ce73e791533b0617074536e0d841242d9e0ee31

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2871

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 12 more files in changeset.
[NO ISSUE] Change the working directory of UDF

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Currently, there are two places for hosting UDFs. One is

"user.home/library", and the other is "app.home/lib/udfs". The former

one was used by Managix and is no longer supported. It also causes

problem for OSX users since it's collided with the system folder. From

now on, all udfs are placed under "app.home/lib/udfs". In the case where

"app.home" is not defined, "user.home/lib/udfs" will be used.

Change-Id: I409039d0ca054069853653b312b9f6187365a73b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2448

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-2148][FUN] Add init parameter for external UDF

- user model changes: no

- storage format changes: no

- interface changes: yes

Added parameters to LibraryFunction in library.xsd.

Details:

By enabling init function in external UDF, a user can reuse the same UDF

implementation with different paramters. One example can be to reuse the

same Machine Learning algorithm with different model files by assigning

different file paths to the UDF parameter.

Change-Id: I567ce0bcac288267595b2565e53fea61e16fbd65

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2107

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 34 more files in changeset.
[NO ISSUE] Use FunctionSignature more often

- user model changes: no

- storage format changes: no

- interface changes: yes

The metadata entity for a function now accepts a FunctionSignature.

It used to accept a dataverse, a function name, and an arity.

Change-Id: If00a914a1e56190ecd12c2687c32b4edc2895018

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2380

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 6 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 982 more files in changeset.
[ASTERIXDB-2180][FUN] Prevent dropping of entities used by functions

Add dependencies to Functional Metadata

Check dependencies before dropping datasets or functions

Add tests

Change-Id: I2f08ff150dfd57432b88381c507814ddb57bd67b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2253

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 29 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 390 more files in changeset.
[ASTERIXDB-1723] Delete managix

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Delete managix installer method and all dependencies

- Remove xml-based configuration

Change-Id: I5c710b6ec870bedd49dd46b74ed96a3665670e9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2155

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1378 more files in changeset.
[NO ISSUE][*DB] Enable loading external UDF for AsterixHyracksIntegrationUtil

- user model changes: no

- storage format changes: no

- interface changes: no

Add external UDF support for AsterixHyracksIntegrationUtil. Developer

may use following system properties to install external UDF for local

environment.

"external.lib" : udf package path.

"external.lib.dataverse": dataverse name (default: test).

"external.lib.libname": library name (default: testlib).

Change-Id: I333238b944b1c6d48a2b80b87a25f600522e0cc4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2078

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +98
    ./ExternalUDFLibrarian.java
    • -0
    • +31
    ./IExternalUDFLibrarian.java
  1. … 6 more files in changeset.
[ASTERIXDB-1949][FUN][ING] Function usage check refactoring in feed

- user model changes: no

- storage format changes: no

- interface changes: yes

Removed several interefaces for maintaining reference count.

Added getFunctions and getFeeds method for MetadataNode.

Details:

1. Change the usage check from reference count to scan feed connections

in all dataverses.

2. Add test case for drop dataverse which has function that is being

used in other dataverse. This operation should be blocked.

3. Fix small bug in apply function metadata which the function dataverse

was not recorded properly.

Change-Id: I568f7af18e69216ff2a5fd3b623a1667809b2f95

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1787

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 25 more files in changeset.
Fix function usage check in feed

1. Add reference count field to Function entity.

2. Add reference count check when drop function.

3. Increate reference count when apply function to feed. Decrease the

reference count when disconnect feed.

4. Add test cases for drop function that is being used and drop function

that is no longer in use.

Change-Id: I04bed3836455997269a4cae7088dcc25c745a3df

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1763

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 23 more files in changeset.
Add install/uninstall UDF to Ansible

1. Add udf.sh for handling install/uninstall UDF packages to the cluster.

2. Remove unnecessary path check in TestLibrarian.

Change-Id: If7ea9640a06e2b691c19a8a819307d84b55a679e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1682

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
AsterixDB Rat Execution Audit & Fixes

Change-Id: I3d771c5b63eaef4b6a0b4da769039ed5bca17736

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1609

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 493 more files in changeset.
Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.

2. FeedConnection metadata dataset is added (pkeys: dataverseName,

feedName, datasetName).

3. Replaced the old intake job + collect job combination with one single

job using SplitOperator.

4. Now one feed can connect to multiple datasets.

5. The disconnect feed job is replaced by ActiveManagerMessage.

6. The new feed life cycle is:

- Create feed

- Connect feed to dataset0, dataset1, dataset2, etc.

- Start feed

- Stop feed

- Disconnect feed

7. New feedEventListner framework by Abdullah

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1259

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 196 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -771
    • +0
    ./ExternalIndexingOperations.java
  1. … 424 more files in changeset.
Replace Servlets with Netty Based HTTP Servers

Change-Id: I3d552d4eb8c868535ca4c41cbcf7e352217b18ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1429

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 67 more files in changeset.
ASTERIXDB-1711: remove "Aql"

- AQLAPIServlet -> FullAPIServlet

- AqlPrimitiveValueProviderFactory -> PrimitiveValueProviderFactory

- remove AqlLogicalPlanAndMetadataImpl

- remore AqlMetadataImplConfig

Change-Id: I8e371941ba509d10193649aaed48a74a08d2d1f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1447

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 9 more files in changeset.
Cleanup AsterixDB Dependencies

Enforce maven dependencies, to ensure that all declared dependencies are

used, and that all used dependencies are declared.

Change-Id: I3cc9d4db57b2138931baa3ef51ea569c45b79a2f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1421

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 43 more files in changeset.
ASTERIXDB-1711: remove more mentions of Asterix

Change-Id: I21666fd54075ac56de8aab76bb7d5cdbec020121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1387

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -46
    • +46
    ./ExternalIndexingOperations.java
  1. … 569 more files in changeset.
Remove Append Only Flag

Change-Id: Id5d6917db8ab29aa01521596f556006e25a502fe

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1385

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 213 more files in changeset.
Deferred result retrieval for the QueryService

And some cleanup:

- fewer JSONExceptions on interfaces

- rename ResultDelivery options (existing HTTP APIs still work)

SYNC -> IMMEDIATE

ASYNC_DEFERRED -> DEFERRED

ASYNC -> ASYNC

- shorten variables

queryMetadataProvider -> metadataProvider

aqlStatements -> statements

compiled -> jobSpec

- rename ResultUtil.displayResults to printResults

Change-Id: I72d53be824d8dbf1d9f547b01f19097d0dc18add

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1373

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 14 more files in changeset.
ASTERIXDB-1711: remove some more Aql-prefixes

Change-Id: I041341fe76e9338b7c568a9eacaf826f8fa0ed46

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1360

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 232 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 253 more files in changeset.