Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][TEST] Migrate feed test cases from AQL to SQLPP

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Moved all feed test cases from AQL to SQLPP with syntax adaptations.

Change-Id: I06d20b09912735003b76df0e337cadf1a7551a97

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2304

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +18
    ./PLEASE_ADD_SQLPP_TEST_CASE
    • -62
    • +0
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.1.ddl.aql
    • -29
    • +0
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.2.update.aql
    • -30
    • +0
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.3.query.aql
    • -24
    • +0
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.4.ddl.aql
    • -60
    • +0
    ./change-feed-with-meta-open-index-in-meta/change-feed-with-meta-open-index-in-meta.1.ddl.aql
    • -26
    • +0
    ./change-feed-with-meta-open-index-in-meta/change-feed-with-meta-open-index-in-meta.2.ddl.aql
    • -58
    • +0
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.1.ddl.aql
    • -30
    • +0
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
    • -27
    • +0
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.3.ddl.aql
    • -31
    • +0
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
    • -25
    • +0
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.5.ddl.aql
    • -60
    • +0
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.1.ddl.aql
    • -30
    • +0
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
    • -31
    • +0
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.3.query.aql
  1. … 377 more files in changeset.
[ASTERIXDB-2182][ING] Use JSON-like syntax for feed configuration

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

To be consistent with the other JSON-like syntax in defining

configurations, CREATE FEED will change to use JSON object instead of

key-value pairs. Also, the <USING> adapter_name syntax is removed. The

adapter name becomes as part of the configuration. After this change,

the feed configuration are specified as:

<CREATE> <FEED> testFeed <WITH> {

"adapter-name" : "localfs",

"path" : "localhost://../../../../../tweets.json",

"format" : "adm",

"type-name" : "typeTweet"

};

This patch also fixes two minor mistakes when drop feed and adapter in

Metadata.

Change-Id: I2dfb0fdf74d28ae0c45dafcf6babb624bd8b2149

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2209

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -28
    • +29
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.1.ddl.aql
    • -12
    • +13
    ./change-feed-with-meta-open-index-in-meta/change-feed-with-meta-open-index-in-meta.1.ddl.aql
    • -12
    • +13
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.1.ddl.aql
    • -13
    • +14
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.1.ddl.aql
    • -12
    • +13
    ./change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.1.ddl.aql
    • -12
    • +13
    ./change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.1.ddl.aql
    • -12
    • +13
    ./change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.1.ddl.aql
    • -13
    • +14
    ./change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.1.ddl.aql
    • -13
    • +14
    ./change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.1.ddl.aql
    • -10
    • +11
    ./change-feed/change-feed.1.ddl.aql
    • -8
    • +8
    ./connect-feed-with-function/connect-feed-with-function.1.ddl.aql
    • -7
    • +7
    ./connect-live-feed/connect-live-feed.0.ddl.aql
    • -7
    • +7
    ./create-feed-with-policy/create-feed-with-policy.1.ddl.aql
    • -7
    • +7
    ./disconnect-live-feed/disconnect-live-feed.0.ddl.aql
  1. … 93 more files in changeset.
[ASTERIXDB-1983] Feed pipeline refactoring for SQL++

- user model changes: no

- storage format changes: no

- interface changes: no

Current implementation of feed uses handcraft AQL queries for creating

feed pipeline. This causes a lot of issues and does not support SQL++

very well. Also, there is an overhead for parsing the query everytime.

In this patch, it's replaced with compiled statement in SQL++ which

provides support for attaching UDF to feed as well.

Details:

1. Remove SubscribeFeedStatement.

2. Remove SubscribeFeed related query compilation code, and reuse the

upsert dataflow.

3. Added SQL++ User Defined Function support for feed, including adding

multiple functions to one feed.

4. Related test cases added.

5. Change the default behavior of feed to be upsert instead of upsert.

'insert-feed' option is provided for experiment uses.

6. This patch also fixes several feed related bugs: [ASTERIXDB-2085]

[ASTERIXDB-2124].

Change-Id: I0ae5a837613780a4d2c90c98139fdc6d5e040cc9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2059

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -57
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.1.ddl.aql
    • -32
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.2.update.aql
    • -27
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.3.server.aql
    • -27
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.4.sleep.aql
    • -29
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.5.update.aql
    • -31
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.6.query.aql
    • -28
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.7.server.aql
    • -28
    • +0
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.8.ddl.aql
    • -0
    • +57
    ./connect-feed-with-function/connect-feed-with-function.1.ddl.aql
    • -0
    • +32
    ./connect-feed-with-function/connect-feed-with-function.2.update.aql
    • -0
    • +27
    ./connect-feed-with-function/connect-feed-with-function.3.ddl.aql
    • -0
    • +45
    ./insert-feed/insert-feed.1.ddl.aql
    • -0
    • +24
    ./insert-feed/insert-feed.2.update.aql
    • -0
    • +19
    ./insert-feed/insert-feed.3.server.aql
    • -0
    • +19
    ./insert-feed/insert-feed.4.sleep.aql
  1. … 79 more files in changeset.
[ASTERIXDB-1949][FUN][ING] Function usage check refactoring in feed

- user model changes: no

- storage format changes: no

- interface changes: yes

Removed several interefaces for maintaining reference count.

Added getFunctions and getFeeds method for MetadataNode.

Details:

1. Change the usage check from reference count to scan feed connections

in all dataverses.

2. Add test case for drop dataverse which has function that is being

used in other dataverse. This operation should be blocked.

3. Fix small bug in apply function metadata which the function dataverse

was not recorded properly.

Change-Id: I568f7af18e69216ff2a5fd3b623a1667809b2f95

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1787

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +26
    ./drop-dataverse-with-function-used-by-feed/drop-dataverse-with-function-used-by-feed.1.ddl.aql
    • -0
    • +39
    ./drop-dataverse-with-function-used-by-feed/drop-dataverse-with-function-used-by-feed.2.ddl.aql
    • -0
    • +23
    ./drop-dataverse-with-function-used-by-feed/drop-dataverse-with-function-used-by-feed.3.update.aql
    • -0
    • +20
    ./drop-dataverse-with-function-used-by-feed/drop-dataverse-with-function-used-by-feed.4.ddl.aql
  1. … 22 more files in changeset.
Fix function usage check in feed

1. Add reference count field to Function entity.

2. Add reference count check when drop function.

3. Increate reference count when apply function to feed. Decrease the

reference count when disconnect feed.

4. Add test cases for drop function that is being used and drop function

that is no longer in use.

Change-Id: I04bed3836455997269a4cae7088dcc25c745a3df

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1763

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +44
    ./drop-function-no-longer-used-by-feed/drop-function-no-longer-used-by-feed.1.ddl.aql
    • -0
    • +24
    ./drop-function-no-longer-used-by-feed/drop-function-no-longer-used-by-feed.2.update.aql
    • -0
    • +21
    ./drop-function-no-longer-used-by-feed/drop-function-no-longer-used-by-feed.3.ddl.aql
    • -0
    • +44
    ./drop-function-used-by-feed/drop-function-used-by-feed.1.ddl.aql
    • -0
    • +23
    ./drop-function-used-by-feed/drop-function-used-by-feed.2.update.aql
    • -0
    • +21
    ./drop-function-used-by-feed/drop-function-used-by-feed.3.ddl.aql
  1. … 18 more files in changeset.
Add function signature check to Connect Feed

1. Revise the exception info when apply an unknown function to feed.

2. Fix the possible NPE in connect feed statement.

3. Add test case for applying undefined function.

Change-Id: I1462b394d84ea7e1eae5a03f98fe8cd39213eb8e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1674

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +39
    ./feed-with-undefined-function/feed-with-undefined-function.1.ddl.aql
    • -0
    • +26
    ./feed-with-undefined-function/feed-with-undefined-function.2.update.aql
  1. … 7 more files in changeset.
Fix continue ingestion bug when exception happens

1. Fix the bug when exception happens, localfs adapter couldn't pick up

a new file to continue the ingestion.

2. Change the exception handling from string to error code.

3. Added a test case.

4. Removed some tailing . in error message.

Change-Id: Ie8656a4d1afabbc1b481eb97509a861b22478676

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1713

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +32
    ./record-reader-with-malformed-input-stream/record-reader-with-malformed-input-stream.1.ddl.aql
    • -0
    • +25
    ./record-reader-with-malformed-input-stream/record-reader-with-malformed-input-stream.2.update.aql
    • -0
    • +19
    ./record-reader-with-malformed-input-stream/record-reader-with-malformed-input-stream.4.sleep.aql
    • -0
    • +22
    ./record-reader-with-malformed-input-stream/record-reader-with-malformed-input-stream.5.update.aql
    • -0
    • +23
    ./record-reader-with-malformed-input-stream/record-reader-with-malformed-input-stream.6.query.aql
    • -0
    • +21
    ./record-reader-with-malformed-input-stream/record-reader-with-malformed-input-stream.8.ddl.aql
  1. … 8 more files in changeset.
Add test cases for alive feed check when connect/disconnect

`connect feed' and `disconnect feed` are not allowed when feed is alive.

Add two test cases to check both exceptions.

Change-Id: Icbcc19da2a9f42bee21b52932ba4f19a0f01aeec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1664

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -0
    • +40
    ./connect-live-feed/connect-live-feed.0.ddl.aql
    • -0
    • +21
    ./connect-live-feed/connect-live-feed.1.update.aql
    • -0
    • +19
    ./connect-live-feed/connect-live-feed.2.sleep.aql
    • -0
    • +21
    ./connect-live-feed/connect-live-feed.3.update.aql
    • -0
    • +41
    ./disconnect-live-feed/disconnect-live-feed.0.ddl.aql
    • -0
    • +21
    ./disconnect-live-feed/disconnect-live-feed.1.update.aql
    • -0
    • +19
    ./disconnect-live-feed/disconnect-live-feed.2.sleep.aql
    • -0
    • +21
    ./disconnect-live-feed/disconnect-live-feed.3.update.aql
  1. … 4 more files in changeset.
Feed Policy Refactoring

1. Fix the framesize inconsistency in feed dataflow.

2. Add a runtime test case for create feed with policy.

3. Fix the FeedRuntimeInputHandler flush() logic. Only flush after the

writer is opened.

4. Refactor FeedPolicyAccessor and BuiltinPolices. Now we only have

spill and discard polices.

5. Remove PolicyEnforcer. Merge the functionality into

FeedPolicyAccessor.

6. Revise SocketServerInputStream. Make the expected exception more

friendly.

7. Fixed one test case fail in change feed.

8. Refactor FeedRuntimeInputHandler consumer thread logic. Change the

poison consumer thread mechanism.

Change-Id: Ibc10139925cfedee66d1263990ba80b94675f182

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1591

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +58
    ./create-feed-with-policy/create-feed-with-policy.1.ddl.aql
    • -0
    • +32
    ./create-feed-with-policy/create-feed-with-policy.2.update.aql
    • -0
    • +26
    ./create-feed-with-policy/create-feed-with-policy.3.server.aql
    • -0
    • +26
    ./create-feed-with-policy/create-feed-with-policy.4.sleep.aql
    • -0
    • +29
    ./create-feed-with-policy/create-feed-with-policy.5.update.aql
    • -0
    • +31
    ./create-feed-with-policy/create-feed-with-policy.6.query.aql
    • -0
    • +27
    ./create-feed-with-policy/create-feed-with-policy.7.server.aql
    • -0
    • +28
    ./create-feed-with-policy/create-feed-with-policy.8.ddl.aql
  1. … 20 more files in changeset.
ASTERIXDB-1854 Fix Feed with user defined function

1. Fix the bug that AQL Function applied to feed can only $x as

parameter.

2. Fix Connect feed statement can only take one user defined function.

3. Fix one metadata bug when store multiple UDF in feed connection.

Change-Id: Ic60582b4198614a1f6a6026fe4c8675c9fec8a97

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1635

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -0
    • +57
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.1.ddl.aql
    • -0
    • +32
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.2.update.aql
    • -0
    • +27
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.3.server.aql
    • -0
    • +27
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.4.sleep.aql
    • -0
    • +29
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.5.update.aql
    • -0
    • +31
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.6.query.aql
    • -0
    • +28
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.7.server.aql
    • -0
    • +28
    ./connect-feed-with-aql-function/connect-feed-with-aql-function.8.ddl.aql
  1. … 7 more files in changeset.
AsterixDB Rat Execution Audit & Fixes

Change-Id: I3d771c5b63eaef4b6a0b4da769039ed5bca17736

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1609

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -1
    • +1
    ./feed-with-external-function/feed-with-external-function.2.lib.aql
    • -1
    • +1
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.2.lib.aql
    • -1
    • +1
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.2.lib.aql
    • -1
    • +1
    ./feed-with-external-parser/feed-with-external-parser.2.lib.aql
    • -1
    • +1
    ./feed-with-meta-pk-in-meta/feed-with-meta-pk-in-meta.1.ddl.aql
    • -1
    • +1
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.2.lib.aql
  1. … 488 more files in changeset.
ASTERIXDB-1310 Autogenerated uuid

Added test case for how to use autogenerated uuid with feed properly.

Change-Id: Ief4fba8a7227a40c5d22293758cbdf4575b14401

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1601

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./feed-push-socket/feed-push-socket.4.sleep.aql
    • -0
    • +71
    ./push-socket-with-auuid/push-socket-with-auuid.1.ddl.aql
    • -0
    • +35
    ./push-socket-with-auuid/push-socket-with-auuid.2.update.aql
    • -0
    • +27
    ./push-socket-with-auuid/push-socket-with-auuid.3.server.aql
    • -0
    • +27
    ./push-socket-with-auuid/push-socket-with-auuid.4.sleep.aql
    • -0
    • +32
    ./push-socket-with-auuid/push-socket-with-auuid.5.update.aql
    • -0
    • +37
    ./push-socket-with-auuid/push-socket-with-auuid.6.query.aql
    • -0
    • +28
    ./push-socket-with-auuid/push-socket-with-auuid.7.server.aql
    • -0
    • +29
    ./push-socket-with-auuid/push-socket-with-auuid.8.ddl.aql
  1. … 2 more files in changeset.
Expose TwitterFirehose Stream to Users

1. Moved TwitterFirehoseStream from Test to external-data which enables

users to access it from interface.

2. Related test cases fix.

3. Code style fix in DataGenerator, TweetGenerator, etc.

Change-Id: I8f899fde0d9a9dbd7891957be2f1241d5acfc4ab

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1590

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 13 more files in changeset.
Feed Exception Revise

1. Add compliation exception for missing Twitter4J library. Try to

expose this exception when create feed.

2. Tweaked 'invalid feed parameter exception'. Expose detailed exception

information to front end.

3. Related test cases fix.

Change-Id: Iee65b2a5365067c9ba2aa1615ac23f0ace41b51e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1534

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 7 more files in changeset.
Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.

2. FeedConnection metadata dataset is added (pkeys: dataverseName,

feedName, datasetName).

3. Replaced the old intake job + collect job combination with one single

job using SplitOperator.

4. Now one feed can connect to multiple datasets.

5. The disconnect feed job is replaced by ActiveManagerMessage.

6. The new feed life cycle is:

- Create feed

- Connect feed to dataset0, dataset1, dataset2, etc.

- Start feed

- Stop feed

- Disconnect feed

7. New feedEventListner framework by Abdullah

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1259

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +3
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.2.update.aql
    • -0
    • +2
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
    • -1
    • +1
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
    • -0
    • +2
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
    • -0
    • +2
    ./change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
    • -0
    • +2
    ./change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
    • -0
    • +2
    ./change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
    • -0
    • +2
    ./change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
    • -0
    • +2
    ./change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
    • -0
    • +2
    ./change-feed/change-feed.2.update.aql
    • -0
    • +46
    ./connect-feed/connect-feed.0.ddl.aql
    • -0
    • +29
    ./connect-feed/connect-feed.1.update.aql
    • -0
    • +20
    ./connect-feed/connect-feed.2.update.aql
    • -0
    • +20
    ./connect-feed/connect-feed.3.query.aql
    • -0
    • +19
    ./connect-feed/connect-feed.4.ddl.aql
  1. … 183 more files in changeset.
Fix for ASTERIXDB-1738: Change feed fails to delete record with meta PK

Change-Id: I5ce8d9c69f96593ee305cfad4b44e486c9f1d6cc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1351

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +61
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.1.ddl.aql
    • -0
    • +27
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.2.update.aql
    • -0
    • +30
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.3.query.aql
    • -0
    • +24
    ./change-feed-with-meta-csv/change-feed-with-meta-csv.4.ddl.aql
  1. … 5 more files in changeset.
Move Twitter Firehose Datasource to Test Source Folder

Change-Id: Iefe2130707012b8ce60f5dfac96635a1a515a076

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1290

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 34 more files in changeset.
+= PollQuery Test File Type

pollquery test type files will run the query every polldelaysecs

[default:1] seconds for up to polltimeoutsecs seconds or until the

correct result is found. This avoids unnecessarily long or dangerously

short sleeps to wait for some asynchronous operation to complete.

Usage: polltimeoutsecs=nnn must be present somewhere in your file,

otherwise an error is thrown. Optionally, polldelaysecs=nnn can be also

present to override the default poll frequency of 1s.

Change-Id: I7e4c67c74debf8253479257a1c54d6426a9531d8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1267

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -19
    • +0
    ./feed-with-external-function/feed-with-external-function.4.sleep.aql
    • -0
    • +33
    ./feed-with-external-function/feed-with-external-function.5.pollquery.aql
    • -32
    • +0
    ./feed-with-external-function/feed-with-external-function.5.query.aql
  1. … 1 more file in changeset.
Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for

feed configuration, which changes the default record

insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1068

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +46
    ./upsert-feed/upsert-feed.1.ddl.aql
    • -0
    • +29
    ./upsert-feed/upsert-feed.2.update.aql
    • -0
    • +26
    ./upsert-feed/upsert-feed.3.server.aql
    • -0
    • +26
    ./upsert-feed/upsert-feed.4.sleep.aql
    • -0
    • +28
    ./upsert-feed/upsert-feed.5.update.aql
    • -0
    • +31
    ./upsert-feed/upsert-feed.6.query.aql
    • -0
    • +27
    ./upsert-feed/upsert-feed.7.server.aql
    • -0
    • +27
    ./upsert-feed/upsert-feed.8.ddl.aql
  1. … 8 more files in changeset.
Fix open field CREATE INDEX syntax (AQL and SQLPP).

Our CREATE INDEX syntax for open fields didn't match our semantics.

Our semantics allow to-be-indexed open field(s) to be MISSING.

We don't support (or offer :-)) tighter semantics than that.

Our syntax didn't require creators to say this, but now it does:

Old: CREATE INDEX myIdx on MyDSet(myTime: datetime) ENFORCED;

New: CREATE INDEX myIdx on MyDSet(myTime: datetime?) ENFORCED;

Have now addressed all of Till's comments (I think).

Change-Id: I32bf8bc5ac1c0e6aa6744ca8f91a197311219814

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1106

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./change-feed-with-meta-open-index-in-meta/change-feed-with-meta-open-index-in-meta.1.ddl.aql
    • -1
    • +1
    ./change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.3.ddl.aql
    • -1
    • +1
    ./change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.1.ddl.aql
    • -1
    • +1
    ./change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.1.ddl.aql
    • -1
    • +1
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.3.ddl.aql
    • -2
    • +2
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.3.ddl.aql
    • -4
    • +4
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.1.ddl.aql
  1. … 781 more files in changeset.
TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1002

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +55
    ./revised-tweet-parser/revised-tweet-parser.1.ddl.aql
    • -0
    • +22
    ./revised-tweet-parser/revised-tweet-parser.2.update.aql
  1. … 13 more files in changeset.
ASTERIXDB-1135 Fix Feed Policy Creation From File

Change-Id: I452ac11714e20e2ecc31ab8b96a9964ca9d9d6cd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1018

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +29
    ./create-policy-from-file/create-policy-from-file.1.ddl.aql
    • -0
    • +27
    ./create-policy-from-file/create-policy-from-file.2.query.aql
    • -0
    • +27
    ./create-policy-from-file/create-policy-from-file.3.ddl.aql
    • -48
    • +0
    ./issue_711_feeds/issue_711_feeds.1.ddl.aql
  1. … 61 more files in changeset.
Cast Record Before Insert for Open Indexes

This change modifies the insert plan to perform cast for open

indexes before inserting to primary index. This avoids modification

of indexes and then doing complex work to try and undo the insert.

In addition, feeds can simply log and remove tuples which have

open fields of incompatible type.

Change-Id: I0eef5813ddbfe1b9c518cd7f92d37c95e8486914

Reviewed-on: https://asterix-gerrit.ics.uci.edu/997

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +33
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.1.ddl.aql
    • -0
    • +26
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.2.lib.aql
    • -0
    • +35
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.3.ddl.aql
    • -0
    • +31
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.4.update.aql
    • -0
    • +30
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.5.query.aql
    • -0
    • +26
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.6.lib.aql
    • -0
    • +26
    ./feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.7.ddl.aql
    • -0
    • +33
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.1.ddl.aql
    • -0
    • +26
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.2.lib.aql
    • -0
    • +36
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.3.ddl.aql
    • -0
    • +31
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.4.update.aql
    • -0
    • +30
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.5.query.aql
    • -0
    • +26
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.6.lib.aql
    • -0
    • +26
    ./feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.7.ddl.aql
  1. … 17 more files in changeset.
Hard Tabs++

Eliminate more hard tabs, widen scope of checkstyle rules to prevent regressions.

Relax whitespace expectations in polygon ctor

Change-Id: Ia3f78fc6705bc79a21cfa97fdfcad30998ab1058

Reviewed-on: https://asterix-gerrit.ics.uci.edu/967

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 455 more files in changeset.
Fix Indexing on Open fields and Meta fields

This change fix the following cases:

1. Build a secondary index on a meta field: success

2. Build an open index on a meta field: failure

3. Build a secondary index on an open field in record part: success

Testing ingestion and querying are working correctly for these cases.

Change-Id: I6195149940f150250a65f2515e9ac9d6de2a33f9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/930

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +59
    ./change-feed-with-meta-open-index-in-meta/change-feed-with-meta-open-index-in-meta.1.ddl.aql
    • -0
    • +26
    ./change-feed-with-meta-open-index-in-meta/change-feed-with-meta-open-index-in-meta.2.ddl.aql
    • -0
    • +57
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.1.ddl.aql
    • -0
    • +28
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
    • -0
    • +27
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.3.ddl.aql
    • -0
    • +31
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
    • -0
    • +25
    ./change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.5.ddl.aql
    • -0
    • +59
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.1.ddl.aql
    • -0
    • +28
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
    • -0
    • +31
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.3.query.aql
    • -0
    • +25
    ./change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.4.ddl.aql
    • -0
    • +57
    ./change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.1.ddl.aql
    • -0
    • +28
    ./change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
    • -0
    • +27
    ./change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.3.ddl.aql
    • -0
    • +31
    ./change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.4.query.aql
  1. … 64 more files in changeset.
Improve External Data

This change enable an adapter to specify its data parser and not have

the user specify it and pass it as part of the adapter configurations.

In addition, it introduces a new parameter "parser-factory" that can be

used to specifies a parser factory instead of using parser to specify

a parser factory.

Change-Id: Iae2560c73fa63e9454f731b8e893ae779a2ac7d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/906

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -2
    • +0
    ./change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.1.ddl.aql
  1. … 33 more files in changeset.
ASTERIXDB-1477 Fix Drop Feed If Not Exists

Change-Id: I9ce76ff5fd6611df5060a1de77742aba9d146ab5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/918

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +28
    ./drop-nonexistent-feed/drop-nonexistent-feed.1.ddl.aql
    • -19
    • +0
    ./feed-with-external-function/feed-with-external-function.6.ddl.aql
    • -0
    • +19
    ./feed-with-external-function/feed-with-external-function.6.lib.aql
    • -0
    • +19
    ./feed-with-external-function/feed-with-external-function.7.ddl.aql
    • -19
    • +0
    ./feed-with-external-function/feed-with-external-function.7.lib.aql
  1. … 2 more files in changeset.
Remove Test Classes From Production Binaries

Change-Id: I1da72566e7d140a1fb965c893dcea6881b99ca26

Reviewed-on: https://asterix-gerrit.ics.uci.edu/901

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +1
    ./change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.1.ddl.aql
  1. … 22 more files in changeset.
ASTERIXDB-1415 Fix Feed Freeze with Indexed Dataset

A feed might cause the system to freeze when connected to

a dataset that has secondary indexes. This case happen if

a frame contains all duplicate records pass the primary index

insert operator. The root cause is that the assign runtime in

the pipeline assumes at least a single record. Hence, a corrupted

tuple will reach the commit operator which will create a bad log

record with a PK of size 0. The log record will cause the log

flusher thread to die.

Change-Id: Ib0fa0bc08cf4224dcf64f83ea72c227f63affc73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/839

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +49
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.1.ddl.aql
    • -0
    • +19
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.2.lib.aql
    • -0
    • +32
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.3.ddl.aql
    • -0
    • +30
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.4.update.aql
    • -0
    • +29
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.5.query.aql
    • -0
    • +19
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.6.lib.aql
    • -0
    • +25
    ./feed-with-multiple-indexes/feed-with-multiple-indexes.7.ddl.aql
  1. … 3 more files in changeset.
Move merged files

    • -0
    • +54
    ./twitter-feed/twitter-feed.1.ddl.aql
    • -0
    • +25
    ./twitter-feed/twitter-feed.2.update.aql
  1. … 162 more files in changeset.