Clone Tools
  • last updated 19 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Support Change Feeds and Ingestion of Records with MetaData

This change allows feeds to perform upserts and deletes

in order to perform replication of an external data source.

The change does so by performing the following:

1. The adapter produces [PK][Record]. (Record == null --> delete)

2. The insert is replaced by an upsert operator.

Change-Id: If136a03d424970132dfb09f0dda56e160d4c0078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/621

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +54
    ./adapter/FeedAdapter.java
  1. … 269 more files in changeset.
ASTERIXDB-1325, ASTERIXDB-1326: fixed leftouterjoin on an external dataset

- Fixed the issue that optimizer ignores index_nl hint on an index on an external dataset.

- Fixed the issue that index-nested-loop-leftouterjoin on an external dataset returns an empty results.

- Modified OptimizerTest to enable mockup HDFSCluster.

- Added an optimizer test on an external dataset.

Change-Id: I91b2e57f0bd1beaa58dd28a84d8e08d308246b26

Reviewed-on: https://asterix-gerrit.ics.uci.edu/672

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 5 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 326 more files in changeset.
Add flush() to IFrameWriter

This method is expected to be used with feeds to push

frames all the way to storage when needed. As of now, it is

needed in two cases:

1. No activities in ingestion node and need to push content

so it can be stored.

2. When the ingestion node needs to move the checkpoint ahead

if the at least once semantics are used.

Two feeds make use of this function. The filesystem feed and

couchbase feed which was introduced as well in this change.

Change-Id: Id862ce9e9b1360864c6976f2aea2137092f51203

Reviewed-on: https://asterix-gerrit.ics.uci.edu/585

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 103 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +66
    ./adapter/AdapterIdentifier.java
  1. … 415 more files in changeset.
First stage of external data cleanup

In this change, different parts of external data were refactored.

The goal was to make it more modular, easier to maintain and allow

higher flexibility for extension in addition to reducing code redundancy.

Change-Id: I04a8c4e494d8d1363992b6fe0bdbe6b2b3b7b767

Reviewed-on: https://asterix-gerrit.ics.uci.edu/566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -95
    • +0
    ./adapter/AzureTweetMetadataEntity.java
    • -155
    • +0
    ./adapter/ClientBasedFeedAdapter.java
    • -62
    • +0
    ./adapter/FileSystemBasedAdapter.java
    • -0
    • +48
    ./adapter/GenericAdapter.java
    • -46
    • +0
    ./adapter/IPullBasedFeedClient.java
    • -0
    • +162
    ./adapter/LookupAdapter.java
  1. … 329 more files in changeset.
Refactored External Data

This change re-arrange asterix module's order. asterix-

external-data is moved in front of asterix-metadata.

Change-Id: I46b60b5e1cc37fd59adc0dd89f374d96502091b2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/559

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +65
    ./adapter/IControlledAdapter.java
  1. … 89 more files in changeset.
Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -7
    • +5
    ./adapter/PushBasedTwitterFeedClient.java
  1. … 107 more files in changeset.
ASTERIXDB-1134: Updated the "push_twitter" adapter to allow better filtering of tweet streams (merged with current master)

Change-Id: I054da2c6b809c68d665e81829567bd7224fbaaf2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/441

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -17
    • +8
    ./adapter/PushBasedTwitterFeedClient.java
  1. … 3 more files in changeset.
Change License Headers

Also tweak the NOTICE file with some extras.

Change-Id:I09bc388089e515d7f51fd39c31bfbbc9f00cf84f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/388

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -10
    • +14
    ./adapter/AzureTweetMetadataEntity.java
    • -10
    • +14
    ./adapter/ClientBasedFeedAdapter.java
    • -10
    • +14
    ./adapter/FileSystemBasedAdapter.java
    • -10
    • +14
    ./adapter/HDFSIndexingAdapter.java
    • -10
    • +14
    ./adapter/IFeedClientFactory.java
    • -10
    • +14
    ./adapter/IPullBasedFeedClient.java
    • -10
    • +14
    ./adapter/NCFileSystemAdapter.java
    • -10
    • +14
    ./adapter/PullBasedAzureFeedClient.java
    • -10
    • +14
    ./adapter/PullBasedAzureTwitterAdapter.java
  1. … 2004 more files in changeset.
Change Java package from edu.uci.ics to org.apache

Change-Id: I2f01d2b5614e9e9c94fda4bf1294a8eba6a26c54

Reviewed-on: https://asterix-gerrit.ics.uci.edu/309

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./adapter/AzureTweetMetadataEntity.java
    • -12
    • +12
    ./adapter/ClientBasedFeedAdapter.java
    • -11
    • +11
    ./adapter/HDFSIndexingAdapter.java
    • -6
    • +6
    ./adapter/PullBasedAzureFeedClient.java
    • -5
    • +5
    ./adapter/PullBasedAzureTwitterAdapter.java
  1. … 2590 more files in changeset.
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: Iefd2a576415ebc1416cba2a3334d2b64f042ba92

Reviewed-on: https://asterix-gerrit.ics.uci.edu/306

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +47
    ./adapter/AzureTweetEntity.java
    • -0
    • +91
    ./adapter/AzureTweetMetadataEntity.java
    • -0
    • +150
    ./adapter/ClientBasedFeedAdapter.java
    • -0
    • +170
    ./adapter/FeedClient.java
    • -0
    • +58
    ./adapter/FileSystemBasedAdapter.java
    • -0
    • +93
    ./adapter/HDFSAdapter.java
    • -0
    • +75
    ./adapter/HDFSIndexingAdapter.java
    • -0
    • +45
    ./adapter/HiveAdapter.java
    • -0
    • +42
    ./adapter/IFeedClient.java
    • -0
    • +36
    ./adapter/IFeedClientFactory.java
    • -0
    • +42
    ./adapter/IPullBasedFeedClient.java
    • -0
    • +67
    ./adapter/NCFileSystemAdapter.java
    • -0
    • +157
    ./adapter/PullBasedAdapter.java
    • -0
    • +186
    ./adapter/PullBasedAzureFeedClient.java
    • -0
    • +99
    ./adapter/PullBasedAzureTwitterAdapter.java
  1. … 3781 more files in changeset.