Clone Tools
  • last updated 21 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Improve Error Handling in Local Directory Feeds

This change improves handling of two error types for filesystem

based feeds. The first one is the handling of IO Errors which

causes the input stream to be closed, and the second one is

reacting to missed filesystem events. In both cases, we scan the

directory and compare it with the history we have in order to

resume from where we last left off.

In addition, this change includes some refactoring in external

data. Particularly, we get rid of the stream provider layer and

instead, stream factories create input streams directly. This

is consistent with record reader factories which create readers

directly without reader providers.

Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/720

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 144 more files in changeset.
Fixed Feed Connect Statement

This change includes two fixes:

1. Feed connect doesn't return until the connection is complete.

2. When using wait for completion, it waits until all the jobs

complete.

Change-Id: I416bf4917b1f5cea687d1202c435f7183136cf1f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/726

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +48
    ./FeedProviderReadyMessage.java
  1. … 39 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 326 more files in changeset.
Add flush() to IFrameWriter

This method is expected to be used with feeds to push

frames all the way to storage when needed. As of now, it is

needed in two cases:

1. No activities in ingestion node and need to push content

so it can be stored.

2. When the ingestion node needs to move the checkpoint ahead

if the at least once semantics are used.

Two feeds make use of this function. The filesystem feed and

couchbase feed which was introduced as well in this change.

Change-Id: Id862ce9e9b1360864c6976f2aea2137092f51203

Reviewed-on: https://asterix-gerrit.ics.uci.edu/585

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 103 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +96
    ./EndFeedMessage.java
    • -0
    • +102
    ./FeedCongestionMessage.java
    • -0
    • +145
    ./FeedMessageService.java
    • -0
    • +99
    ./FeedReportMessage.java
    • -0
    • +98
    ./FeedTupleCommitAckMessage.java
    • -0
    • +81
    ./FeedTupleCommitResponseMessage.java
    • -0
    • +126
    ./MessageListener.java
    • -0
    • +111
    ./MessageReceiver.java
    • -0
    • +68
    ./NodeReportMessage.java
    • -0
    • +68
    ./PrepareStallMessage.java
    • -0
    • +134
    ./RemoteSocketMessageListener.java
    • -0
    • +113
    ./ScaleInReportMessage.java
    • -0
    • +160
    ./SocketMessageListener.java
    • -0
    • +128
    ./StorageReportFeedMessage.java
  1. … 403 more files in changeset.