Clone Tools
  • last updated 24 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Improve Error Handling in Local Directory Feeds

This change improves handling of two error types for filesystem

based feeds. The first one is the handling of IO Errors which

causes the input stream to be closed, and the second one is

reacting to missed filesystem events. In both cases, we scan the

directory and compare it with the history we have in order to

resume from where we last left off.

In addition, this change includes some refactoring in external

data. Particularly, we get rid of the stream provider layer and

instead, stream factories create input streams directly. This

is consistent with record reader factories which create readers

directly without reader providers.

Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/720

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 144 more files in changeset.
Support Change Feeds and Ingestion of Records with MetaData

This change allows feeds to perform upserts and deletes

in order to perform replication of an external data source.

The change does so by performing the following:

1. The adapter produces [PK][Record]. (Record == null --> delete)

2. The insert is replaced by an upsert operator.

Change-Id: If136a03d424970132dfb09f0dda56e160d4c0078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/621

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +50
    ./RecordWithPKDataParser.java
    • -0
    • +49
    ./TestRecordWithPKParser.java
    • -2
    • +10
    ./factory/ADMDataParserFactory.java
    • -2
    • +1
    ./factory/AbstractRecordStreamParserFactory.java
    • -13
    • +18
    ./factory/DelimitedDataParserFactory.java
    • -7
    • +14
    ./factory/HiveDataParserFactory.java
    • -47
    • +42
    ./factory/RecordWithMetadataParserFactory.java
    • -0
    • +77
    ./factory/TestRecordWithPKParserFactory.java
  1. … 256 more files in changeset.
Enabled Feed Tests and Added External Library tests

Feed tests had been switched off for a while due to having too many

sporadic failures. Now, we are switching them back on.

In addition, a new set of tests have been added to test that external

library works as expected.

Change-Id: Idd1fccd136fa2645b2707bbf7c04e60991ae8d4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/625

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 204 more files in changeset.
ASTERIXDB-1271: Revisit UUID Implementation

- Updated UUID implementation to eliminate marshalling to/from a pair

of longs, replacing with byte [].

- Eliminate all but one call to SecureRandom at init time, avoid risk

of entropy exhaustion

- Fix exceptions thrown

- Incorporate constant to represent number of bytes, String chars in

an instance of AUUID

- Split generated from parsed AUUID

- Eliminate intermediate StringBuilder construction on priting paths,

optimize imports

Change-Id: I9e90d42f6b62f80ad180dbd0c8c852db85b14173

Reviewed-on: https://asterix-gerrit.ics.uci.edu/610

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 10 more files in changeset.
ASTERIXDB-1281 - Interval format update to AQL and ADM

The new interval format takes a more generic approach to representing intervals.

Here is an example for a date interval:

interval(date("2012-01-01”), date(”2013-04-01”))

Note that the interval type is defined by the arguments to the interval expression.

Currently only date, time, and datetime types are supported for intervals. The new

format is used for ADM and AQL.

In addition to the format change, the internal byte structure of an interval has been

updated. The format looks like the following:

byte tag, T start, T end (where T is a date, time or datetime type)

Note how the tag has been moved to the front. Also with the new sturcture, an

interval is variable length, not fixed length as before.

Change-Id: I009c71b7a445d141e228ba15d56d0b6cf3c8a3f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/602

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 70 more files in changeset.
Some exception cleanup

- Use HyracksDataException and IOException for errors ingesting external

data (instead of a mixture of AlgebricksException, AsterixException, and

HyracksDataException).

- Avoid unnecessary wrapping of exceptions.

- Wrap exception at boundaries (e.g. HiveRecordParser)

- ADMLexerException and ADMDataParser.ParseException extend

HyracksDataException

- Tighter exception declarations and handling for internal methods.

Change-Id: Icf9591046c44f5fa2281874ff0c98d780e741267

Reviewed-on: https://asterix-gerrit.ics.uci.edu/603

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 58 more files in changeset.
Add Support for Upsert Operation

This change allows users to execute upsert commands which

couples delete if found with insert. It locks the primary

keys before doing the search ensuring consistency.

Change-Id: I8999000331795a5949d621d2dd003903e057a521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 251 more files in changeset.
Add flush() to IFrameWriter

This method is expected to be used with feeds to push

frames all the way to storage when needed. As of now, it is

needed in two cases:

1. No activities in ingestion node and need to push content

so it can be stored.

2. When the ingestion node needs to move the checkpoint ahead

if the at least once semantics are used.

Two feeds make use of this function. The filesystem feed and

couchbase feed which was introduced as well in this change.

Change-Id: Id862ce9e9b1360864c6976f2aea2137092f51203

Reviewed-on: https://asterix-gerrit.ics.uci.edu/585

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +107
    ./RecordWithMetadataParser.java
    • -0
    • +100
    ./factory/RecordWithMetadataParserFactory.java
  1. … 100 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 416 more files in changeset.
First stage of external data cleanup

In this change, different parts of external data were refactored.

The goal was to make it more modular, easier to maintain and allow

higher flexibility for extension in addition to reducing code redundancy.

Change-Id: I04a8c4e494d8d1363992b6fe0bdbe6b2b3b7b767

Reviewed-on: https://asterix-gerrit.ics.uci.edu/566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +1148
    ./ADMDataParser.java
    • -0
    • +522
    ./AbstractDataParser.java
    • -0
    • +208
    ./DelimitedDataParser.java
    • -0
    • +385
    ./HiveRecordParser.java
    • -0
    • +144
    ./TweetParser.java
    • -0
    • +56
    ./factory/ADMDataParserFactory.java
    • -0
    • +51
    ./factory/AbstractRecordStreamParserFactory.java
    • -0
    • +95
    ./factory/DelimitedDataParserFactory.java
    • -0
    • +67
    ./factory/HiveDataParserFactory.java
    • -0
    • +68
    ./factory/RSSParserFactory.java
    • -0
    • +67
    ./factory/TweetParserFactory.java
  1. … 332 more files in changeset.