Clone Tools
  • last updated 20 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Support Change Feeds and Ingestion of Records with MetaData

This change allows feeds to perform upserts and deletes

in order to perform replication of an external data source.

The change does so by performing the following:

1. The adapter produces [PK][Record]. (Record == null --> delete)

2. The insert is replaced by an upsert operator.

Change-Id: If136a03d424970132dfb09f0dda56e160d4c0078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/621

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +7308
    ./resources/beer.csv
  1. … 256 more files in changeset.
ASTERIXDB-1302 ASTERIXDB-1301 Fix Socket Feed Connection

A bug causes a read lock to never be released when a feed is

connected with "wait-for-completion" set to false. The bug

was fixed and a test case was added.

Another bug was causing the socket feed to not receive

connections correctly. The bug was fixed and a test case

was added.

Additionally, this change ensures that adapters have absolute

partitions to ensure consistency with regards to feed log

manager.

Change-Id: I8f6e982440d3577343f2479c3779653a9c3db614

Reviewed-on: https://asterix-gerrit.ics.uci.edu/660

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

  1. … 49 more files in changeset.
Remove Unnecessary Prints in Tests

Change-Id: I4def109c0a8b40343d59d5689f8a791249174478

Reviewed-on: https://asterix-gerrit.ics.uci.edu/655

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 3 more files in changeset.
Enabled Feed Tests and Added External Library tests

Feed tests had been switched off for a while due to having too many

sporadic failures. Now, we are switching them back on.

In addition, a new set of tests have been added to test that external

library works as expected.

Change-Id: Idd1fccd136fa2645b2707bbf7c04e60991ae8d4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/625

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +1565
    ./java/org/apache/asterix/external/classad/ClassAd.java
    • -0
    • +25
    ./java/org/apache/asterix/external/classad/ClassAdFunc.java
    • -0
    • +278
    ./java/org/apache/asterix/external/classad/ClassAdTime.java
    • -0
    • +492
    ./java/org/apache/asterix/external/classad/ClassAdUnParser.java
    • -0
    • +66
    ./java/org/apache/asterix/external/classad/Common.java
    • -0
    • +120
    ./java/org/apache/asterix/external/classad/EvalState.java
    • -0
    • +280
    ./java/org/apache/asterix/external/classad/ExprList.java
    • -0
    • +401
    ./java/org/apache/asterix/external/classad/ExprTree.java
    • -0
    • +144
    ./java/org/apache/asterix/external/classad/ExprTreeHolder.java
  1. … 190 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 326 more files in changeset.
ASTERIXDB-1304: remove interfaces and consolidate

Change-Id: Ibb8819f51c655fadaf47fdbad0af5f66b261b2b5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/643

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 14 more files in changeset.
Support Sending Messages Alongside Frame Data

This change supports sending messages with records. The tuple Appender

reserves 100 bytes for a message. Before sending the frame, it appends

The message in the last tuple position. The message is read from the

task context as the shared object between different operators in the

pipeline. The first use of this feature will be within feeds to request

acks for at least once semantics.

Change-Id: Iaa23e9f8a909ddcafc1c3ee95181092eb04ee1ad

Reviewed-on: https://asterix-gerrit.ics.uci.edu/605

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 13 more files in changeset.
ASTERIXDB-1281 - Interval format update to AQL and ADM

The new interval format takes a more generic approach to representing intervals.

Here is an example for a date interval:

interval(date("2012-01-01”), date(”2013-04-01”))

Note that the interval type is defined by the arguments to the interval expression.

Currently only date, time, and datetime types are supported for intervals. The new

format is used for ADM and AQL.

In addition to the format change, the internal byte structure of an interval has been

updated. The format looks like the following:

byte tag, T start, T end (where T is a date, time or datetime type)

Note how the tag has been moved to the front. Also with the new sturcture, an

interval is variable length, not fixed length as before.

Change-Id: I009c71b7a445d141e228ba15d56d0b6cf3c8a3f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/602

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 71 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 415 more files in changeset.
First stage of external data cleanup

In this change, different parts of external data were refactored.

The goal was to make it more modular, easier to maintain and allow

higher flexibility for extension in addition to reducing code redundancy.

Change-Id: I04a8c4e494d8d1363992b6fe0bdbe6b2b3b7b767

Reviewed-on: https://asterix-gerrit.ics.uci.edu/566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 329 more files in changeset.
Refactored External Data

This change re-arrange asterix module's order. asterix-

external-data is moved in front of asterix-metadata.

Change-Id: I46b60b5e1cc37fd59adc0dd89f374d96502091b2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/559

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 93 more files in changeset.
ASTERIXDB-1198: make ISerializerDeserializer implementations in AsterixDB stateless.

Change-Id: I808b6583db6b48a0e9000dcf0ac88c03c336c3a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/522

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

  1. … 63 more files in changeset.
Fix .jj and Markdown Licenses

Change-Id: Iadb5074d0e0d4fb2b7d4303aa405e9fc91f4bd21

Reviewed-on: https://asterix-gerrit.ics.uci.edu/426

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 55 more files in changeset.
Fix for ASTERIXDB-951, ASTERIXDB-1074

Change-Id: Ibdb5074d0e0d4fb2b7d4303aa405c9fc90f4bd09

Reviewed-on: https://asterix-gerrit.ics.uci.edu/378

Reviewed-by: Chen Li <chenli@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Heri Ramampiaro <heriram@gmail.com>

  1. … 8 more files in changeset.
Change License Headers

Also tweak the NOTICE file with some extras.

Change-Id:I09bc388089e515d7f51fd39c31bfbbc9f00cf84f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/388

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2004 more files in changeset.
Change Java package from edu.uci.ics to org.apache

Change-Id: I2f01d2b5614e9e9c94fda4bf1294a8eba6a26c54

Reviewed-on: https://asterix-gerrit.ics.uci.edu/309

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2590 more files in changeset.
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: Iefd2a576415ebc1416cba2a3334d2b64f042ba92

Reviewed-on: https://asterix-gerrit.ics.uci.edu/306

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 3781 more files in changeset.
Introduces Feeds 2.0

commit c3f577861fc705d848c1641605689cadd6973bae

Merge: ebc4cae fc0c2c0

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Fri Jun 26 13:04:05 2015 -0700

Merge branch 'raman/feeds_2_release' of https://code.google.com/p/asterixdb-sandbox into raman/feeds_2_release

Conflicts:

asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/FeedServlet.java

asterix-external-data/src/main/java/edu/uci/ics/asterix/external/library/java/JObjectAccessors.java

commit ebc4cae21a7302869f953df1ebda601e798d12d2

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Sat Jun 20 17:14:45 2015 -0700

Introduces Feeds 2.0

Some of the prominent chnages introduced are as follows

a) Support for building a cascade network of feeds (via secondary feeds feature)

b) Feed Management Console for tracking active feeds and associated metrics

c) Support for elastic runtime for data ingestion

d) Improved fault-tolerance with support for logging of failed records

Documentation has been added at asterix-doc/src/site/markdown/feeds/

commit fc0c2c0549a6ee8b202e57607d2e110478cd57bb

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Sat Jun 20 17:14:45 2015 -0700

Introduces Feeds 2.0

Some of the prominent chnages introduced are as follows

a) Support for building a cascade network of feeds (via secondary feeds feature)

b) Feed Management Console for tracking active feeds and associated metrics

c) Support for elastic runtime for data ingestion

d) Improved fault-tolerance with support for logging of failed records

Documentation has been added at asterix-doc/src/site/markdown/feeds/

Change-Id: I498f01c591a229aaf51cec43ab20f3e5c4f072f4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/297

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +76
    ./resources/library_descriptor.xml
  1. … 375 more files in changeset.
This change covers three points: 1. move the generation of UUIDs out of parsers 2. fixes a bug in the metadata lock when performing load operations 3. the external data scan operator is again a leaf operator 4. made sure the startup call doesn't return until the servers are ready to receive requests Merged master into branch for the second time.

The following commits from your working branch will be included:

commit da929c990249fe7f0699417fd7f7ea34908355af

Author: Abdullah Alamoudi <bamousaa@gmail.com>

Date: Fri Jan 16 11:51:14 2015 +0300

Moved generation of UUIDs out of parsers

Change-Id: I32437493cf48c79417f4280e9d8a85cfb1559b62

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/205

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

  1. … 60 more files in changeset.
- Fixed Type Casting issue - Reorganized duplicated internal class in the DelimitedDataParser and DelimitedDataParserFactory - Prevented a user from creating an inverted index on a dataset with a variable-length PK - INT64 is now the default type - Issue 852 fixed

Change-Id: I2d71e8a21da4f709c3259a3d3f678c640f9e1160

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/192

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 847 more files in changeset.
Issue 804: Spell "Adapter" consistently

Change-Id: I3141c1ddd2357ba8dd8ad94d6b98abc78151cd72

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/148

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 35 more files in changeset.
Implement TokenizeOperator in addition to the changes made by Zachary Heilbron. Now, the compiler builds the bulk-load plan. For other details, please refer to the corresponding Hyracks changes.

Change-Id: I646539d88ea2bdc6da4fbf2b6e9460a6189125ff

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/79

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 116 more files in changeset.
apply code conventions to files containing tabs

Change-Id: I429b02253a5008c50540a79075c257d01bc39a99

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/112

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 12 more files in changeset.
added test for installation and use of an external adaptor

  1. … 12 more files in changeset.
added an additional test for feed ingestion

  1. … 13 more files in changeset.
modified test library descriptor xml to account for schema change

miscellaneous/minor changes + support for installing adapters

  1. … 99 more files in changeset.
checkpoint

  1. … 54 more files in changeset.
pending changes

  1. … 1 more file in changeset.
added external UDF that initializes it with a dictionary as part of initialization and uses it in subsequent invocations

    • -0
    • +189
    ./resources/data/countriesCapitals.txt
  1. … 7 more files in changeset.