Clone Tools
  • last updated 14 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix record loss for certain certain feed type

1. Fix blindly replace connector between FeedCollector and

AssignOperator.

2. Wrap AssignOperator into the FeedMetaOperator to make sure the

operators inside (udf, accessor, etc.) can handle messages in the

feed workflow.

3. Revise feed connection job merge function.

4. Test case fix.

Change-Id: I4e1f7dd3621482a11feb675a93b826ae2cb965a7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1652

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 5 more files in changeset.
ASTERIXDB-1859 Revise create feed exception.

1. Fixed unknown feed datatype exception.

2. Revised SocketServerInputStreamFactory exception handling.

Change-Id: I4d04b3031a3457e4d04c2b921a4db3fd7f1cbed0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1621

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 3 more files in changeset.
AsterixDB Rat Execution Audit & Fixes

Change-Id: I3d771c5b63eaef4b6a0b4da769039ed5bca17736

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1609

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 493 more files in changeset.
Prevent hangs on active runtime stop

Change-Id: I2e60f633cac8e835dcc7211e87d104ecbb8947b0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1608

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Expose TwitterFirehose Stream to Users

1. Moved TwitterFirehoseStream from Test to external-data which enables

users to access it from interface.

2. Related test cases fix.

3. Code style fix in DataGenerator, TweetGenerator, etc.

Change-Id: I8f899fde0d9a9dbd7891957be2f1241d5acfc4ab

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1590

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +1209
    ./apache/asterix/external/generator/DataGenerator.java
    • -0
    • +154
    ./apache/asterix/external/generator/TweetGenerator.java
  1. … 12 more files in changeset.
Feed Exception Revise

1. Add compliation exception for missing Twitter4J library. Try to

expose this exception when create feed.

2. Tweaked 'invalid feed parameter exception'. Expose detailed exception

information to front end.

3. Related test cases fix.

Change-Id: Iee65b2a5365067c9ba2aa1615ac23f0ace41b51e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1534

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 7 more files in changeset.
Unify structure of Cluster Controller and Node Controller

Change-Id: Ife3a002371a07ee9cdd32e1ffd50cc775bf1d453

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1514

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 152 more files in changeset.
Cleanup and bug fixes in Feeds pipeline

The bug fix is for the MessagingFrameTupleAppender. It used

to consume one extra byte per frame.

Cleanups are for feeds. These include:

1. Remove the use of feed dataflow marker. Feeds which

need to send progress can and should do that without

a marker thread.

2. Lock the memory component for feed commit to be able

to add information to the memory component's metadata

page safely.

In addition, this change introduces a frame level callback

for index operations.

Change-Id: Ie97b2133ebecb7380cf0ba336e60ed714d06f8ee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1523

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -10
    • +26
    ./apache/asterix/external/api/IAdapterFactory.java
  1. … 59 more files in changeset.
Enable Feed Changes to work with BAD project

Extracts the ActiveListener

Enables listeners to survive after job executions

Change-Id: Ib62184b67aff564475ef9b58790ff96409195b77

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1524

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 1 more file in changeset.
ASTERIXDB-1791, ASTERIXDB-1796: fix failure handling in runtime operators.

This change includes the following parts:

- Fix the implementation of fail() and close() in several runtime operators

to avoid file handle leak and job hang;

- Add an erase method to RunFileWriter which closes files before deleting

them in order release the holding disk space;

- Call RunFileWriter.close() and RunFileReader.close() in "finally" blocks.

- Fix RunFileReader to not truncate files to be deleted - it is not the root

cause of un-released disk space - open deleted files are the root cuase;

- Check file handle leaks in LangExecutionUtil.tearDown().

Change-Id: I203168171e6dac16b57d2eda960823e3810e22a3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1513

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 51 more files in changeset.
Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.

2. FeedConnection metadata dataset is added (pkeys: dataverseName,

feedName, datasetName).

3. Replaced the old intake job + collect job combination with one single

job using SplitOperator.

4. Now one feed can connect to multiple datasets.

5. The disconnect feed job is replaced by ActiveManagerMessage.

6. The new feed life cycle is:

- Create feed

- Connect feed to dataset0, dataset1, dataset2, etc.

- Start feed

- Stop feed

- Disconnect feed

7. New feedEventListner framework by Abdullah

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1259

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -6
    • +0
    ./apache/asterix/external/feed/api/IFeed.java
  1. … 183 more files in changeset.
Change name resolution order to Node name -> DNS

Change-Id: I2bbfc8d9d8469c444f6c70bd3a77654892aeb42f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1491

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -24
    • +12
    ./apache/asterix/external/util/NodeResolver.java
  1. … 2 more files in changeset.
Enable commit runtime extension

Change-Id: I98083ea5e93cb5f45d92c5dfbacfee1020fad57a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1485

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 24 more files in changeset.
Improve reading from and writing to Metadata pages

This change introduces a new interface IComponentMetadata.

Each LSM component is associated with a metadata object

which can be used to read and write arbitrary data to

the metadata pages of components. When flushing a

component, data in its metadata component is automatically

flushed to the disk component. For merge operations,

the IO Callback is responsible for merging the components'

metadata pages.

Change-Id: Id95ef33c0a0bc1abb3fc3ecdea5611ee4acd6dfa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1476

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Ian Maxon <imaxon@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 152 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 413 more files in changeset.
Subset hive-exec, make twitter4j a provided dep.

Change-Id: Iee4276f540ec8552181bfb452882654b5faa17df

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1427

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 14 more files in changeset.
Load parser from classpath in ParserFactoryProvider

Change the behavior of ParserFactoryProvider to allow it loads parser

from classpath.

Change-Id: I2ac039fe3daaf0636cf004289bd0c8a3229197a9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1416

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 5 more files in changeset.
Runtime ErrorCode fix in external data

1. Refactored most of the exceptions in external-data package to use

ErrorCode.

2. This change only covers RuntimeDataException/HyracksDataExcetion.

3. Refactor errorcode names

Change-Id: Ida810a56bf4aef1394879f088a6a5e8f82c60b74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1374

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 83 more files in changeset.
ASTERIXDB-1714: Eliminate dependency on org.json

Change-Id: Ie9c5400fd134ae75d43385255af7794e968b1c7e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1392

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -99
    • +61
    ./apache/asterix/external/parser/TweetParser.java
  1. … 132 more files in changeset.
ASTERIXDB-1711: remove more mentions of Asterix

Change-Id: I21666fd54075ac56de8aab76bb7d5cdbec020121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1387

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 559 more files in changeset.
Remove Append Only Flag

Change-Id: Id5d6917db8ab29aa01521596f556006e25a502fe

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1385

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 208 more files in changeset.
Fix for ASTERIXDB-1738: Change feed fails to delete record with meta PK

Change-Id: I5ce8d9c69f96593ee305cfad4b44e486c9f1d6cc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1351

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 8 more files in changeset.
remove dead code and curly braces in switches

Change-Id: I9b23c3e8acd38e3a3ffce5da797f57c064b667c4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1370

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -97
    • +47
    ./apache/asterix/external/parser/ADMDataParser.java
  1. … 1 more file in changeset.
ASTERIXDB-1711: remove some more Aql-prefixes

Change-Id: I041341fe76e9338b7c568a9eacaf826f8fa0ed46

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1360

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 221 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 251 more files in changeset.
Cleanup FileSplit and FileReference

This change gives FileSplit and FileReference specific meaning to

avoid confusion of an absolute vs relative, local vs global, inside

an IO device vs outside IO devices.

In addition, it enables better abstraction of global partitions and

delegate the responsibility of choosing which partition goes to which

IO device to the IO Manager through the introduction of FileDeviceComputer

In details:

Previously, the LocalResource in Hyracks had partition (storage partition)

and there is no such thing in Hyracks. This scope leak is bad. In addition

The local resource had a name and a path. they were always the same and so

the name was removed.

The storage partition was instead moved to asterixdb implementation of the

serialized object in the local resource.

With all of these changes, the cluster controller (compiler) only needs to

know about partitions and relative paths. It doesn't need to worry about

heterogenous Node setups and different io device configurations. For File

assignment to IO devices, a new interface (IFileDeviceComputer) was

introduced which can be overriden by applications to have their own

strategy for distributing files among IO devices.

Change-Id: I4fac508bf9af5a3bed41a3cf4464d2cbfecf2f61

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1352

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 276 more files in changeset.
Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one

more case for orderedList.

2. For OrderedList bug, change UnorderedListBuilder to

OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1339

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -29
    • +70
    ./apache/asterix/external/parser/TweetParser.java
Unify runtime type exceptions by using error code and message template.

-- fixed string_join to be able to handle self-described list;

-- fixed the input arity of day_of_week;

-- added tests for exceptions.

RuntimeDataException is a newly added super class for errors in the data in the runtime.

It has the following subclasses:

-- IncompatibleTypeException, e.g., 1 + "2"

-- InvalidDataFormatException, e.g., boolean(1)

-- OverflowException, e.g., int8(1024)

-- TypeMismatchException, e.g., substr(1, 2)

-- UnderflowException, e.g., int8(-1024)

-- UnsupportedItemTypeException, e.g., string_concat(["a", 1, "c"])

-- UnsupportedTypeException, e.g., rectangle("1,2 3,4") + rectangle("2,5 7,8")

All "root-cause" exceptions thrown from runtime functions have an error code.

Going forward, all "root-cause" exceptions thrown from asterixdb

should have an error code.

Change-Id: Ie4fff8f5e64ffb027910a4899c0246b37ed5bce7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1313

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 431 more files in changeset.
Fix Upsert Pipeline

Change-Id: I5c19d448f9664ecaeac600668a6dbdcf40673c56

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1308

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 10 more files in changeset.
Add user-stream for Twitter Adaptor

1. Add user-stream option for Twitter Adaptor

2. Refactor part of TwitterRecordReaderFactory

3. To create a user-stream feed, using following ddl:

create feed TwitterFeed using twitter_user_stream(

("format"="twitter-status"),

("type-name"="Tweet"),

...

// rest is same as push feed

Change-Id: I99cdd4cb667306d378317616f9811dfce3e6d838

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1272

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +193
    ./apache/asterix/external/util/TwitterUtil.java