Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix for ASTERIXDB-1738: Change feed fails to delete record with meta PK

Change-Id: I5ce8d9c69f96593ee305cfad4b44e486c9f1d6cc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1351

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 8 more files in changeset.
remove dead code and curly braces in switches

Change-Id: I9b23c3e8acd38e3a3ffce5da797f57c064b667c4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1370

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -97
    • +47
    ./apache/asterix/external/parser/ADMDataParser.java
  1. … 1 more file in changeset.
ASTERIXDB-1711: remove some more Aql-prefixes

Change-Id: I041341fe76e9338b7c568a9eacaf826f8fa0ed46

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1360

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 221 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 251 more files in changeset.
Cleanup FileSplit and FileReference

This change gives FileSplit and FileReference specific meaning to

avoid confusion of an absolute vs relative, local vs global, inside

an IO device vs outside IO devices.

In addition, it enables better abstraction of global partitions and

delegate the responsibility of choosing which partition goes to which

IO device to the IO Manager through the introduction of FileDeviceComputer

In details:

Previously, the LocalResource in Hyracks had partition (storage partition)

and there is no such thing in Hyracks. This scope leak is bad. In addition

The local resource had a name and a path. they were always the same and so

the name was removed.

The storage partition was instead moved to asterixdb implementation of the

serialized object in the local resource.

With all of these changes, the cluster controller (compiler) only needs to

know about partitions and relative paths. It doesn't need to worry about

heterogenous Node setups and different io device configurations. For File

assignment to IO devices, a new interface (IFileDeviceComputer) was

introduced which can be overriden by applications to have their own

strategy for distributing files among IO devices.

Change-Id: I4fac508bf9af5a3bed41a3cf4464d2cbfecf2f61

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1352

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 276 more files in changeset.
Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one

more case for orderedList.

2. For OrderedList bug, change UnorderedListBuilder to

OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1339

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -29
    • +70
    ./apache/asterix/external/parser/TweetParser.java
Unify runtime type exceptions by using error code and message template.

-- fixed string_join to be able to handle self-described list;

-- fixed the input arity of day_of_week;

-- added tests for exceptions.

RuntimeDataException is a newly added super class for errors in the data in the runtime.

It has the following subclasses:

-- IncompatibleTypeException, e.g., 1 + "2"

-- InvalidDataFormatException, e.g., boolean(1)

-- OverflowException, e.g., int8(1024)

-- TypeMismatchException, e.g., substr(1, 2)

-- UnderflowException, e.g., int8(-1024)

-- UnsupportedItemTypeException, e.g., string_concat(["a", 1, "c"])

-- UnsupportedTypeException, e.g., rectangle("1,2 3,4") + rectangle("2,5 7,8")

All "root-cause" exceptions thrown from runtime functions have an error code.

Going forward, all "root-cause" exceptions thrown from asterixdb

should have an error code.

Change-Id: Ie4fff8f5e64ffb027910a4899c0246b37ed5bce7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1313

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 431 more files in changeset.
Fix Upsert Pipeline

Change-Id: I5c19d448f9664ecaeac600668a6dbdcf40673c56

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1308

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 10 more files in changeset.
Add user-stream for Twitter Adaptor

1. Add user-stream option for Twitter Adaptor

2. Refactor part of TwitterRecordReaderFactory

3. To create a user-stream feed, using following ddl:

create feed TwitterFeed using twitter_user_stream(

("format"="twitter-status"),

("type-name"="Tweet"),

...

// rest is same as push feed

Change-Id: I99cdd4cb667306d378317616f9811dfce3e6d838

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1272

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +193
    ./apache/asterix/external/util/TwitterUtil.java
Enhanced Insert AQL

The optional "as Variable" provides a variable binding for the inserted records

The optional "returning Query" allows users to run simple

queries/functions on the records returned by the insert, and can refer

to the variable bound in "as Variable"

Allow commits to be non-sink operators (contnue job pipeline after commit)

Additionally, this change makes small modifications to

the extension code to prepare for the BAD extension

Also made the OptimizerTests able to work for Extensions

Change-Id: I65789d2a861d15232dd29156a6987d0635ec6c94

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1150

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +27
    ./apache/asterix/external/feed/watch/FeedActivityDetails.java
  1. … 74 more files in changeset.
More Dependency Cleanup

Change-Id: I9cd6f1c9829c1a71f8b10994823a5f1fbe09d797

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1294

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 7 more files in changeset.
Move Twitter Firehose Datasource to Test Source Folder

Change-Id: Iefe2130707012b8ce60f5dfac96635a1a515a076

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1290

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -1189
    • +0
    ./apache/asterix/external/util/DataGenerator.java
    • -154
    • +0
    ./apache/asterix/external/util/TweetGenerator.java
  1. … 31 more files in changeset.
ASTERIXDB-1589, ASTERIXDB-1627 addition and fix.

- Add support for proxy when using TwitterFeed.

- Fix corrupted polygon values when created from open list of doubles.

- Add regression test for ASTERIXDB-1627.

Change-Id: Ia8bdd0fbb63310212364ab9e52d34fec675a4c71

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1261

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -11
    • +27
    ./apache/asterix/external/util/TwitterUtil.java
  1. … 6 more files in changeset.
Run Active Shutdown On Separate Thread

- As feed shutdown can be slow, do it on another thread to not tie up

worker.

- use nc thread executor for feed adapter thread

- error handling

Change-Id: I8fd9bc454b290420682160364ac78e4b91a9abc3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1223

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 3 more files in changeset.
Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for

feed configuration, which changes the default record

insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1068

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 14 more files in changeset.
Separate static cluster properties from runtime state

This change includes the following:

- Remove cluster static properties from AsterixClusterProperties.

- Add new ClusterProperties class for static cluster properties.

- Rename AsterixClusterProperties to ClusterStateManager.

- Refactor cluster static properties.

Change-Id: I3da72f639476f081ba8a0ef21ec176d32ff8bdec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1183

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 35 more files in changeset.
avoid some exception wrapping

Change-Id: Ia8fff04c314eb146ce0a73beb0b8ec412cb2d280

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1179

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 13 more files in changeset.
Small Cleanup

Change-Id: Ib527f2eee283faf75c04323b6961c31a6d739d07

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1145

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -4
    • +10
    ./apache/asterix/external/api/IAdapterFactory.java
  1. … 5 more files in changeset.
Refactor Messaging

In this refactoring, each message implementation includes

a handle method. This avoids bloating of message brokers

and enable better extensibility for messaging.

Change-Id: I7c918bf504058c98ecf89f5b019503278e9aa01f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1128

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 112 more files in changeset.
Refactor Active Listeners

Change-Id: I260c8608329523f56dc54780d87d796f838505cf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1118

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 24 more files in changeset.
Introduce MessagingNetworkManager for NC2NC AppMessaging

This change introduces MessagingNetworkManager to NodeControllerService.

The MessagingNetworkManager is used to open channels that are used for NC2NC

application messaging. The read and write interfaces of the messaging channels

are set by a MessagingChannelInterfaceFactory which is set by the application.

Change-Id: I5c0bd7c11c1e78954ebceff49cb274d8073a64bd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/897

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 74 more files in changeset.
TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1002

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -83
    • +208
    ./apache/asterix/external/parser/TweetParser.java
    • -14
    • +19
    ./apache/asterix/external/util/TwitterUtil.java
  1. … 6 more files in changeset.
Add Asterix Extension Manager

More extension support is added. A user can now provide implementations

for the IExtension interface which will give them more control over

the behavior of the system and give them the ability to add custom

features.

Initial customizations include:

1. Metadata Tuple Translators

2. Metadata Datasets

3. Query Translators

4. Statement Handlers

5. Lang Compilation Provider

Change-Id: I280268495cc3aad00f898cba21f7299f7120ce5c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1017

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 218 more files in changeset.
ASTERIXDB-1478: fix the utf8 reader.

1. Fix the ASTERIXDB-1478.

2. Add the utf8 testCases.

Change-Id: Idb302dc604fcd71811de550d3d4bd727c81a13ee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1077

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 7 more files in changeset.
small refactoring

Change-Id: I37eab1645416e3aad6119bba527c5e3b4b98fddc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1052

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Fix Synchronization Issue in FeedRecordDataflowController

Change-Id: Ia290499132c320ce99402e698c76ba6d944f8f3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1051

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

small ADMParser/AdmLexer cleanup

- do not declare IOExceptions that are not thrown

- improve formatting of lexer template

Change-Id: Id032f0c83ebef108a9eddb7242c586d5481217e4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1032

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.
Add Maven Plugin for Grammar Extension

This change introduces a new asterix maven plugin for grammar

extension. The plugin takes a base grammar file and an extension

file and combine them to generate an extended grammar. A base

grammar can be extended by overriding production nodes, adding

extending existing nodes, or introducing new terminals and

non terminals.

In addition, a mvn verify goal was added to generate the parser

java files from the grammar generated by the test.

Change-Id: Iaa2d11782d43dd8f27d69e347ed0fc8797d79dad

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1011

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 56 more files in changeset.
Cast Record Before Insert for Open Indexes

This change modifies the insert plan to perform cast for open

indexes before inserting to primary index. This avoids modification

of indexes and then doing complex work to try and undo the insert.

In addition, feeds can simply log and remove tuples which have

open fields of incompatible type.

Change-Id: I0eef5813ddbfe1b9c518cd7f92d37c95e8486914

Reviewed-on: https://asterix-gerrit.ics.uci.edu/997

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 30 more files in changeset.
Add Test NodeController, Test Data Generator, and Marker Logs

This test enable creating a node controller for unit test purposes.

The Node controller is identical to the regular node controller

except that it doesn't communicate with a cluster controller at all.

In this change, Test Data Generator is introduced which should

facilitate writing unit test cases which requires data generation.

The change also includes enabling feeds to send progress data. progress

information can then be sent through the pipeline and persisted in the

transaction logs and primary index component. A Unit test case has

been created to test adding progress markers to logs and index

components and then reading them.

The last part of this change is the addition of marker logs and their

callbacks. They enable components to create arbitrary logs and get a

callback when they are written to the transaction logs. Initial set of

unit tests were added for marker logs.

Change-Id: I3b9aa8de758b7d26ca34868b16e5ce693e0c0243

Reviewed-on: https://asterix-gerrit.ics.uci.edu/962

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +35
    ./apache/asterix/external/api/IFeedMarker.java
  1. … 68 more files in changeset.