Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix upsert deadlock and upsert with filtered primary only

This change fixes a deadlock that happens when 3 operations

an upsert, a search and a flush happen simulteniously.

If all the memory components are full, the upsert

gets blocked, the upsert could've obtained a lock on the

search key which would block the search not allowing it

to exit the components and not allowing the components

to be cleared and reused.

In addition, the change refactors common LSM index code.

Change-Id: I93fac0f27ab0b3cc071ff38aef90d850cbbce488

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1762

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 84 more files in changeset.
Feed Adapter and RecordReader Clean

1. Cleaned DatasourceFactoryProvider with service interface

2. Cleaned StreamRecordReaderProvider with service interface

3. Delayed the Inputstream binding to configuration phase.

4. Added one simple test case for StreamRecordReaderProvider.

5. Moved twitter library check into configure method.

6. Refactored parser interfaces and variable names to make them

consistent with the record reader intreface.

Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1535

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 18 more files in changeset.
Add Error code for unknown pending operation

Change-Id: I41d0bed5afdfee9b2c29d0218e794247f13e6350

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1739

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +124
    ./apache/asterix/external/operators/ExternalFilesIndexModificationOperatorDescriptor.java
  1. … 557 more files in changeset.
Rename several type tags to improve error messages:

- orderedlist -> array

- unorderedlist -> multiset

- record -> object

- int8 -> tinyint

- int16 -> smallint

- int32 -> integer

- int64 -> bigint

Change-Id: I5a2eb6534f80af102832f079dfcb92fab5fc34fa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1724

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -23
    • +23
    ./apache/asterix/external/parser/ADMDataParser.java
  1. … 150 more files in changeset.
Carry filter in 2ndary-to-primary index search

Change-Id: I287f1dbd230aa649f1350114abf0a1d47e2bb53c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1720

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 55 more files in changeset.
Fix continue ingestion bug when exception happens

1. Fix the bug when exception happens, localfs adapter couldn't pick up

a new file to continue the ingestion.

2. Change the exception handling from string to error code.

3. Added a test case.

4. Removed some tailing . in error message.

Change-Id: Ie8656a4d1afabbc1b481eb97509a861b22478676

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1713

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 11 more files in changeset.
Fix NULL/MISSING field value handing in Java UDF

Add NULL/MISSING value support in Java UDF. This fixed the NPE when there

is a NULL/MISSING field in a record.

Change-Id: Icf110b2f5b3658847967f5ee7163fb2e47713813

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1695

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Remove static cc application context instance

Change-Id: Ia2e250405967ec880e7af6387aa981f39b3392c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1606

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 167 more files in changeset.
Cleanup storage exceptions

Change-Id: I020d2b4b1f4ae48fc2df0b720e70a1ce95867d34

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1619

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 185 more files in changeset.
ASTERIXDB-1883: FeedRuntimeInputHandler issues

Recent commit https://asterix-gerrit.ics.uci.edu/#/c/1591/ includes a

number of new issues in FeedRuntimeInputHandler:

- hangs caused by race condition with mutex & inbox on close (observed

on Jenkins)

- CPU spin on disk spilling on empty inbox

- The writer is not flushed in as many cases as before

Change-Id: I7e091f65eb5f3a76277803b3197d490d3ef2fc04

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1677

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
Fix ASTERIXDB-1690

Fix the deadlock problem ASTERIXDB-1690 in FileSystemWatcher

Change-Id: Iad358fdeeb47f5d5884fed8806a234f8b3196bec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1393

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 1 more file in changeset.
Feed Policy Refactoring

1. Fix the framesize inconsistency in feed dataflow.

2. Add a runtime test case for create feed with policy.

3. Fix the FeedRuntimeInputHandler flush() logic. Only flush after the

writer is opened.

4. Refactor FeedPolicyAccessor and BuiltinPolices. Now we only have

spill and discard polices.

5. Remove PolicyEnforcer. Merge the functionality into

FeedPolicyAccessor.

6. Revise SocketServerInputStream. Make the expected exception more

friendly.

7. Fixed one test case fail in change feed.

8. Refactor FeedRuntimeInputHandler consumer thread logic. Change the

poison consumer thread mechanism.

Change-Id: Ibc10139925cfedee66d1263990ba80b94675f182

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1591

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 20 more files in changeset.
Fix record loss for certain certain feed type

1. Fix blindly replace connector between FeedCollector and

AssignOperator.

2. Wrap AssignOperator into the FeedMetaOperator to make sure the

operators inside (udf, accessor, etc.) can handle messages in the

feed workflow.

3. Revise feed connection job merge function.

4. Test case fix.

Change-Id: I4e1f7dd3621482a11feb675a93b826ae2cb965a7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1652

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 5 more files in changeset.
ASTERIXDB-1859 Revise create feed exception.

1. Fixed unknown feed datatype exception.

2. Revised SocketServerInputStreamFactory exception handling.

Change-Id: I4d04b3031a3457e4d04c2b921a4db3fd7f1cbed0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1621

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 3 more files in changeset.
AsterixDB Rat Execution Audit & Fixes

Change-Id: I3d771c5b63eaef4b6a0b4da769039ed5bca17736

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1609

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 493 more files in changeset.
Prevent hangs on active runtime stop

Change-Id: I2e60f633cac8e835dcc7211e87d104ecbb8947b0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1608

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Expose TwitterFirehose Stream to Users

1. Moved TwitterFirehoseStream from Test to external-data which enables

users to access it from interface.

2. Related test cases fix.

3. Code style fix in DataGenerator, TweetGenerator, etc.

Change-Id: I8f899fde0d9a9dbd7891957be2f1241d5acfc4ab

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1590

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +1209
    ./apache/asterix/external/generator/DataGenerator.java
    • -0
    • +154
    ./apache/asterix/external/generator/TweetGenerator.java
  1. … 12 more files in changeset.
Feed Exception Revise

1. Add compliation exception for missing Twitter4J library. Try to

expose this exception when create feed.

2. Tweaked 'invalid feed parameter exception'. Expose detailed exception

information to front end.

3. Related test cases fix.

Change-Id: Iee65b2a5365067c9ba2aa1615ac23f0ace41b51e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1534

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 7 more files in changeset.
Unify structure of Cluster Controller and Node Controller

Change-Id: Ife3a002371a07ee9cdd32e1ffd50cc775bf1d453

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1514

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 152 more files in changeset.
Cleanup and bug fixes in Feeds pipeline

The bug fix is for the MessagingFrameTupleAppender. It used

to consume one extra byte per frame.

Cleanups are for feeds. These include:

1. Remove the use of feed dataflow marker. Feeds which

need to send progress can and should do that without

a marker thread.

2. Lock the memory component for feed commit to be able

to add information to the memory component's metadata

page safely.

In addition, this change introduces a frame level callback

for index operations.

Change-Id: Ie97b2133ebecb7380cf0ba336e60ed714d06f8ee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1523

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -10
    • +26
    ./apache/asterix/external/api/IAdapterFactory.java
  1. … 59 more files in changeset.
Enable Feed Changes to work with BAD project

Extracts the ActiveListener

Enables listeners to survive after job executions

Change-Id: Ib62184b67aff564475ef9b58790ff96409195b77

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1524

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 1 more file in changeset.
ASTERIXDB-1791, ASTERIXDB-1796: fix failure handling in runtime operators.

This change includes the following parts:

- Fix the implementation of fail() and close() in several runtime operators

to avoid file handle leak and job hang;

- Add an erase method to RunFileWriter which closes files before deleting

them in order release the holding disk space;

- Call RunFileWriter.close() and RunFileReader.close() in "finally" blocks.

- Fix RunFileReader to not truncate files to be deleted - it is not the root

cause of un-released disk space - open deleted files are the root cuase;

- Check file handle leaks in LangExecutionUtil.tearDown().

Change-Id: I203168171e6dac16b57d2eda960823e3810e22a3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1513

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 51 more files in changeset.
Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.

2. FeedConnection metadata dataset is added (pkeys: dataverseName,

feedName, datasetName).

3. Replaced the old intake job + collect job combination with one single

job using SplitOperator.

4. Now one feed can connect to multiple datasets.

5. The disconnect feed job is replaced by ActiveManagerMessage.

6. The new feed life cycle is:

- Create feed

- Connect feed to dataset0, dataset1, dataset2, etc.

- Start feed

- Stop feed

- Disconnect feed

7. New feedEventListner framework by Abdullah

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1259

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -6
    • +0
    ./apache/asterix/external/feed/api/IFeed.java
  1. … 183 more files in changeset.
Change name resolution order to Node name -> DNS

Change-Id: I2bbfc8d9d8469c444f6c70bd3a77654892aeb42f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1491

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -24
    • +12
    ./apache/asterix/external/util/NodeResolver.java
  1. … 2 more files in changeset.
Enable commit runtime extension

Change-Id: I98083ea5e93cb5f45d92c5dfbacfee1020fad57a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1485

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 24 more files in changeset.
Improve reading from and writing to Metadata pages

This change introduces a new interface IComponentMetadata.

Each LSM component is associated with a metadata object

which can be used to read and write arbitrary data to

the metadata pages of components. When flushing a

component, data in its metadata component is automatically

flushed to the disk component. For merge operations,

the IO Callback is responsible for merging the components'

metadata pages.

Change-Id: Id95ef33c0a0bc1abb3fc3ecdea5611ee4acd6dfa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1476

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Ian Maxon <imaxon@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 152 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 413 more files in changeset.
Subset hive-exec, make twitter4j a provided dep.

Change-Id: Iee4276f540ec8552181bfb452882654b5faa17df

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1427

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 14 more files in changeset.
Load parser from classpath in ParserFactoryProvider

Change the behavior of ParserFactoryProvider to allow it loads parser

from classpath.

Change-Id: I2ac039fe3daaf0636cf004289bd0c8a3229197a9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1416

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 5 more files in changeset.