Clone Tools
  • last updated 11 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2008][CLUS] Only add pending removal if node known

[ASTERIXDB-2023][ING] Introduce Enums instead of using bytes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Only nodes which are known to cluster manager are added

to the list of nodes pending removal. Other nodes are ignored

- Enums introduced:

- ActiveEvent.Kind

- ActivePartitionMessage.Event

- Remove AdapterRuntimeManager

- Remove AdapterExecutor

Change-Id: I7044896559798426c04a3f46861bc5335b25d140

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 88 more files in changeset.
[ASTERIXDB-1992][ING] Suspend/Resume active entities

- user model changes: wait for completion only returns

when the entity becomes inactive.

- storage format changes: no

- interface changes:

01) Introduce:

IMetadataLockManager for entity locking.

IActiveNotificationHandler for handling active events.

IRetryPolicy for recovery of failed active jobs.

IActiveEntityController for controlling active entities.

02) IJobLifecycleListener.notifyJobFinish now passes the

JobStatus and Exceptions.

03) IActiveEntityEventsListener.isActive() returns true,

if entity is active, false otherwise.

04) IActiveEntityEventsListener.unregister() removes the

listener upon entity deletion.

05) IActiveEntityEventsListener.getJobFailure() returns

the job failure if the entity is in a failed state.

06) IStatementExecutor.getComponentProvider() returns

the storage component provider.

07) IStatementExecutor.getApplicationContext() returns

the application context.

08) IMetadataManager.upsertEntity to perfrom metadata

entities' upsert operations.

09) IMetadataNode.upsertEntity to perfrom metadata

entities' upsert operations

10) ICcApplicationContext.getMetadataLockManager() returns

the lock manger.

11) ICcApplicationContext.getClusterStateManager() returns

the cluster state manager.

details:

- Starting and stopping of active entities now go through

their listeners rather than having the listener indicating

active entity and not having it indicating inactive entity.

This facilitates suspend/resume operations of long running

active jobs for the sake of DDL operations or topology

changes.

- Unit tests for the vast majority of code paths and

different possible scenarios have been added.

Change-Id: Ifeac8c73e6bad39a13663b84a52121356e3c6b40

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1875

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 130 more files in changeset.
[ASTERIXDB-2000][RT] Fix roundtrip-ability for INF in JSON format.

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Use INF to represent infinity.

Change-Id: I13687822433744afe1ad3439029b2c0887563f9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1901

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -2
    • +14
    ./apache/asterix/external/parser/ADMDataParser.java
  1. … 34 more files in changeset.
[ASTERIXDB-1989] Move to com.rometools version of Rome

Change-Id: Ia8dd896f6abe34532f2cce70a4fc7632d7a21910

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1886

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
[ASTERIXDB-1950][ING][API] Add IT for ActiveEntity stats

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

1. Added one IT for feed stats request API.

2. Minor tweak to the API to produce proper JSON.

Change-Id: Iecbe00bcd93dcfd4ef9b800ce5719e85ae2cfd42

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1862

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 13 more files in changeset.
[ASTERIXDB-1970][ING] Fix Active Stats Test

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Active Stats Test fails because it issues many async

calls and doesn't wait until the calls complete. In

addition, it bypass the active event inbox incorrectly.

Change-Id: I518a6b1f7d8e86703ee5537869d207e609a7c293

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1865

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 5 more files in changeset.
[ASTERIXDB-1950][ING][API] REST API for ActiveEntity stats

- user model changes: yes

Added ActiveEntity stats REST API.

- storage format changes: no

- interface changes: yes

Changed ActiveEntityEventsListener & EventSubscrber interfaces.

Added getStats method to IActiveRuntime.

Details:

1. Added HttpAPI for active feed stats.

2. Replaced FeedEventSubscriber with WaitForStateSubscriber.

3. Added StatsSubscriber for monitoring stats request.

4. Moved the message related methods from FeedEventsListener to

ActiveEntityEventsListener for possible reuses in other cases.

Change-Id: I46b48b52a1c9906510c5bdce778d1672845f75ca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1748

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +56
    ./apache/asterix/external/feed/watch/AbstractSubscriber.java
    • -0
    • +43
    ./apache/asterix/external/feed/watch/StatsSubscriber.java
  1. … 27 more files in changeset.
[ASTERIXDB-1443][FEED] Remove Frame Distributor

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- FrameDistributor and DistributeFeedFrameWriter are not used

anymore.

Change-Id: I27c1ff99ce797923dd709d181387560e4f9448a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1853

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Change DataflowHelperFactory not to require Task Context

Change-Id: I9dcd95dbefca131c4bbdb43306f00f6f8ea60800

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1758

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 16 more files in changeset.
Rename TaskUtil.getInSharedMap(..) -> TaskUtil.get(..).

Change-Id: Ic795c988ca63db7c760a2c8e7638451945cb72aa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1817

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 6 more files in changeset.
Clean up numeric type conversion code

- Add new conversion methods to ITypeConvertorComputer interface

- Move remaining type conversion code from ATypeHierarchy

to ITypeConvertorComputer implementations

- Add type demotion parameter to chose whether it fails

if input value is out of range for the target type ('strict' mode)

or not ('lax' mode)

- Clean up error messages

Change-Id: I68b78e2dc9ebf78799698a1c90bcd7cb5150137d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1800

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 27 more files in changeset.
Remove unnecessary integer boxing in AInt*

Change-Id: I6ecefc4e8617371de00fb430edfe24fca2b1f92f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1797

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 10 more files in changeset.
Fixed race condition during ncMap lookup

Change-Id: I1bfbe712c100f48011a516c373ac8994028dc3dd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1792

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -44
    • +14
    ./apache/asterix/external/util/NodeResolver.java
  1. … 1 more file in changeset.
Fix uniontype compatibility issue in TweetParser

1. Fix compatibility issue of UnionType in TweetParser.

2. Add unit test for TweetParser, covering missing attribute record

filtering.

Change-Id: Id7e316d20f929236b3fa17618bb8f87d28ac7efc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1786

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -16
    • +29
    ./apache/asterix/external/parser/TweetParser.java
  1. … 3 more files in changeset.
Reduce exception wrapping for casting and printing

Change-Id: I1911125b2fa93b743bea8cd16211e3bd464dbf1e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1782

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 29 more files in changeset.
Fix upsert deadlock and upsert with filtered primary only

This change fixes a deadlock that happens when 3 operations

an upsert, a search and a flush happen simulteniously.

If all the memory components are full, the upsert

gets blocked, the upsert could've obtained a lock on the

search key which would block the search not allowing it

to exit the components and not allowing the components

to be cleared and reused.

In addition, the change refactors common LSM index code.

Change-Id: I93fac0f27ab0b3cc071ff38aef90d850cbbce488

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1762

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 84 more files in changeset.
Feed Adapter and RecordReader Clean

1. Cleaned DatasourceFactoryProvider with service interface

2. Cleaned StreamRecordReaderProvider with service interface

3. Delayed the Inputstream binding to configuration phase.

4. Added one simple test case for StreamRecordReaderProvider.

5. Moved twitter library check into configure method.

6. Refactored parser interfaces and variable names to make them

consistent with the record reader intreface.

Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1535

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 18 more files in changeset.
Add Error code for unknown pending operation

Change-Id: I41d0bed5afdfee9b2c29d0218e794247f13e6350

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1739

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +124
    ./apache/asterix/external/operators/ExternalFilesIndexModificationOperatorDescriptor.java
  1. … 557 more files in changeset.
Rename several type tags to improve error messages:

- orderedlist -> array

- unorderedlist -> multiset

- record -> object

- int8 -> tinyint

- int16 -> smallint

- int32 -> integer

- int64 -> bigint

Change-Id: I5a2eb6534f80af102832f079dfcb92fab5fc34fa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1724

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -23
    • +23
    ./apache/asterix/external/parser/ADMDataParser.java
  1. … 150 more files in changeset.
Carry filter in 2ndary-to-primary index search

Change-Id: I287f1dbd230aa649f1350114abf0a1d47e2bb53c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1720

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 55 more files in changeset.
Fix continue ingestion bug when exception happens

1. Fix the bug when exception happens, localfs adapter couldn't pick up

a new file to continue the ingestion.

2. Change the exception handling from string to error code.

3. Added a test case.

4. Removed some tailing . in error message.

Change-Id: Ie8656a4d1afabbc1b481eb97509a861b22478676

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1713

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 11 more files in changeset.
Fix NULL/MISSING field value handing in Java UDF

Add NULL/MISSING value support in Java UDF. This fixed the NPE when there

is a NULL/MISSING field in a record.

Change-Id: Icf110b2f5b3658847967f5ee7163fb2e47713813

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1695

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Remove static cc application context instance

Change-Id: Ia2e250405967ec880e7af6387aa981f39b3392c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1606

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 167 more files in changeset.
Cleanup storage exceptions

Change-Id: I020d2b4b1f4ae48fc2df0b720e70a1ce95867d34

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1619

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 185 more files in changeset.
ASTERIXDB-1883: FeedRuntimeInputHandler issues

Recent commit https://asterix-gerrit.ics.uci.edu/#/c/1591/ includes a

number of new issues in FeedRuntimeInputHandler:

- hangs caused by race condition with mutex & inbox on close (observed

on Jenkins)

- CPU spin on disk spilling on empty inbox

- The writer is not flushed in as many cases as before

Change-Id: I7e091f65eb5f3a76277803b3197d490d3ef2fc04

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1677

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
Fix ASTERIXDB-1690

Fix the deadlock problem ASTERIXDB-1690 in FileSystemWatcher

Change-Id: Iad358fdeeb47f5d5884fed8806a234f8b3196bec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1393

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 1 more file in changeset.
Feed Policy Refactoring

1. Fix the framesize inconsistency in feed dataflow.

2. Add a runtime test case for create feed with policy.

3. Fix the FeedRuntimeInputHandler flush() logic. Only flush after the

writer is opened.

4. Refactor FeedPolicyAccessor and BuiltinPolices. Now we only have

spill and discard polices.

5. Remove PolicyEnforcer. Merge the functionality into

FeedPolicyAccessor.

6. Revise SocketServerInputStream. Make the expected exception more

friendly.

7. Fixed one test case fail in change feed.

8. Refactor FeedRuntimeInputHandler consumer thread logic. Change the

poison consumer thread mechanism.

Change-Id: Ibc10139925cfedee66d1263990ba80b94675f182

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1591

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 20 more files in changeset.
Fix record loss for certain certain feed type

1. Fix blindly replace connector between FeedCollector and

AssignOperator.

2. Wrap AssignOperator into the FeedMetaOperator to make sure the

operators inside (udf, accessor, etc.) can handle messages in the

feed workflow.

3. Revise feed connection job merge function.

4. Test case fix.

Change-Id: I4e1f7dd3621482a11feb675a93b826ae2cb965a7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1652

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 5 more files in changeset.
ASTERIXDB-1859 Revise create feed exception.

1. Fixed unknown feed datatype exception.

2. Revised SocketServerInputStreamFactory exception handling.

Change-Id: I4d04b3031a3457e4d04c2b921a4db3fd7f1cbed0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1621

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

  1. … 3 more files in changeset.