Clone Tools
  • last updated 10 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2310][STO]Enforce Key Uniquness using PKIndex

- user model changes: no

- storage format changes: yes. Primary key index

now has bloom filters.

- interface changes: no

Details:

- Add bloom filters to primary key index.

- Introduce LSMPrimaryInsertOperator to separate uniqueness check from

the primary index. When the primary key index is available, it will be

used for uniqueness check. This implementation of this operation is

similar to LSMPrimaryUpsertOperator.

Change-Id: I7a52bb75ee5b14521972999df2f45ba62adc5af1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2453

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -33
    • +12
    ./LockThenSearchOperationCallback.java
    • -3
    • +3
    ./PrimaryIndexModificationOperationCallback.java
  1. … 51 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +0
    ./PrimaryIndexModificationOperationCallback.java
  1. … 625 more files in changeset.
[ASTERIXDB-2522][TX] Skip WAIT record during lock conflicts

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- A small optimization to our deadlock-free locking potocol by skipping

the WAIT record. Once the partial frame has been flushed and previous

records are committed, the locks held by the write thread will

eventually be released by the log flusher thread. There is no need to

force the writer thread to wait for the log flusher thread.

Change-Id: I6ef3979d6393d45a6b7b2eb5f09f147299b5cd9f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3232

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -16
    • +3
    ./PrimaryIndexModificationOperationCallback.java
[NO ISSUE][STO] Ensure First Component ID is Initialized

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Initialize the component id generator from the primary

index checkpoint, if exits, as soon as it is created.

- Ensure the first component id is passed to all indexes.

Change-Id: I246f9373f950e2f9a2c63f86746462e42a3f1c62

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2948

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./PrimaryIndexOperationTrackerFactory.java
  1. … 13 more files in changeset.
[NO ISSUE][OTH] Remove Unused Imports

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Iafff39073d0fedaff74a26ef7e3260008a79ff0c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2893

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

    • -1
    • +0
    ./AbstractIndexModificationOperationCallback.java
  1. … 67 more files in changeset.
[ASTERIXDB-1130][STO]: JSON serialization for persisted objects

- user model changes: no

- storage format changes:

This change replaces the use of Java serialization for persisted objects

such as dataset/index metadata, checkpoints, ect...

This will break backward compatibly with any existing AsterixDB instance.

However, the change is needed to enable future backward compatibility support

for persisted objects.

- interface changes:

IJsonSerializable: contains API to serialize a class as a JsonNode.

IPersistedResourceRegistry: contains a mapping between an IJsonSerializable

class and a unique type id. An IPersistedResourceRegistry is responsible

for generating the class identifier in the JSON output.

The class identifier will always contain the following attributes:

@type: a unique type id that identifies the object type.

@version: the version of the serialized class.

@class: the serialized class full name.

Any registered class with PersistedResourceRegistry must provide

a static fromJson(IPersistedResourceRegistry, JsonNode) method for

deserialization. This is ensured during the class registration process.

Change-Id: I5b103e06eab6627dbfe9d531caae1a3ac4b296da

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2752

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +16
    ./PrimaryIndexOperationTrackerFactory.java
    • -4
    • +22
    ./SecondaryIndexOperationTrackerFactory.java
  1. … 127 more files in changeset.
[NO ISSUE] Incremental cleanup of deprecated exception ctors

Change-Id: I1e7c3655828fc6530cef83ea502a6cfbf41acddf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2533

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -1
    • +1
    ./LockThenSearchOperationCallbackFactory.java
    • -1
    • +1
    ./PrimaryIndexInstantSearchOperationCallback.java
    • -1
    • +1
    ./PrimaryIndexInstantSearchOperationCallbackFactory.java
    • -2
    • +2
    ./PrimaryIndexSearchOperationCallback.java
    • -1
    • +1
    ./PrimaryIndexSearchOperationCallbackFactory.java
    • -1
    • +1
    ./SecondaryIndexInstanctSearchOperationCallbackFactory.java
    • -1
    • +1
    ./SecondaryIndexInstantSearchOperationCallback.java
    • -1
    • +1
    ./SecondaryIndexModificationOperationCallback.java
  1. … 232 more files in changeset.
[ASTERIXDB-1952][TX][IDX] Filter logs pt.2

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Add a log type specifically for filters

- Only log change when filter actually widens

- Stop logging of index + filter tuple during modification

- Redo index and filter tuples separately via their logs

Change-Id: Ie9e7795d9c8c212e8610dcb9bb5d26ec9fbbee8a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1857

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -26
    • +49
    ./AbstractIndexModificationOperationCallback.java
    • -4
    • +4
    ./PrimaryIndexModificationOperationCallback.java
  1. … 45 more files in changeset.
[ASTERIXDB-2299] Set log type properly during modifications

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Previously we have a deadlock-free protocol during normal ingestion

pipeline. When the try lock fails, we flush the frame partially so that

they can release locks, and log a WAIT record to wake up after that.

However, after logging the WAIT record, we didn't set the log type

back to UPDATE. This seriously degrades the ingestion performance

afterwords since all updates log records would become WAIT log records,

which require heavier logic to process upon the log is flushed to

disk.

Change-Id: Ibcf93072ca0833cb24ba6719796f58df56384c3b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2424

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +2
    ./PrimaryIndexModificationOperationCallback.java
[ASTERIXDB-1972][COMP][RT][TX] index-only plan

- user model changes: no

- storage format changes: no

- interface changes: IAccessMethod, ILSMIndexOperationContext,

IIndexAccessor

Details:

- Implement an index-only plan

- Add a SET option that disables the index-only plan

Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1866

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

    • -0
    • +68
    ./SecondaryIndexInstanctSearchOperationCallbackFactory.java
    • -0
    • +72
    ./SecondaryIndexInstantSearchOperationCallback.java
  1. … 423 more files in changeset.
[ASTERIXDB-2231][STO] Separate primary op tracker for each partition

- user model changes: no

- storage format changes: no.

- interface changes: yes.

Details:

- Separate primary index operation tracker for each partition, instead

of having a global one on each NC to achieve better scalability.

- As a coordinated change, separate component id generator for each

partition as well.

- Add partition to transaction context so that transaction operations

can operate on proper op tracker.

- Fixes [ASTERIXDB-2232] to calculate dataset partitions correctly.

Change-Id: I9eb3854d2343e45beeccb87b0d434e5f4efd69c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2263

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +1
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -2
    • +7
    ./PrimaryIndexOperationTrackerFactory.java
    • -1
    • +1
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -2
    • +3
    ./SecondaryIndexOperationTrackerFactory.java
  1. … 58 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -2
    • +2
    ./SecondaryIndexSearchOperationCallbackFactory.java
  1. … 984 more files in changeset.
[NO ISSUE][OTR] Remove AppRuntimeContextProviderForRecovery

- user model changes: no

- storage format changes: no

- interface changes: yes

- Remove IAppRuntimeContextProvider

Details:

AppRuntimeContextProviderForRecovery is not needed since it has

reference to ApplicationContext. ApplicationContext itself has all

required references by other classes.

Change-Id: I264b86b1bfff37c137936f620745025f0fb837ad

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2265

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +1
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -1
    • +1
    ./SecondaryIndexModificationOperationCallbackFactory.java
  1. … 9 more files in changeset.
[ASTERIXDB-2167][TX][RT] Remove TxnId from JobSpecification

- user model changes: no

- storage format changes: no

- interface changes: IJobEventListenerFactory

details:

- Remove the TxnId from the compiled job specification

- This enables one job spec to be used by multiple jobs/transactions

- Runtime operators who need the TxnId will pull it from the EventListener

Change-Id: I9526d50b31aebc3bf971d95ba3edf29c0c1066a7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2154

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -4
    • +3
    ./LockThenSearchOperationCallbackFactory.java
    • -4
    • +3
    ./PrimaryIndexInstantSearchOperationCallbackFactory.java
    • -4
    • +3
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -4
    • +3
    ./PrimaryIndexSearchOperationCallbackFactory.java
    • -4
    • +3
    ./SecondaryIndexModificationOperationCallbackFactory.java
  1. … 22 more files in changeset.
[ASTERIXDB-2171][STO] Remove Temporary Datasets

- user model changes: no

- storage format changes: no

- interface changes: yes

- Removed all reference to temporary datasets.

Details:

- Remove temporary datasets grammar but reserve

<TEMPORARY> token.

- Remove special callbacks for temp datasets.

- Remove metadata garbage collector thread.

- Remove isTemp flag from datasets.

- Remove special storage path for temp datasets.

- Remove temp datasets test cases.

Change-Id: Ib50ea27e2ec9c114508cd4a109cf646e2f20dcb0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2172

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -54
    • +0
    ./TempDatasetIndexModificationOperationCallback.java
    • -80
    • +0
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -81
    • +0
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
  1. … 367 more files in changeset.
[NO ISSUE][TX] Introduce Atomic Transactions

- user model changes: no

- storage format changes: no

- interface changes: yes

- Redesign and simplify ITransactionManager API

- Redesign and simplify ITransactionContext API

Details:

- Introduce atomic transactions. Unlike entity level transaction,

atomic transaction do not generate any entity commit logs and

may modify multiple primary indexes. Therefore, either all the

operations of an atomic transaction will be committed or nothing.

Atomic transactions are used by metadata transactions, while other

transactions still use entity level transactions.

- Add index resource id to AbstractOperationCallback.

- Refactor metadata index modification code.

- Remove unused class MutableResourceId

- Remove unused class FieldsHashValueGenerator

- Add test case for concurrent metadata transactions.

Change-Id: I13db1c15f8afbdaae608ff0a7468fe62bf1daccd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2156

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +1
    ./AbstractIndexModificationOperationCallback.java
    • -2
    • +2
    ./LockThenSearchOperationCallback.java
    • -3
    • +3
    ./LockThenSearchOperationCallbackFactory.java
    • -2
    • +2
    ./PrimaryIndexInstantSearchOperationCallback.java
    • -7
    • +7
    ./PrimaryIndexInstantSearchOperationCallbackFactory.java
    • -3
    • +2
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -3
    • +3
    ./PrimaryIndexSearchOperationCallback.java
    • -2
    • +2
    ./PrimaryIndexSearchOperationCallbackFactory.java
    • -3
    • +2
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -2
    • +2
    ./SecondaryIndexSearchOperationCallback.java
    • -1
    • +1
    ./SecondaryIndexSearchOperationCallbackFactory.java
    • -3
    • +2
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -3
    • +2
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
  1. … 25 more files in changeset.
[ASTERIXDB-1911][HYR,RT,CLUS] Fixes and Improvements for Deployed Jobs

Rename Predistributed Jobs to Deployed Jobs

Enable job executions to have a map of job parameters

Add an Asterix function to retrieve these parameters

which are can be assigned when the job is run, e.g. for Deployed jobs

Allow Deployed jobs to have new TxnIds and JobIds for each execution

Allow simultaneous execution of one Deployed Job

Change-Id: I8f493c1fa977d07dfe8a875f9ebe9515d01d1473

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2045

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Xikui Wang <xkkwww@gmail.com>

    • -1
    • +5
    ./LockThenSearchOperationCallbackFactory.java
    • -1
    • +5
    ./PrimaryIndexInstantSearchOperationCallbackFactory.java
    • -1
    • +5
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -1
    • +5
    ./PrimaryIndexSearchOperationCallbackFactory.java
    • -1
    • +5
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -1
    • +5
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -1
    • +5
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
  1. … 66 more files in changeset.
[NO ISSUE][TX] Rename Asterix JobId to TxnId

- user model changes: no

- storage format changes: yes

- Txn log jobId changed from int to long.

- interface changes: yes

Update APIs to use long TxnId instead of int JobId

Details:

- Rename TxnId -> TxnEntityId.

- Rename Asterix JobId -> TxnId.

- Rename Asterix JobIdFactory -> TxnIdFactory.

- Change TxnId size from int to long and update

log sizes accordingly.

Change-Id: I0905595a50195b83c1afae5dde88e5502ad21b9f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2149

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -1
    • +1
    ./AbstractIndexModificationOperationCallback.java
    • -1
    • +1
    ./LockThenSearchOperationCallback.java
    • -4
    • +4
    ./LockThenSearchOperationCallbackFactory.java
    • -4
    • +4
    ./PrimaryIndexInstantSearchOperationCallbackFactory.java
    • -4
    • +4
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -4
    • +4
    ./PrimaryIndexSearchOperationCallbackFactory.java
    • -4
    • +4
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -4
    • +4
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -4
    • +4
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
  1. … 67 more files in changeset.
[ASTERIXDB-2008][CLUS] Only add pending removal if node known

[ASTERIXDB-2023][ING] Introduce Enums instead of using bytes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Only nodes which are known to cluster manager are added

to the list of nodes pending removal. Other nodes are ignored

- Enums introduced:

- ActiveEvent.Kind

- ActivePartitionMessage.Event

- Remove AdapterRuntimeManager

- Remove AdapterExecutor

Change-Id: I7044896559798426c04a3f46861bc5335b25d140

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -3
    • +3
    ./LockThenSearchOperationCallback.java
    • -3
    • +3
    ./PrimaryIndexInstantSearchOperationCallback.java
    • -2
    • +2
    ./PrimaryIndexModificationOperationCallback.java
    • -1
    • +1
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -1
    • +1
    ./SecondaryIndexModificationOperationCallbackFactory.java
  1. … 98 more files in changeset.
[ASTERIXDB-1992][ING] Suspend/Resume active entities

- user model changes: wait for completion only returns

when the entity becomes inactive.

- storage format changes: no

- interface changes:

01) Introduce:

IMetadataLockManager for entity locking.

IActiveNotificationHandler for handling active events.

IRetryPolicy for recovery of failed active jobs.

IActiveEntityController for controlling active entities.

02) IJobLifecycleListener.notifyJobFinish now passes the

JobStatus and Exceptions.

03) IActiveEntityEventsListener.isActive() returns true,

if entity is active, false otherwise.

04) IActiveEntityEventsListener.unregister() removes the

listener upon entity deletion.

05) IActiveEntityEventsListener.getJobFailure() returns

the job failure if the entity is in a failed state.

06) IStatementExecutor.getComponentProvider() returns

the storage component provider.

07) IStatementExecutor.getApplicationContext() returns

the application context.

08) IMetadataManager.upsertEntity to perfrom metadata

entities' upsert operations.

09) IMetadataNode.upsertEntity to perfrom metadata

entities' upsert operations

10) ICcApplicationContext.getMetadataLockManager() returns

the lock manger.

11) ICcApplicationContext.getClusterStateManager() returns

the cluster state manager.

details:

- Starting and stopping of active entities now go through

their listeners rather than having the listener indicating

active entity and not having it indicating inactive entity.

This facilitates suspend/resume operations of long running

active jobs for the sake of DDL operations or topology

changes.

- Unit tests for the vast majority of code paths and

different possible scenarios have been added.

Change-Id: Ifeac8c73e6bad39a13663b84a52121356e3c6b40

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1875

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 140 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -1
    • +1
    ./AbstractIndexModificationOperationCallback.java
    • -1
    • +1
    ./LockThenSearchOperationCallback.java
    • -2
    • +1
    ./LockThenSearchOperationCallbackFactory.java
    • -1
    • +1
    ./PrimaryIndexInstantSearchOperationCallback.java
    • -1
    • +1
    ./PrimaryIndexInstantSearchOperationCallbackFactory.java
    • -7
    • +7
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -5
    • +5
    ./PrimaryIndexOperationTrackerFactory.java
    • -1
    • +1
    ./PrimaryIndexSearchOperationCallback.java
    • -1
    • +1
    ./PrimaryIndexSearchOperationCallbackFactory.java
    • -6
    • +6
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -1
    • +1
    ./SecondaryIndexSearchOperationCallback.java
    • -1
    • +1
    ./SecondaryIndexSearchOperationCallbackFactory.java
    • -6
    • +6
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -7
    • +8
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
  1. … 557 more files in changeset.
Index-Only Plan Patch Set 4: Index SearchCursor adjustment

- Modify the search cursors to maintain the result of proceed()

since the result will be required to check whether the given tuple

is qualified to be part of an index-only plan result or not.

More details will be followed in the next patch set.

- Fix the search cursors to call cancel() correctly.

Change-Id: I299b1858b7875ffc116f8f3115d319fe7b53a537

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1630

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -1
    • +10
    ./PrimaryIndexInstantSearchOperationCallback.java
  1. … 5 more files in changeset.
Remove static cc application context instance

Change-Id: Ia2e250405967ec880e7af6387aa981f39b3392c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1606

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -2
    • +2
    ./PrimaryIndexOperationTrackerFactory.java
    • -2
    • +2
    ./SecondaryIndexOperationTrackerFactory.java
  1. … 180 more files in changeset.
Fix transaction logs and optimize upserts

Previously, Transaction logs didn't log previous image

which made it difficult to undo aborted transactions

correctly. This change fixes that by always recording

previous image.

In addition, Upsert was performed as a delete if found

followed by an insert with two logs. This change makes

it a single operation with a single transaction log.

Change-Id: Ice5296267033cd7debe76894c864c6411f761d83

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1554

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -17
    • +49
    ./AbstractIndexModificationOperationCallback.java
    • -1
    • +3
    ./LockThenSearchOperationCallback.java
    • -1
    • +2
    ./LockThenSearchOperationCallbackFactory.java
    • -2
    • +3
    ./PrimaryIndexInstantSearchOperationCallback.java
    • -1
    • +2
    ./PrimaryIndexInstantSearchOperationCallbackFactory.java
    • -15
    • +6
    ./PrimaryIndexModificationOperationCallback.java
    • -9
    • +7
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -1
    • +2
    ./PrimaryIndexSearchOperationCallback.java
    • -6
    • +7
    ./PrimaryIndexSearchOperationCallbackFactory.java
    • -12
    • +5
    ./SecondaryIndexModificationOperationCallback.java
    • -11
    • +9
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -1
    • +2
    ./SecondaryIndexSearchOperationCallback.java
    • -6
    • +4
    ./TempDatasetIndexModificationOperationCallback.java
    • -8
    • +9
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -8
    • +9
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
  1. … 48 more files in changeset.
Unify structure of Cluster Controller and Node Controller

Change-Id: Ife3a002371a07ee9cdd32e1ffd50cc775bf1d453

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1514

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -4
    • +4
    ./PrimaryIndexOperationTrackerFactory.java
    • -4
    • +4
    ./SecondaryIndexOperationTrackerFactory.java
  1. … 159 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +45
    ./PrimaryIndexOperationTrackerFactory.java
    • -45
    • +0
    ./PrimaryIndexOperationTrackerProvider.java
    • -0
    • +45
    ./SecondaryIndexOperationTrackerFactory.java
    • -45
    • +0
    ./SecondaryIndexOperationTrackerProvider.java
  1. … 424 more files in changeset.
ASTERIXDB-1711: remove more mentions of Asterix

Change-Id: I21666fd54075ac56de8aab76bb7d5cdbec020121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1387

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

    • -3
    • +3
    ./PrimaryIndexModificationOperationCallback.java
    • -2
    • +2
    ./PrimaryIndexOperationTrackerProvider.java
    • -2
    • +2
    ./SecondaryIndexOperationTrackerProvider.java
  1. … 568 more files in changeset.
Cleanup FileSplit and FileReference

This change gives FileSplit and FileReference specific meaning to

avoid confusion of an absolute vs relative, local vs global, inside

an IO device vs outside IO devices.

In addition, it enables better abstraction of global partitions and

delegate the responsibility of choosing which partition goes to which

IO device to the IO Manager through the introduction of FileDeviceComputer

In details:

Previously, the LocalResource in Hyracks had partition (storage partition)

and there is no such thing in Hyracks. This scope leak is bad. In addition

The local resource had a name and a path. they were always the same and so

the name was removed.

The storage partition was instead moved to asterixdb implementation of the

serialized object in the local resource.

With all of these changes, the cluster controller (compiler) only needs to

know about partitions and relative paths. It doesn't need to worry about

heterogenous Node setups and different io device configurations. For File

assignment to IO devices, a new interface (IFileDeviceComputer) was

introduced which can be overriden by applications to have their own

strategy for distributing files among IO devices.

Change-Id: I4fac508bf9af5a3bed41a3cf4464d2cbfecf2f61

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1352

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -11
    • +13
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -9
    • +11
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -9
    • +11
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -9
    • +11
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
    • -13
    • +14
    ./UpsertOperationCallbackFactory.java
  1. … 283 more files in changeset.
Fix Upsert to Never Enforce the First Operation

Change-Id: I8ec784e2d6ff39758ab701d4f36fc85c278178f2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1336

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -1
    • +6
    ./LockThenSearchOperationCallback.java
  1. … 1 more file in changeset.
Small Cleanup Towards Fixing LifeCycle Issues

Before this change, dataset lifecycle manager was providing a set

of functionalities that are loosly related to management of datasets

and indexes. However, it was not clear what the possible states of

a dataset or an index and what is the responsibility of each object.

This change takes the first step towards fixing this area. Indexes

of a dataset are now grouped together under a single lifecycle class

A resource aka dataset must be created outside the lifecycle manager

and registered with it before it can be assigned resources (memory)

and before it can be used by any operation. This is still not the

case.

Change-Id: I84005a33837725f41ae63297a3711215dccce1d8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1148

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +3
    ./PrimaryIndexModificationOperationCallbackFactory.java
    • -3
    • +3
    ./SecondaryIndexModificationOperationCallbackFactory.java
    • -3
    • +3
    ./TempDatasetPrimaryIndexModificationOperationCallbackFactory.java
    • -3
    • +3
    ./TempDatasetSecondaryIndexModificationOperationCallbackFactory.java
  1. … 66 more files in changeset.