Clone Tools
  • last updated 19 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][NET] Re-resolve InetSocketAddresses on failure

Support IP address updates on replicas & CC by re-resolving the IP address

on connection failure

Change-Id: I2532a27c2abb8b891df668b0adc95cc33da99620

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4965

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 9 more files in changeset.
[NO ISSUE] Support multipart dataverse names

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Support dataverse names that consist of multiple parts

- Introduce DataverseName class and adopt it throughout the system

- Add testcases for multipart dataverse names

Change-Id: I1313b2dc0e8df6a9b0ded48dea122afc3ba5dbe7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4004

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 283 more files in changeset.
[NO ISSUE][*DB][REPL] Identifier comparability of unknown hosts

Ensure that replication identifiers comparisons are not influenced by

address resolution

Change-Id: I6d7aa87777d17a3416d16cfad23b2773b1614a02

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3536

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[NO ISSUE][OTH] Add API To Get Dataset Size

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add an API that returns the on disk total size of a dataset

and its indexes on a node.

Change-Id: Iaff87bbe1f2417689f7827deaf03fcddd64ca7e4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3425

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +81
    ./DatasetCopyIdentifier.java
  1. … 1 more file in changeset.
[NO ISSUE][*DB][STO] indicate null/missing value fix in index checkpoint

- user model changes: no

- storage format changes: yes (backwards-compatible)

- interface changes: no

Details:

- add marker on inintial IndexCheckpoint, propagate on subsequent checkpoints

- ignore extra JSON fields when deserializing IndexCheckpoint

Change-Id: Id93ab9d16887b37cf6c0d011950e7c57f1a1d646

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3324

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 625 more files in changeset.
[NO ISSUE][OTH] Add Creation Time to Client Request

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add creation time to ClientRequest to identify when the

request was acknowledged by the CC.

- Allow Receptionist extensions to throw exceptions as

specified by the IReceptionist API.

- Allow extensions to set non-json request optional parameters.

- Add convention method in ResourceReference to get the

resource's fully qualified dataset name.

- Fix typo in metadata provider method name.

Change-Id: Ib45b5bec8bb2f127aa1263d4f7ac4f8e2e368208

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3217

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 4 more files in changeset.
[ASTERIXDB-2422][STO] Introduce compressed storage

- user model changes: yes

- Add new configuration in the with-caluse to enable compression

- Add new nc configuration in the config file

- storage format changes: yes

- Pages of the primary index can be compressed

- Add a companion file (Look Aside File) with the compressed index

- Allow optional values in the LocalResource

- Add compression information in Metadata.Dataset

- interface changes: yes

- ICCApplicationContext:

- Add getCompressionManager()

- IBufferCache:

- Add getCompressedFileWriter(int fileId)

- ICachedPageInternal:

- Add setCompressedPageOffset(long offset)

- Add getCompressedPageOffset()

- Add setCompressedPageSize(int size)

- Add getCompressedPageSize()

Details:

- Add new integration test for this patch

- Fix ASTERIXDB-2464

- Add ddl-with-clause type validator

Additional details in the design document:

https://cwiki.apache.org/confluence/display/ASTERIXDB/Compression+in+AsterixDB

Change-Id: Idde6f37c810c30c7f1a5ee8bcbc1e3e5f4410031

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2857

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +49
    ./ICompressionManager.java
  1. … 191 more files in changeset.
[NO ISSUE][REPL] Ensure Valid Component ID is Initialized On Replica Sync

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Currently, the first time a replica is synchronized from master,

the valid component id on each replicated index's initial checkpoint

will be the initial value of a component id (-1). This value is

fixed when the the replica receives a flushed component from

the index. However, if the master fails before any component is

flushed to a replica and that replica is promoted to master, it

will start from an invalid component id. This change ensures that

the initial checkpoint of replicated indexes is initialized to

the maximum component id that appears on master. This will ensure

that if the replica is promoted, it will at least start from

a component that wasn't previously used on master.

- Replace assertion of component ids validation by illegal state.

Change-Id: I85395ad823a630725c4cab4bead1c61546dc61ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2973

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 6 more files in changeset.
[NO ISSUE][TX] Create New Log File Before Deleting Old Files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure next log file is created after a sharp check

point before deleting old files. This is to prevent

the case if a crash happens right after deleting the

old files but before creating the new one, then the

next time the system starts up, it will start with

log file id 0 which is wrong.

- Log the details of latest index checkpoint when the

low watermark of the new checkpoint is less than the

low watermakr of the latest checkpoint.

Change-Id: I4817f697b43daff55726909ab074ec30a1c224ce

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2959

Reviewed-by: Michael Blow <mblow@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 3 more files in changeset.
[NO ISSUE][STO] Ensure First Component ID is Initialized

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Initialize the component id generator from the primary

index checkpoint, if exits, as soon as it is created.

- Ensure the first component id is passed to all indexes.

Change-Id: I246f9373f950e2f9a2c63f86746462e42a3f1c62

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2948

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 13 more files in changeset.
[ASTERIXDB-2444][STO] Avoid Using System Clock in Storage

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Replace the usage of system clock timestamps in LSM

index components file names by a sequencer. The next

sequence id to use is determined by checking the list

of existing components on disk. Note that due to a

rollback, an index checkpoint file may have last valid

component sequence which is greater than what is on disk.

This should not cause any issues since only components

that have a sequence greater than that appears in the

checkpoint will be deleted.

- Replace the usage of system clock timestamps in LSM

index components ids by a monotonically increasing

sequencer. The sequencer is initialized after restarts

by the last valid component id that appears in the

index checkpoint.

- Refactor the logic to generate flush/merge file names.

- Refactor the logic to check invalid components.

- Adapt test cases to new naming format.

Change-Id: I9dff8ffb38ce8064a199d03b070ed1f5b924b8a4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2927

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 22 more files in changeset.
[NO ISSUE][OTH] Remove Unnecessary Null Checks

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Remove unnecessary null checks before instanceof.

Change-Id: I870df76623f5f983a90b5437abf7e3993a54c744

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2895

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 14 more files in changeset.
[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- This change introduces some improvements to storage

operations.

- Local RecoveryManager is now extensible.

- Bulk loaders now call the IO callback similar to

Flushes, making them less special and creating a

unified lifecycle for adding an index component.

- As a result, The IndexCheckpointManager doesn't need

to have a special treatment for components loaded

through the bulk load operation.

- Component Id have been added to the index checkpoint

files.

- Cleanup for the code of local recovery for failed flush

operations.

- Ensure that after local recovery of flushes, primary

and secondary indexes have the same index for mutable

memory component.

- The use of WAIT logs to ensure in-flight flushes

are scheduled didn't work as expected. A new log type

WAIT_FOR_FLUSHES was introduced to acheive the expected

behavior.

- The local test framework was made Extensible to support

more use cases.

- Test cases were added for component ids in checkpoint files.

The following scenarios were covered:

- Primary and secondary both have values when a flush is

shceduled.

- Primary have values but not secondary when a flush is

scheduled.

- Primary is empty and an index is created through bulk

load.

- Primary has a single component and secondary is created

through bulk load.

- Primary has multiple components and secondary is created

through bulk load.

- Each primary opTracker now keeps a list of ongoing flushes.

- FlushDataset now waits only for flushes only and

not all io operations.

- Previously, we had many flushes scheduled on open datasets.

This was not detected but after this change, a failure

is thrown in such cases.

- Flush operations dont need to extend the comparable

interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2632

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 88 more files in changeset.
[NO ISSUE][REPL] Ignore LSNs of Partially Replicated Indexes

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- When determining low watermark, ignore LSN of replicated

indexes with no checkpoints.

- Guard logs in case of unexpected min LSN read failures.

- Ensure only one replica is synchronized at a time to prevent

possible merge operations from deleting files being synchronized

to another replica concurrently.

- Ensure index metadata files are replicated first to allow

replicas to find any existing files in case of re-synchronization.

- Ensure replication channel is closed on replication failures.

Change-Id: I9ca08da29bdd8fc4406f2df7e6eb32601caf9388

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2534

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 5 more files in changeset.
[NO ISSUE][API] Add Storage Stats API

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a new API on NCs for storage stats.

The API returns the total size of each index

as well as the id and size of each disk component.

Change-Id: Ie0345b00cfd825cbb77664c7617e8301c4f84e6d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2486

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +71
    ./ResourceStorageStats.java
  1. … 2 more files in changeset.
[NO ISSUE][STO] Platform Independent File Path Split

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Avoid regex while splitting file paths to

eliminate invalid regex on Windows platform.

Change-Id: Id3d6bae7e35b26c6fd5685757a7c13ea86811f14

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2391

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][REPL] Add API to Release Partition

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add API to allow an NC to release a partition

by flushing all of its resources and unregistering

all of its replicas. This API can be used to allow

graceful failover of a cluster partition.

- Add test case.

Change-Id: I7c8bbfc123904f13aae9f640dbefc7d17df69a42

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2349

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 10 more files in changeset.
[NO ISSUE][STO] Remove Storage Auto Migration Code

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- It was decided to not support auto storage migration

from old to newer version at this point in time. This

change removes the storage migration code and its test.

Change-Id: I6faa816ca71913f18ff6017e86c8a29253728572

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2305

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 6 more files in changeset.
[ASTERIXDB-2234][STO] Delete Invalid Components

- user model changes: no

- storage format changes: no

- interface changes: yes

- IIndexCheckpointManager:

(+) getValidComponentTimestamp

(+) getCheckpointCount

(-) advanceLowWatermark

Details:

- Delete any component with start timestamp

after the index checkpoint valid component

timestamp on NC startup/shutdown.

- Add test case for deleting invalid components.

Change-Id: Ib11782edd79c7ef0c8949cb08e863b0ec1687a87

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2270

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 5 more files in changeset.
[ASTERIXDB-2195][REPL] Replace Static Replication

- user model changes: no

- storage format changes: no

- interface changes: yes

- Redesigned all replication interfaces

Details:

- Replace static replication and fault tolerance by

dynamic storage API.

- Remove static based fault tolerance strategies.

- Redesign replication APIs and classes to smaller

maintainable parts.

- Clean up replication properties.

- Unify logic for checkpoints when replication is

enabled.

- Remove static replication test cases.

- Add replication runtime test cases for:

- Bulkload component replication.

- Memory component recovery.

- Flushed component replication.

- Add replication integration test for:

- Resync failed replica.

Change-Id: Ic5c4b0ac199a4530c807e558c8aebb1eb1284048

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2252

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 252 more files in changeset.
[ASTERIXDB-2198][REPL] Introduce Dynamic Replica Placement

- user model changes: no

- storage format changes: no

- interface changes: yes

- Add IReplicationMessage and IReplicaTask.

- Add notifyMetadataNodeChange to IFaultToleranceStrategy.

- Add register to IReplicationManager to allow registering

replicas at runtime.

Details:

- Add cluster APIs for:

- changing partition master node.

- changing metadata node.

- Add NC storage management API for promoting a partition replica

to master replica.

- Implement changing metadata node at runtime in

MetadataNodeFaultToleranceStrategy.

- Allow MetadataNodeFaultToleranceStrategy to have zero replica

at initialization.

- Add a flag to LangExecutionUtil to skip storage distribution

check at the end of each test.

- Add test case for metadata node failover as follows:

1- start with nc1 as metadata node.

2- add replica for metadata partition on nc2 at runtime.

3- performs metadata transactions on nc1.

4- promote metadata partition on nc2.

5- failover metadata node to nc2.

6- ensure the effects of the metadata transactions on (2) exists.

7- performs more metadata transactions on nc2.

8- ensure the effects of the metadata transactions on (7) exists.

Change-Id: I11f82efcad29d2c37324fe9d3c11d872b0348f49

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2215

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 59 more files in changeset.
[NO ISSUE][TEST] Add NC Storage API Test

- user model changes: no

- storage format changes: no

- interface changes: yes

- Add IPartitionReplica to use it at the

APIs level.

- Rename IStorageSubsystem -> IReplicaManager

Details:

- Add option to TestExecutor to target specific

NC end point.

- Add storage API test case.

Change-Id: I76c336a66e32036a34d30ce6d3a31195c342c4a9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2195

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +57
    ./IReplicaManager.java
  1. … 16 more files in changeset.
[NO ISSUE][STO] Introduce Index Checkpoints

- user model changes: no

- storage format changes: yes

- Add index checkpoints.

- Use index checkpoint to determine low watermark

during recovery.

- interface changes: yes

- Introduce IIndexCheckpointManager for managing

indexes checkpoints.

- Introduce IIndexCheckpointProvider for tracking

IIndexCheckpointManager references.

Details:

- Unify LSM flush/merge operations completion order.

- Introduce index checkpoints which contains:

- Index low watermark.

- Latest valid LSM component

- Mapping between master replica and local replica.

- Use index checkpoints instead of LSM component metadata

for identifying low watermark in recovery.

- Use index checkpoints in replication instead of overwriting

LSN byte offset in replica component metadata.

- Replace LSN_MAP used in replication by index checkpoints.

- Replace NIO Files.find by Commons FileUtils.listFiles to

avoid no NoSuchFileException on any file deletion.

Change-Id: Ib22800002bf8ea3660242e599b3f5f20678301a8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2200

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +93
    ./IIndexCheckpointManager.java
    • -0
    • +40
    ./IIndexCheckpointManagerProvider.java
    • -0
    • +98
    ./IndexCheckpoint.java
  1. … 43 more files in changeset.
[NO ISSUE][API] Add NC Storage API

- user model changes: no

- storage format changes: no

- interface changes: yes

Add IStorageSubsystem to track storage partitions

replicas.

Details:

- Add NC API port.

- Add storage API to NCs.

- Add StorageSubsystem to track storage partitions

replicas.

Change-Id: I120d9892bc9fe5a73395cd5a2ddc30b51b73ced2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2190

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +55
    ./IStorageSubsystem.java
    • -0
    • +100
    ./PartitionReplica.java
    • -0
    • +68
    ./ReplicaIdentifier.java
  1. … 23 more files in changeset.
[ASTERIXDB-2177][STO] Use Fixed Storage Root Dir Name

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Eliminate the need to read the storage root dir name

from cluster properties and use a fixed name (storage).

- Eliminate the need to maintain root_metadata file.

Change-Id: I4e9772e9da10cff33f11353610788ba541a35571

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2182

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 14 more files in changeset.
[NO ISSUE][STO] Adapt Storage Structure To Rebalance

- user model changes: no

- storage format changes: no

- interface changes: yes

-- Added IResource#setPath to use for the resource

storage migration.

Details:

- Unify storage structure to support dataset rebalance:

Old format:

./storage/partition_#/dataverse/datasetName_idx_indexName

New format:

./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName

- Adapt recovery and replication to new storage structure.

- Add old structure -> new structure NC migration task.

- Add CompatibilityUtil to ensure NC can be upgraded during

NC startup.

- Centralize the logic for parsing file path to its components in

ResourceReference/DatasetResourceReference.

- Add storage structure migration test case.

- Add test case for recovery after rebalance.

Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2181

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +77
    ./DatasetResourceReference.java
    • -0
    • +44
    ./IndexPathElements.java
    • -0
    • +110
    ./ResourceReference.java
  1. … 37 more files in changeset.
Introduce Strategy Based Replication and Fault-Tolerance

This change includes the following:

- Introduce new APIs for Replication and Fault-Tolerance Strategies.

- Add configuration in cluster description file for high-availability.

- Add built-in replication strategies (Metadata_Only, Chained_Declustering)

- Add built-in fault-tolerance strategies (Auto, Metadata_Node)

- Remove none cluster state functionality from ClusterStateManager.

- Add customizable NC startup sequence

Change-Id: I1d1012f5541ce786f127866efefb9f3db434fedd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1405

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +74
    ./IndexFileProperties.java
  1. … 111 more files in changeset.