Clone Tools
  • last updated 14 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2370][*DB] Modify the inconsistent method name "isFull"

- taken from https://github.com/apache/asterixdb/pull/7

The method is named as "isFull". However, the method does set the boolean

variable full. Thus, rename the method as "setFull" should be more

intuitive than "isFull" since "isFull" is a query asking whether the

object is full or not.

Change-Id: I99bf451367171afb3fc0511a6170eb4c492fc69f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2629

Contrib: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -1
    • +1
    ./management/LogReplicationManager.java
[NO ISSUE][REPL] Use String.compareTo To Sort Replicated Files

- user model changes: no

- storage format changes: no

- interface changes: no

- Use String.compareTo to sort replicated files

instead of custom comparator.

Change-Id: I45c106d5cff6e07d16a9a7ce286a2ac563a80338

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2547

Reviewed-by: Michael Blow <mblow@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -12
    • +2
    ./sync/ReplicaFilesSynchronizer.java
[NO ISSUE][REPL] Ignore LSNs of Partially Replicated Indexes

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- When determining low watermark, ignore LSN of replicated

indexes with no checkpoints.

- Guard logs in case of unexpected min LSN read failures.

- Ensure only one replica is synchronized at a time to prevent

possible merge operations from deleting files being synchronized

to another replica concurrently.

- Ensure index metadata files are replicated first to allow

replicas to find any existing files in case of re-synchronization.

- Ensure replication channel is closed on replication failures.

Change-Id: I9ca08da29bdd8fc4406f2df7e6eb32601caf9388

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2534

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +12
    ./sync/ReplicaFilesSynchronizer.java
  1. … 3 more files in changeset.
[NO ISSUE] Simplify IoUtil delete API

Change-Id: I0dabcb642fa3007b3b6e9d30be9911a22ed8f252

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2502

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-1952][TX][IDX] Filter logs pt.2

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Add a log type specifically for filters

- Only log change when filter actually widens

- Stop logging of index + filter tuple during modification

- Redo index and filter tuples separately via their logs

Change-Id: Ie9e7795d9c8c212e8610dcb9bb5d26ec9fbbee8a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1857

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 46 more files in changeset.
[ASTERIXDB-2229][OTR] Restore Thread Names in Thread Pool

- user model changes: no

- storage format changes: no

- interface changes: no

- Restore thread names to their original names

before returning them to the pool of the NC

app.

- Remove explicit thread name resets.

- Delete unused ThreadExecutor class.

Change-Id: I3bda1b65e7aefd35d2b8cfa814f73369c3bf5a18

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2447

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 9 more files in changeset.
[ASTERIXDB-2188] Ensure recovery of component ids

- user model changes: no

- storage format changes: yes.

Flush log record format changes.

- interface changes: no

Details:

- Add flush component ids to the flush log record. Upon

seeing a flush log record during recovery, schedule

a flush to all indexes in this partition s.t. LSN>maxDiskLSN

to ensure component ids are properly maintained upon

failed flushes.

- Add a test case to ensure the correctness of the recovery logic

of component ids

Change-Id: I8c1fc2b209cfb9d3dafa216771d2b7032eb99e75

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2408

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 17 more files in changeset.
[NO ISSUE][NET] Networking Improvements

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Set keep alive and no TCP delay options

on socket channels.

- Cancel key and close IPC handle on failed

read/writes to avoid getting the same failures

with every NetworkThread loop.

Change-Id: I60c1f9cfe2ea577fca14cd2e98c6461c49df011a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2418

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 5 more files in changeset.
[NO ISSUE][REPL] Exclude Non-Replicated Datasets From Delta Recovery

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Exclude non-replicated datasets files from

delta recovery.

- Fix used read buffer for large replication

requests.

Change-Id: Ic734af7becf26082e79fae52bd2c01ba567c1c99

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2412

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -3
    • +6
    ./messaging/PartitionResourcesListTask.java
  1. … 1 more file in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -5
    • +4
    ./management/LogReplicationManager.java
  1. … 981 more files in changeset.
[NO ISSUE][OTR] Logging Cleanup

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- No pretty printing for replica status.

- Log ignored exception in debug level.

Change-Id: I7e4a37e9621ef5bdb27105b17e883800212e594b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2276

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE][TX] Remove Unused Fields in LogRecord

- user model changes: no

- storage format changes: no

- interface changes: yes

- Remove nodeId from ILogRecord

Details:

- Remove the following fields which are

no longer needed from LogRecord:

- numOfFlushedIndexes

- nodeId

- The removed fields were not persisted

and only used for remote logs.

Change-Id: If293953e51525e27110909203941444c6d7eb344

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2269

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 5 more files in changeset.
[ASTERIXDB-2195][REPL] Clean Masked Files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Clean invalid masked files before promoting

a partition or sending partition files list

to master.

- Let replica calculate component id instead

of sending it from master.

- Add tests for:

- Deleting masked component.

- Deleting masked file.

Change-Id: Ib0f0159159faf87b9f5fd2eca3956dd90633bcfa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2268

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -1
    • +1
    ./messaging/PartitionResourcesListTask.java
  1. … 6 more files in changeset.
[NO ISSUE][OTR] Remove Non-ASCII Char

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Icc3c78786d87fa9387b2c59fd2c18eff1b971da6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2261

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[ASTERIXDB-2195][REPL] Replace Static Replication

- user model changes: no

- storage format changes: no

- interface changes: yes

- Redesigned all replication interfaces

Details:

- Replace static replication and fault tolerance by

dynamic storage API.

- Remove static based fault tolerance strategies.

- Redesign replication APIs and classes to smaller

maintainable parts.

- Clean up replication properties.

- Unify logic for checkpoints when replication is

enabled.

- Remove static replication test cases.

- Add replication runtime test cases for:

- Bulkload component replication.

- Memory component recovery.

- Flushed component replication.

- Add replication integration test for:

- Resync failed replica.

Change-Id: Ic5c4b0ac199a4530c807e558c8aebb1eb1284048

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2252

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +39
    ./api/IReplicationWorker.java
    • -0
    • +168
    ./api/PartitionReplica.java
    • -0
    • +128
    ./api/ReplicationDestination.java
    • -70
    • +0
    ./functions/ReplicaFilesRequest.java
    • -60
    • +0
    ./functions/ReplicaIndexFlushRequest.java
    • -71
    • +0
    ./functions/ReplicaLogsRequest.java
    • -354
    • +0
    ./functions/ReplicationProtocol.java
    • -0
    • +44
    ./logging/RemoteLogRecord.java
    • -0
    • +102
    ./logging/RemoteLogsNotifier.java
    • -0
    • +88
    ./logging/RemoteLogsProcessor.java
    • -0
    • +49
    ./logging/TxnAck.java
  1. … 238 more files in changeset.
[ASTERIXDB-1564][CONF] Replace Remaining Usage of JUL

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace remaining usage of java.util.logging

by Log4j2.

Change-Id: I894348b09a862693ae31de7c2f4768877a5297db

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2254

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 9 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -18
    • +11
    ./management/ReplicationChannel.java
    • -34
    • +24
    ./management/ReplicationManager.java
    • -13
    • +6
    ./recovery/RemoteRecoveryManager.java
  1. … 387 more files in changeset.
[ASTERIXDB-1723] Delete managix

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Delete managix installer method and all dependencies

- Remove xml-based configuration

Change-Id: I5c710b6ec870bedd49dd46b74ed96a3665670e9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2155

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -21
    • +42
    ./management/ReplicationChannel.java
    • -36
    • +49
    ./management/ReplicationManager.java
    • -17
    • +21
    ./recovery/RemoteRecoveryManager.java
  1. … 1374 more files in changeset.
[ASTERIXDB-2198][REPL] Introduce Dynamic Replica Placement

- user model changes: no

- storage format changes: no

- interface changes: yes

- Add IReplicationMessage and IReplicaTask.

- Add notifyMetadataNodeChange to IFaultToleranceStrategy.

- Add register to IReplicationManager to allow registering

replicas at runtime.

Details:

- Add cluster APIs for:

- changing partition master node.

- changing metadata node.

- Add NC storage management API for promoting a partition replica

to master replica.

- Implement changing metadata node at runtime in

MetadataNodeFaultToleranceStrategy.

- Allow MetadataNodeFaultToleranceStrategy to have zero replica

at initialization.

- Add a flag to LangExecutionUtil to skip storage distribution

check at the end of each test.

- Add test case for metadata node failover as follows:

1- start with nc1 as metadata node.

2- add replica for metadata partition on nc2 at runtime.

3- performs metadata transactions on nc1.

4- promote metadata partition on nc2.

5- failover metadata node to nc2.

6- ensure the effects of the metadata transactions on (2) exists.

7- performs more metadata transactions on nc2.

8- ensure the effects of the metadata transactions on (7) exists.

Change-Id: I11f82efcad29d2c37324fe9d3c11d872b0348f49

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2215

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +35
    ./api/IReplicaTask.java
    • -0
    • +40
    ./api/IReplicationMessage.java
    • -4
    • +99
    ./functions/ReplicationProtocol.java
    • -26
    • +57
    ./management/ReplicationChannel.java
    • -0
    • +22
    ./management/ReplicationManager.java
    • -0
    • +89
    ./messaging/CheckpointPartitionIndexesTask.java
    • -0
    • +84
    ./messaging/DeleteFileTask.java
    • -0
    • +74
    ./messaging/PartitionResourcesListResponse.java
    • -0
    • +80
    ./messaging/PartitionResourcesListTask.java
    • -0
    • +108
    ./messaging/ReplicateFileTask.java
    • -0
    • +73
    ./recovery/FileSynchronizer.java
    • -0
    • +84
    ./recovery/ReplicaFilesSynchronizer.java
    • -0
    • +67
    ./recovery/ReplicaSynchronizer.java
    • -0
    • +1
    ./storage/ReplicaResourcesManager.java
  1. … 47 more files in changeset.
[NO ISSUE][TEST] Add NC Storage API Test

- user model changes: no

- storage format changes: no

- interface changes: yes

- Add IPartitionReplica to use it at the

APIs level.

- Rename IStorageSubsystem -> IReplicaManager

Details:

- Add option to TestExecutor to target specific

NC end point.

- Add storage API test case.

Change-Id: I76c336a66e32036a34d30ce6d3a31195c342c4a9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2195

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +97
    ./storage/PartitionReplica.java
  1. … 17 more files in changeset.
[NO ISSUE][STO] Introduce Index Checkpoints

- user model changes: no

- storage format changes: yes

- Add index checkpoints.

- Use index checkpoint to determine low watermark

during recovery.

- interface changes: yes

- Introduce IIndexCheckpointManager for managing

indexes checkpoints.

- Introduce IIndexCheckpointProvider for tracking

IIndexCheckpointManager references.

Details:

- Unify LSM flush/merge operations completion order.

- Introduce index checkpoints which contains:

- Index low watermark.

- Latest valid LSM component

- Mapping between master replica and local replica.

- Use index checkpoints instead of LSM component metadata

for identifying low watermark in recovery.

- Use index checkpoints in replication instead of overwriting

LSN byte offset in replica component metadata.

- Replace LSN_MAP used in replication by index checkpoints.

- Replace NIO Files.find by Commons FileUtils.listFiles to

avoid no NoSuchFileException on any file deletion.

Change-Id: Ib22800002bf8ea3660242e599b3f5f20678301a8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2200

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -99
    • +89
    ./management/ReplicationChannel.java
    • -33
    • +16
    ./management/ReplicationManager.java
    • -14
    • +2
    ./storage/LSMComponentLSNSyncTask.java
    • -15
    • +6
    ./storage/LSMIndexFileProperties.java
    • -88
    • +30
    ./storage/ReplicaResourcesManager.java
  1. … 41 more files in changeset.
[ASTERIXDB-2177][STO] Use Fixed Storage Root Dir Name

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Eliminate the need to read the storage root dir name

from cluster properties and use a fixed name (storage).

- Eliminate the need to maintain root_metadata file.

Change-Id: I4e9772e9da10cff33f11353610788ba541a35571

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2182

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 14 more files in changeset.
[NO ISSUE][STO] Adapt Storage Structure To Rebalance

- user model changes: no

- storage format changes: no

- interface changes: yes

-- Added IResource#setPath to use for the resource

storage migration.

Details:

- Unify storage structure to support dataset rebalance:

Old format:

./storage/partition_#/dataverse/datasetName_idx_indexName

New format:

./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName

- Adapt recovery and replication to new storage structure.

- Add old structure -> new structure NC migration task.

- Add CompatibilityUtil to ensure NC can be upgraded during

NC startup.

- Centralize the logic for parsing file path to its components in

ResourceReference/DatasetResourceReference.

- Add storage structure migration test case.

- Add test case for recovery after rebalance.

Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2181

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -33
    • +8
    ./storage/LSMComponentProperties.java
    • -40
    • +6
    ./storage/LSMIndexFileProperties.java
    • -57
    • +23
    ./storage/ReplicaResourcesManager.java
  1. … 36 more files in changeset.
[NO ISSUE][TX] Rename Asterix JobId to TxnId

- user model changes: no

- storage format changes: yes

- Txn log jobId changed from int to long.

- interface changes: yes

Update APIs to use long TxnId instead of int JobId

Details:

- Rename TxnId -> TxnEntityId.

- Rename Asterix JobId -> TxnId.

- Rename Asterix JobIdFactory -> TxnIdFactory.

- Change TxnId size from int to long and update

log sizes accordingly.

Change-Id: I0905595a50195b83c1afae5dde88e5502ad21b9f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2149

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -30
    • +30
    ./management/ReplicationManager.java
  1. … 75 more files in changeset.
[ASTERIXDB-1871][ASTERIXDB-2095] Stop Consumer Thread on Deallocate

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently there is a chance that a MaterializingPipelinedPartition

is deallocated before the consuming thread starts (e.g. due to job

abort) and therefore the consuming thread will not be interrupted

leading to leaked threads and files. This change checks if partition

was deallocated before the consumer starts, then the consumer thread

cleans up any files then exits.

- Make TaskAttemptId non-final class to mock it.

- Add test case.

Change-Id: I18c9fb085c149f41a202fff83aa6ec3aaeba6a77

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2143

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 5 more files in changeset.
[NO ISSUE][HYR][*DB] Minor refactoring / address SonarQube comments

Change-Id: Icf10b6df0fdc006675d8f0da6fd06d50200c6b6a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2098

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 59 more files in changeset.
[ASTERIXDB-1945][STO] Cleanup Buffer Cache API

- user model changes: no

- storage format changes: no

- interface changes: yes

INcApplicationContext

- removed IFileMapProvider getFileMapManager();

to hide FileMapManager from other components;

IStorageManager

- IFileMapProvider getFileMapProvider(INCServiceContext ctx);

to hide FileMapManager from other components;

IFileHandle

- added FileReference getFileReference();

to avoid unnecessary casts;

IIOManager

- public void deleteWorkspaceFiles() throws

HyracksDataException;

added throws;

ILSMIndexFileManager

- void createDirs() throws HyracksDataException;

added throws;

IInvertedIndex

- added void purge() throws HyracksDataException;

a. InvertedIndexes don't implement the ITreeIndex interface.

b. when we deactivate a disk component, we need to purge it so

the buffer cache doesn't go through each page.

c. this need to be revisited, ASTERIXDB-1944

IFileMapManager

- int registerFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- FileReference unregisterFile(int fileId) throws

HyracksDataException;

return value added for future refernece of the file;

IBufferCache

- int createFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- void deleteFile(int fileId) throws HyracksDataException;

remove the dirty page flag since there's no dirty page;

- int openFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- added void deleteFile(FileReference file) throws

HyracksDataException;

we used to have this public methods in both BufferCache

and VirtualBufferCache. Now we lifted it into the interface.

AbstractLSMIndex

- removed protected abstract void

destroyMemoryComponent(ILSMMemoryComponent c)

throws HyracksDataException;

It is because turned out when we deactivate, we actually

destroy them. However, because of the not well defined API,

double destroy was okay and so we used to do double destroy.

Details:

This change fixes the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I0a973c2adb2e7fdcbbf18c7b888af3de5f0acc74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1843

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 161 more files in changeset.
Revert "ASTERIXDB-1945 [STO] Cleanup Buffer Cache API"

This reverts commit ae3daf6ef3397e583637360dc460c6391e03dc29.

Change-Id: I5e4e23f43a68e82c38fb8d1d7f4c0d01985c3a10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1842

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 161 more files in changeset.
ASTERIXDB-1945 [STO] Cleanup Buffer Cache API

Fix the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I15565b07afdc94ac74c608bfe4480fa09dcf8f1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1840

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 161 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -2
    • +2
    ./storage/ReplicaResourcesManager.java
  1. … 571 more files in changeset.