Clone Tools
  • last updated 21 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Merge commit 'fa6ce67' from stabilization-f69489

Change-Id: I01e1810f1c44d9ecaf569dd22577c33e8702c682

  1. … 2 more files in changeset.
[NO ISSUE][REPL] Wait For Dataset IO Before Replica Sync

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add API in DatasetLifecycleManager to wait for IO on datasets

matching a replication strategy.

- Before synchronizing the files on a replica, wait for any

on-going IO operations on replicated datasets to make sure

the set of files won't change while the replica is being

synchronized (e.g. a merge operation won't delete a file

that we wanted to send to a replica).

Change-Id: I01ed5c9379cf7ae249faeef624d5226ea699cf22

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3446

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 625 more files in changeset.
[ASTERIXDB-2520][RT] Make Dataset Memory Reservation Idempotent

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure thread-safety of metadata node takeover.

- To account for multiple failures of metadata node, make

dataset memory reservation idempotent.

Change-Id: I360226187e176ce3a0ccdcd7a1b611a01d906394

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3207

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.
Merge commit '1ff69c0' from 'stabilization-f69489' into 'master'

Change-Id: Ic8aa90be3ba3645ad22d21cc54abc83ab7008598

  1. … 24 more files in changeset.
[ASTERIXDB-2506][STO] Checkpoint Datasets

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add new config to specify dataset checkpoint interval

and default its value to 10 minutes.

- Flush datasets which were not flushed for the dataset

checkpoint interval.

- Run dataset checkpoint logic as part of the CheckpointThread.

- Improve dataset async flush API to accept any LSM index predicate.

- Cleanup exception handling in CheckpoinThread.

- Add test case for dataset checkpoint.

Change-Id: I38a73a43a4b1b7d3a8ac79dd579ed4ef8c9c6a9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3106

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +13
    ./PrimaryIndexOperationTracker.java
  1. … 18 more files in changeset.
[ASTERIXDB-2494][RT] Ensure Dataset Ref Counters Are Thread-safe

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Dataset/Index reference counters are modified concurrently by

multiple threads and should be thread-safe. Thread-safety issues

in these counters could result in leaving a dataset allocated

in memory even though it can be evicted.

Change-Id: I9328df660f463bd45bfc003b1e44c9df2702cc90

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3089

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-2497][TX] Ensure Log Record Flush LSN is Set

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Set the flush LSN of log records before giving it

to the log flusher to avoid reading invalid value

if the flush is completed before setting the LSN.

- Ensure log record LSN is thread-safe.

- Warn in case of a flush with LSN = 0.

Change-Id: Ifc605c2d794339a3dc5004b462eca50ec103c717

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3087

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2 more files in changeset.
[ASTERIXDB-2497][TX] Ensure Log Record Flush LSN is Set

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Set the flush LSN of log records before giving it

to the log flusher to avoid reading invalid value

if the flush is completed before setting the LSN.

- Ensure log record LSN is thread-safe.

- Warn in case of a flush with LSN = 0.

Change-Id: Ifc605c2d794339a3dc5004b462eca50ec103c717

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3087

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

(cherry picked from commit 440e3a56b6f7d888c07575a898b79fec8848e06a)

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3573

  1. … 2 more files in changeset.
[ASTERIXDB-2494][RT] Log Allocated Datasets on Allocation Failure

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Log the currently allocated datasets on dataset memory

allocation failure.

Change-Id: Iefabb9c080c8937a2d2fd928dadde4993d4dd226

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3070

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][STO] Ensure First Component ID is Initialized

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Initialize the component id generator from the primary

index checkpoint, if exits, as soon as it is created.

- Ensure the first component id is passed to all indexes.

Change-Id: I246f9373f950e2f9a2c63f86746462e42a3f1c62

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2948

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./DatasetLSMComponentIdGeneratorFactory.java
  1. … 12 more files in changeset.
[ASTERIXDB-2444][STO] Avoid Using System Clock in Storage

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Replace the usage of system clock timestamps in LSM

index components file names by a sequencer. The next

sequence id to use is determined by checking the list

of existing components on disk. Note that due to a

rollback, an index checkpoint file may have last valid

component sequence which is greater than what is on disk.

This should not cause any issues since only components

that have a sequence greater than that appears in the

checkpoint will be deleted.

- Replace the usage of system clock timestamps in LSM

index components ids by a monotonically increasing

sequencer. The sequencer is initialized after restarts

by the last valid component id that appears in the

index checkpoint.

- Refactor the logic to generate flush/merge file names.

- Refactor the logic to check invalid components.

- Adapt test cases to new naming format.

Change-Id: I9dff8ffb38ce8064a199d03b070ed1f5b924b8a4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2927

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 23 more files in changeset.
[ASTERIXDB-1130][STO]: JSON serialization for persisted objects

- user model changes: no

- storage format changes:

This change replaces the use of Java serialization for persisted objects

such as dataset/index metadata, checkpoints, ect...

This will break backward compatibly with any existing AsterixDB instance.

However, the change is needed to enable future backward compatibility support

for persisted objects.

- interface changes:

IJsonSerializable: contains API to serialize a class as a JsonNode.

IPersistedResourceRegistry: contains a mapping between an IJsonSerializable

class and a unique type id. An IPersistedResourceRegistry is responsible

for generating the class identifier in the JSON output.

The class identifier will always contain the following attributes:

@type: a unique type id that identifies the object type.

@version: the version of the serialized class.

@class: the serialized class full name.

Any registered class with PersistedResourceRegistry must provide

a static fromJson(IPersistedResourceRegistry, JsonNode) method for

deserialization. This is ensured during the class registration process.

Change-Id: I5b103e06eab6627dbfe9d531caae1a3ac4b296da

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2752

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +17
    ./AsterixVirtualBufferCacheProvider.java
    • -0
    • +15
    ./CorrelatedPrefixMergePolicyFactory.java
    • -1
    • +17
    ./DatasetLSMComponentIdGeneratorFactory.java
  1. … 125 more files in changeset.
[NO ISSUE][STO] Skip deleting unallocated memory component

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- This change fixes the component delete logic

- It first ensures that memory is allocated for the index

- Then it checks whether the memory component is to be deleted

- This is important since there might be cases where primary

index has the memory allocated but not the secondary and

without allocating secondary and deleting it, we could end

up with memory components with different ids.

Change-Id: I0c6c7968830f3c9241bd036c0a330be1400349b4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2668

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE][STO] Improve logging when failing to schedule a flush

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Log partition info when failing to schedule a flush operation.

- Fix isCurrentMutableComponentEmpty() to consider the

UNREADABLE_UNWRITABLE state.

Change-Id: I9bbc0be36633b896becc16d3eeddeef980db5802

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2662

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +18
    ./PrimaryIndexOperationTracker.java
  1. … 1 more file in changeset.
[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- This change introduces some improvements to storage

operations.

- Local RecoveryManager is now extensible.

- Bulk loaders now call the IO callback similar to

Flushes, making them less special and creating a

unified lifecycle for adding an index component.

- As a result, The IndexCheckpointManager doesn't need

to have a special treatment for components loaded

through the bulk load operation.

- Component Id have been added to the index checkpoint

files.

- Cleanup for the code of local recovery for failed flush

operations.

- Ensure that after local recovery of flushes, primary

and secondary indexes have the same index for mutable

memory component.

- The use of WAIT logs to ensure in-flight flushes

are scheduled didn't work as expected. A new log type

WAIT_FOR_FLUSHES was introduced to acheive the expected

behavior.

- The local test framework was made Extensible to support

more use cases.

- Test cases were added for component ids in checkpoint files.

The following scenarios were covered:

- Primary and secondary both have values when a flush is

shceduled.

- Primary have values but not secondary when a flush is

scheduled.

- Primary is empty and an index is created through bulk

load.

- Primary has a single component and secondary is created

through bulk load.

- Primary has multiple components and secondary is created

through bulk load.

- Each primary opTracker now keeps a list of ongoing flushes.

- FlushDataset now waits only for flushes only and

not all io operations.

- Previously, we had many flushes scheduled on open datasets.

This was not detected but after this change, a failure

is thrown in such cases.

- Flush operations dont need to extend the comparable

interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2632

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -6
    • +33
    ./PrimaryIndexOperationTracker.java
  1. … 87 more files in changeset.
[NO ISSUE][STO] Add consistency to flush lifecycle

- user model changes: no

- storage format changes: yes

- renamed AbstractLSMIOOperationCallbackFactory

to LSMIOOperationCallbackFactory

- useless classes have been removed.

- LSMBTreeIOOperationCallbackFactory

- LSMBTreeWithBuddyIOOperationCallbackFactory

- LSMInvertedIndexIOOperationCallbackFactory

- LSMRTreeIOOperationCallbackFactory

- interface changes: yes

Details:

- Previously, flushes have different lifecycle depending

on the memory component state

- not allocated

- allocated

- modified

- In certain cases, flush operations are skipped alltogether

- IO Operation callbacks became complicated and difficult

to maintain since calls are done differently in different

cases.

- In certain cases, afterFinalize is called on the IO

Operation callbacks even if beforeOperation was never

called.

- In this change, flushes go through the same lifecycle

events regardless of the state of the memory component.

- In addition, primary and secondary memory components

would reside in different virtual buffer caches due

to skipped flushes, or due to having the secondary

index created when the primary index's memory component

is residing on the virtual buffer cache with index !=0.

- Moreover, when flushes are lagging and all memory

components are being flushed, search operations assumes

the oldest of the memory component is the newest and

produces incorrect results.

- In addition, in case of a failed flush of a component,

the IO scheduler would skip it and flush the next

component. This would produce a bad state on disk.

- In this change, a failed flush can be retried. otherwise,

all future flushes of the component fail due to the failure

of the previously failed flush.

- Previously, when a component fails to modify an index due

to flush failures, it assumes disk is full.

- With this change, the modification failure reports the

original cause of the failed flush.

Change-Id: I29f7992ec6c0f71c5b63d45800b2fb590d651e4b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2584

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +42
    ./DatasetInfoProvider.java
    • -2
    • +1
    ./DatasetLSMComponentIdGeneratorFactory.java
    • -25
    • +31
    ./PrimaryIndexOperationTracker.java
  1. … 156 more files in changeset.
[NO ISSUE][RT] Inherit Closeable in IFrameOperationCallback

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Primary upsert operator uses an IFrameOperationCallback

to perform per frame operations. In this change,

IFrameOperationCallback extends Closeable and allow the

callback to release resources on the close of the pipeline.

Change-Id: Iaacd6538c27d5498868256616a793e0ebcec3729

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2536

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 4 more files in changeset.
[NO ISSUE] Incremental cleanup of deprecated exception ctors

Change-Id: I1e7c3655828fc6530cef83ea502a6cfbf41acddf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2533

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 240 more files in changeset.
[NO ISSUE][RT] Fix wait for IO operations

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Some operations such as close dataset, delete components,

and drop index need to wait for IO operations.

- Before this change, the wait for IO operation would just check

the count of IO operations on the dataset info. This is not

enough as a flush might have started by writing the flush log to

the log tail but only on the flush of that log, we trigger the

flush operation and the count of IO operation increases.

- To address this problem, we write a wait log before we check the

IO operation count ensuring that any flush logs in the log tail

have been flushed and counts incremented.

Change-Id: Ibfa883410cd24e0af54732f7ea6f1b4eb2184e8e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2495

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE][TX] Ensure Uncommitted Atomic Txns Not Flushed

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Make all metadata indexes modifications as

force modifications.

- Do not decrement ops of atomic transactions

until they fully commit or abort to prevent

flushing partial records.

- Do not schedule flush if a force modification

starts before the flush log is written to disk.

- Unify code path for completing operations

after commit/abort in op tracker.

- Remove unneeded update log commit notification.

- Add test case for failing flush due to force

modification.

Change-Id: If8d5df630f1d9119002ef91da5c282da18901acc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2456

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -22
    • +31
    ./PrimaryIndexOperationTracker.java
  1. … 8 more files in changeset.
[NO ISSUE][STO] Correct Metadata Datasets VBC Partitions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure metadata datasets number of VBCs is

always one.

Change-Id: I3217129dcf250287f6ccdf3223fad00a6b2fd66f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2422

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 1 more file in changeset.
[NO ISSUE][TEST] Test flushing of empty component

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Some use cases require flushing of component that have no data

but some Metadata. This change adds a test to ensure that always

works.

Change-Id: If921323dfe03cbd70edc3a8ea8e01226d7527bb3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2421

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 2 more files in changeset.
[ASTERIXDB-2188] Ensure recovery of component ids

- user model changes: no

- storage format changes: yes.

Flush log record format changes.

- interface changes: no

Details:

- Add flush component ids to the flush log record. Upon

seeing a flush log record during recovery, schedule

a flush to all indexes in this partition s.t. LSN>maxDiskLSN

to ensure component ids are properly maintained upon

failed flushes.

- Add a test case to ensure the correctness of the recovery logic

of component ids

Change-Id: I8c1fc2b209cfb9d3dafa216771d2b7032eb99e75

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2408

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +10
    ./PrimaryIndexOperationTracker.java
  1. … 16 more files in changeset.
[NO ISSUE][TXN] Prevent deadlock in Metadata transactions

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Flushes in metadata datasets are triggerred by entity update

logs, unlike regular transactions where flushes are triggerred

by entity commit logs.

- Because entity update logs can be writting to disk before the

operation completes, there is a chance that an operation that

caused the component to be full exits after the log is flushed

and so, a flush operation is not scheduled.

- This change proposes a simple fix. The fix is that metadata

operation will also check if a flush is needed and will schedule

one if needed.

Change-Id: I07a18840dc54fe052b7bd294595f816f6d8a4d2f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2413

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -5
    • +10
    ./PrimaryIndexOperationTracker.java
  1. … 1 more file in changeset.
[ASTERIXDB-2231][STO] Separate primary op tracker for each partition

- user model changes: no

- storage format changes: no.

- interface changes: yes.

Details:

- Separate primary index operation tracker for each partition, instead

of having a global one on each NC to achieve better scalability.

- As a coordinated change, separate component id generator for each

partition as well.

- Add partition to transaction context so that transaction operations

can operate on proper op tracker.

- Fixes [ASTERIXDB-2232] to calculate dataset partitions correctly.

Change-Id: I9eb3854d2343e45beeccb87b0d434e5f4efd69c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2263

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -2
    • +7
    ./DatasetLSMComponentIdGeneratorFactory.java
    • -78
    • +114
    ./DatasetLifecycleManager.java
  1. … 57 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 984 more files in changeset.
[NO ISSUE][TX] Remove Unused Fields in LogRecord

- user model changes: no

- storage format changes: no

- interface changes: yes

- Remove nodeId from ILogRecord

Details:

- Remove the following fields which are

no longer needed from LogRecord:

- numOfFlushedIndexes

- nodeId

- The removed fields were not persisted

and only used for remote logs.

Change-Id: If293953e51525e27110909203941444c6d7eb344

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2269

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 4 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 388 more files in changeset.
[NO ISSUE][STO] Introduce Index Checkpoints

- user model changes: no

- storage format changes: yes

- Add index checkpoints.

- Use index checkpoint to determine low watermark

during recovery.

- interface changes: yes

- Introduce IIndexCheckpointManager for managing

indexes checkpoints.

- Introduce IIndexCheckpointProvider for tracking

IIndexCheckpointManager references.

Details:

- Unify LSM flush/merge operations completion order.

- Introduce index checkpoints which contains:

- Index low watermark.

- Latest valid LSM component

- Mapping between master replica and local replica.

- Use index checkpoints instead of LSM component metadata

for identifying low watermark in recovery.

- Use index checkpoints in replication instead of overwriting

LSN byte offset in replica component metadata.

- Replace LSN_MAP used in replication by index checkpoints.

- Replace NIO Files.find by Commons FileUtils.listFiles to

avoid no NoSuchFileException on any file deletion.

Change-Id: Ib22800002bf8ea3660242e599b3f5f20678301a8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2200

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 43 more files in changeset.