asterix-transactions

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Merge branch 'stabilization-f69489' into 'mad-hatter'

Change-Id: I07cab31b76128dfa5cea9b49620b1c074b385c81

  1. … 1 more file in changeset.
[NO ISSUE][STO] Force Checkpoints/Index Metadata Files to Disk

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure the system checkpoint file is forced to disk.

- Ensure index metadata files and checkpoints are forced

to disk.

Change-Id: I27ab8ed39e23760b514640d77532a1dc24a97992

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3943

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE][TXN] Remove Synchronization on TransactionContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently the flag indicating if a transaction timed out

synchronizes on the transaction context which makes it

a hot spot when multiple threads on the same transactions

attempting to access that flag. This change removes that

synchronization and replaces it with a volatile variable.

Change-Id: Ib14c07a051d27a5b47c62ff35769670e6ab01fd2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3579

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][STO] Fix Time Unit Comparison For Periodic Flushes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The dataset checkpoint interval is specified in the

configuration using seconds while the dataset last

flush time is tracked in nanos. This change fixes the

comparison by converting the dataset checkpoint

interval to nanos.

Change-Id: Id5a9b86207ea25ba44840af01324041525ac5bd9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3542

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Merge commit '8c99391' from stabilization-f69489

Change-Id: I139c265ba998c32ec049c8a8bbd7a5a213895d2a

  1. … 7 more files in changeset.
[NO ISSUE][STO] Tolerate Corrupted System Checkpoint Files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not delete corrupted checkpoint files until a valid

checkpoint is persisted. This ensure a forged checkpoint

that will force recovery to start from the begining is

always used until a valid checkpoint is found.

- Attempt to read the latest checkpoint file right after

writing it and before attempting cleaning up invalid and

old checkpoint files.

- Use on disk files to determine next checkpoint id to

account for existing corrupted checkpoints.

- Maintain two older checkpoint files in addition to the

latest one.

- Catch all exceptions on checkpointing failures since it is

a try operation.

- Add test scenairo for the new checkpoints clean up behavior.

Change-Id: Iea689f5a644351491d9748273bb2158e8179f54d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3496

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 6 more files in changeset.
Merge commit '0561d10' from stabilization-f69489

Change-Id: I18115329ce7ab3501e7fcc9c6dc06d0e87c97688

  1. … 1 more file in changeset.
[NO ISSUE][REPL] Suspend Dataset Checkpointing on Replica Sync

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Before synchronizing replicas, stop datasets checkpointing to

prevent new files from being generated due to async IO operations

triggered by checkpointing.

- Instead of sync'ing current files to replicas then scheduling a flush

and sync'ing any newly generated files, just flush datasets before

the initial sync then sync all the files in one go.

Change-Id: I058fd48bc0fb89a1e16448ce516c3410bb4d681d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3469

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE][OTH] Add API To Get Dataset Size

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add an API that returns the on disk total size of a dataset

and its indexes on a node.

Change-Id: Iaff87bbe1f2417689f7827deaf03fcddd64ca7e4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3425

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2310][STO]Enforce Key Uniquness using PKIndex

- user model changes: no

- storage format changes: yes. Primary key index

now has bloom filters.

- interface changes: no

Details:

- Add bloom filters to primary key index.

- Introduce LSMPrimaryInsertOperator to separate uniqueness check from

the primary index. When the primary key index is available, it will be

used for uniqueness check. This implementation of this operation is

similar to LSMPrimaryUpsertOperator.

Change-Id: I7a52bb75ee5b14521972999df2f45ba62adc5af1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2453

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 51 more files in changeset.
[NO ISSUE][STO] Ensure Invalid Merged Components Are Deleted

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When cleaning up an index's invalid components, check the

components end sequence in addition to the start sequence

to ensure invalid merged components with are cleaned too.

- Add test case.

Change-Id: I80d72b1b614718e2e283bc72a874c140d178d1e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3331

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE] Bump versions to 0.[39].4.2-SNAPSHOT

Change-Id: If0cf05002c8019c1e1af43a18e75e617c0f7d4c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3299

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 101 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 621 more files in changeset.
[ASTERIXDB-2491][TXN] Recovery fixes

- user model changes: no

- storage format changes: yes[1]

- interface changes: no

Details:

- Change field offset type from integer16 to integer32

- Add recovery test

- Add version to log entries

[1] LogRecord format change with this patch; old LogRecord format can still

be processed, however old instances cannot read new log format

Change-Id: Iaf14b9a73a0239763bfeb0ce2d81cf952e6d72d3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3065

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 22 more files in changeset.
[NO ISSUE][*DB] LogManagerWithReplication optimizations

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- eliminate boxing/unboxing of txnids

- remove redundant hashset contains call

Change-Id: I6e5068c6fe3f6226eabd5403440682c52662421e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3251

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[ASTERIXDB-2522][TX] Skip WAIT record during lock conflicts

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- A small optimization to our deadlock-free locking potocol by skipping

the WAIT record. Once the partial frame has been flushed and previous

records are committed, the locks held by the write thread will

eventually be released by the log flusher thread. There is no need to

force the writer thread to wait for the log flusher thread.

Change-Id: I6ef3979d6393d45a6b7b2eb5f09f147299b5cd9f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3232

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Advance version to AsterixDB 0.9.4.2-SNAPSHOT and Hyracks 0.3.4.2-SNAPSHOT

Change-Id:I47827d49e8a589ae885b134bf5c7557fae3b0620

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3209

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
Release Apache AsterixDB 0.9.4.1 and 0.3.4.1

Change-Id:I0af32fba00277c516b2671cc3567d61a2765f526

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3208

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 101 more files in changeset.
[NO ISSUE] Ensure serialVersionUIDs are present

Serializable classes should define a serialVersionUID; define those

where missing. For classes present in 0.9.4.x, initialize the value

to the auto-configured serialVersionUID on those versions of the class

Change-Id: Ia92d0c81f3681dce7ec0f643e2edf77e2ac4dbc5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3141

Tested-by: Michael Blow <mblow@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 51 more files in changeset.
Merge commit '1ff69c0' from 'stabilization-f69489' into 'master'

Change-Id: Ic8aa90be3ba3645ad22d21cc54abc83ab7008598

  1. … 24 more files in changeset.
[ASTERIXDB-2506][STO] Checkpoint Datasets

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add new config to specify dataset checkpoint interval

and default its value to 10 minutes.

- Flush datasets which were not flushed for the dataset

checkpoint interval.

- Run dataset checkpoint logic as part of the CheckpointThread.

- Improve dataset async flush API to accept any LSM index predicate.

- Cleanup exception handling in CheckpoinThread.

- Add test case for dataset checkpoint.

Change-Id: I38a73a43a4b1b7d3a8ac79dd579ed4ef8c9c6a9b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3106

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 18 more files in changeset.
[NO ISSUE] Update third-party library versions

Change-Id: I752876c0d5144019f228cb5e758892560d2c042d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3090

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

  1. … 29 more files in changeset.
[ASTERIXDB-2497][TX] Ensure Log Record Flush LSN is Set

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Set the flush LSN of log records before giving it

to the log flusher to avoid reading invalid value

if the flush is completed before setting the LSN.

- Ensure log record LSN is thread-safe.

- Warn in case of a flush with LSN = 0.

Change-Id: Ifc605c2d794339a3dc5004b462eca50ec103c717

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3087

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2 more files in changeset.
[ASTERIXDB-2497][TX] Ensure Log Record Flush LSN is Set

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Set the flush LSN of log records before giving it

to the log flusher to avoid reading invalid value

if the flush is completed before setting the LSN.

- Ensure log record LSN is thread-safe.

- Warn in case of a flush with LSN = 0.

Change-Id: Ifc605c2d794339a3dc5004b462eca50ec103c717

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3087

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

(cherry picked from commit 440e3a56b6f7d888c07575a898b79fec8848e06a)

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3573

  1. … 2 more files in changeset.
Merge release-0.9.4 into master

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f4

  1. … 38 more files in changeset.
[NO ISSUE][REPL] Ensure Valid Component ID is Initialized On Replica Sync

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Currently, the first time a replica is synchronized from master,

the valid component id on each replicated index's initial checkpoint

will be the initial value of a component id (-1). This value is

fixed when the the replica receives a flushed component from

the index. However, if the master fails before any component is

flushed to a replica and that replica is promoted to master, it

will start from an invalid component id. This change ensures that

the initial checkpoint of replicated indexes is initialized to

the maximum component id that appears on master. This will ensure

that if the replica is promoted, it will at least start from

a component that wasn't previously used on master.

- Replace assertion of component ids validation by illegal state.

Change-Id: I85395ad823a630725c4cab4bead1c61546dc61ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2973

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 7 more files in changeset.
[ASTERIXDB-2444][STO] Avoid Using System Clock in Checkpoints

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Replace the usage of system clock timestamps in checkpoints

by a sequencer.

- Update Asterix/Hyracks storage version to reflect the recent

changes in storage.

- This change is expected to break storage backward compatibility.

Change-Id: Idc061e6eaccfb308b29a5a263b77a0a849694d4f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2961

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 5 more files in changeset.
[NO ISSUE][TX] Create New Log File Before Deleting Old Files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure next log file is created after a sharp check

point before deleting old files. This is to prevent

the case if a crash happens right after deleting the

old files but before creating the new one, then the

next time the system starts up, it will start with

log file id 0 which is wrong.

- Log the details of latest index checkpoint when the

low watermark of the new checkpoint is less than the

low watermakr of the latest checkpoint.

Change-Id: I4817f697b43daff55726909ab074ec30a1c224ce

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2959

Reviewed-by: Michael Blow <mblow@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 3 more files in changeset.
Advance version to AsterixDB 0.9.5-SNAPSHOT and Hyracks 0.3.5-SNAPSHOT

Change-Id:Ied2b685f015e0f35563ebd0510231a5d663494f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2953

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 99 more files in changeset.
Release Apache AsterixDB 0.9.4 and 0.3.4

Change-Id:Ib015ae6e13f38eb726512a9c5ddeeda428ad1dff

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2952

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 100 more files in changeset.