Clone Tools
  • last updated 25 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2541][STO] Introduce GreedyScheduler

- user model changes: yes.

Add new option: storage.io.scheduler (async/greedy)

- storage format changes: no.

- interface changes: yes.

Introduce IIndexCursorStats

Details:

- Introduce GreedyScheduler that always executes the merge

operation with the smallest number of remaining pages to minimize

the number of disk components

- Introduce IIndexCursorStats to collect the statistics of index scans.

This allows GreedyScheduler to know the remaning pages of merge

operations.

- Extend AbstractIoOperation so that GreedyScheduler can pause/resume

merge operations if needed.

Change-Id: I38fe394d1180d4e3f6796064c0e6c6630b6ad303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3284

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

    • -0
    • +102
    ./hyracks/storage/am/lsm/common/impls/GreedyScheduler.java
  1. … 59 more files in changeset.
[ASTERIXDB-2600][STO] Introduce ConcurrentMergePolicy

- user model changes: yes. Add a new merge policy and make it as default

- storage format changes: no.

- interface changes: no.

Details:

- Introduce ConcurrentMergePolicy that performs concurrent merges

without the maximum component size.

- Make this merge policy as the default merge policy in AsterixDB since

the PrefixMergePolicy has made some wrong design decisions.

Change-Id: I2ed79847584b9fe846d62ad56ee094863538a2a2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3463

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 31 more files in changeset.
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 128 more files in changeset.
[ASTERIXDB-2599][STO] Cleanup compression LAFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Cleanup compression LAFs after merge.

- Delete LAF files when merged components contain

previous components.

- Make sure the recovery after non-graceful shutdown

(in the middle of cleanup) deletes the component and its LAF.

Change-Id: I17adb6145f7bf77470fd82f04321faf7a4007bf7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3498

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2310][STO]Enforce Key Uniquness using PKIndex

- user model changes: no

- storage format changes: yes. Primary key index

now has bloom filters.

- interface changes: no

Details:

- Add bloom filters to primary key index.

- Introduce LSMPrimaryInsertOperator to separate uniqueness check from

the primary index. When the primary key index is available, it will be

used for uniqueness check. This implementation of this operation is

similar to LSMPrimaryUpsertOperator.

Change-Id: I7a52bb75ee5b14521972999df2f45ba62adc5af1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2453

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 52 more files in changeset.
[NO ISSUE][RT] Improve PreclusteredGroupWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Modified PreclusteredGroupWriter to only save group fields

from a last tuple in a frame instead of the whole frame

- move PermutingFrameTupleReference and PermutingTupleReference

from 'hyracks-storage-am-common' to 'hyracks-dataflow-common'

Change-Id: Ic75de2e6b64d0aacaf48096ecc9d47fc8e95c9cf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3351

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 34 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 625 more files in changeset.
[NO ISSUE][API][MTD] Refactor query service servlet

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Refactor query service servlet in addition to some code

clean-ups.

Change-Id: Idbc81219cbe2f337b3c4aaf4a242ef2ebeb165c5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3038

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 7 more files in changeset.
[NO ISSUE][STO] Let ILSMIOOperations clean up for themselves

- user model changes: no

- storage format changes: no

- interface changes: yes

remove ILSMIndex.cleanUpFilesForFailedOperation

remove ILSMIOOperation.getComponentFiles

add ILSMIOOperation.cleanup

Details:

Instead of having ILSMIndex instances clean up for ILSMIOOperations, let

the ILSMIOOperations cleanup their component files.

Also avoids a dependency of the interface ILSMIOOperation on

o.a.hyracks.storage.am.lsm.common.impls.LSMComponentFileReferences.

Change-Id: I97a31d018139dcdea15196ae52d91eed5e780fbd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3034

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2422][STO] Introduce compressed storage

- user model changes: yes

- Add new configuration in the with-caluse to enable compression

- Add new nc configuration in the config file

- storage format changes: yes

- Pages of the primary index can be compressed

- Add a companion file (Look Aside File) with the compressed index

- Allow optional values in the LocalResource

- Add compression information in Metadata.Dataset

- interface changes: yes

- ICCApplicationContext:

- Add getCompressionManager()

- IBufferCache:

- Add getCompressedFileWriter(int fileId)

- ICachedPageInternal:

- Add setCompressedPageOffset(long offset)

- Add getCompressedPageOffset()

- Add setCompressedPageSize(int size)

- Add getCompressedPageSize()

Details:

- Add new integration test for this patch

- Fix ASTERIXDB-2464

- Add ddl-with-clause type validator

Additional details in the design document:

https://cwiki.apache.org/confluence/display/ASTERIXDB/Compression+in+AsterixDB

Change-Id: Idde6f37c810c30c7f1a5ee8bcbc1e3e5f4410031

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2857

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 191 more files in changeset.
[ASTERIXDB-2453] Add Improved Constant Merge Policy

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The current constant merge policy is unsuable because of its high

merge cost, i.e., O(N*N) where N is the number of flushes. This patch

replaces the previous constant merge policy with a more efficient policy

that still enforces a maximum number of components but greatly lowers

the merge cost.

- Extend AbstractLSMIndex with a method to return the total number of

flushes, based on the file name sequencer. This is required by the new

policy.

Change-Id: Ie5f83a4d5fdd3f036b823c906df1760f5110ae0a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2971

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[NO ISSUE][REPL] Ensure Valid Component ID is Initialized On Replica Sync

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Currently, the first time a replica is synchronized from master,

the valid component id on each replicated index's initial checkpoint

will be the initial value of a component id (-1). This value is

fixed when the the replica receives a flushed component from

the index. However, if the master fails before any component is

flushed to a replica and that replica is promoted to master, it

will start from an invalid component id. This change ensures that

the initial checkpoint of replicated indexes is initialized to

the maximum component id that appears on master. This will ensure

that if the replica is promoted, it will at least start from

a component that wasn't previously used on master.

- Replace assertion of component ids validation by illegal state.

Change-Id: I85395ad823a630725c4cab4bead1c61546dc61ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2973

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 7 more files in changeset.
[NO ISSUE][STO] Ensure Files From Failed Bulkload Are Deleted

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- If a bulkload is interrupted at the end-stage (i.e. after all

tuples have been added but before the component is added to

the index's disk components), then the bulkload operation will

fail but the files from the failed operation are not deleted.

This change ensures that if a bulkload fails in such stage,

the files of the generated component are deleted to avoid

file is already mapped exception if the bulkload is attempted

again.

- Ensure disk component is only destroyed once in case of

bulkload cleanup.

- Stop attempting to write a checkpoint early if the failure was

due to the thread being interrupted.

Change-Id: I7f63054ac72a9482e779e49eb4da658a08fe7e9d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2970

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE][STO] Ensure Index Component Sequence Uniqueness

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Currently, if two or more flushes are scheduled before any

of them create its files on disk, we will end up reusing

the same component sequence since the last used component

sequence is based on the files on disk. This change ensures

that the last use component sequence is only initialized

from the disk files and after that the last used component

sequence is incremented in memory.

Change-Id: I478ca3e2c9a98bab14a9145cc8b23eadbd0eab08

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2958

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][STO] Report batch operation failure before exiting

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- In certain cases, caller of a batch operation call are

interested in failure events.

- In those cases, we used to report failure after exiting

the components but with this change, failure reporting

happens before the exit.

Change-Id: I0c22b6bddfe8f12ef8e3c59dae0b0c585137a126

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2956

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 3 more files in changeset.
[NO ISSUE][STO] Ensure First Component ID is Initialized

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Initialize the component id generator from the primary

index checkpoint, if exits, as soon as it is created.

- Ensure the first component id is passed to all indexes.

Change-Id: I246f9373f950e2f9a2c63f86746462e42a3f1c62

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2948

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 12 more files in changeset.
[ASTERIXDB-2444][STO] Avoid Using System Clock in Storage

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Replace the usage of system clock timestamps in LSM

index components file names by a sequencer. The next

sequence id to use is determined by checking the list

of existing components on disk. Note that due to a

rollback, an index checkpoint file may have last valid

component sequence which is greater than what is on disk.

This should not cause any issues since only components

that have a sequence greater than that appears in the

checkpoint will be deleted.

- Replace the usage of system clock timestamps in LSM

index components ids by a monotonically increasing

sequencer. The sequencer is initialized after restarts

by the last valid component id that appears in the

index checkpoint.

- Refactor the logic to generate flush/merge file names.

- Refactor the logic to check invalid components.

- Adapt test cases to new naming format.

Change-Id: I9dff8ffb38ce8064a199d03b070ed1f5b924b8a4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2927

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 19 more files in changeset.
[NO ISSUE][OTH] Log level / SonarQube fixes

Change-Id: I8e9a31ebea1eac6bed26c57666db1fd4b1208295

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2891

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1 more file in changeset.
[NO ISSUE][OTH] Log Level Fixes

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Ib5fb7e3140bf36cf4ba5857b9fda8e6cc4bdfa20

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2883

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 10 more files in changeset.
[NO ISSUE][OTH] Log Level Fixes

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I6c83103e16b7a356167639a7a4d5c70d93335a53

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2865

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 3 more files in changeset.
[NO ISSUE][STO] Notify of completion of IO request in finally

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Notify completion of a request in a finally clause to ensure

waiting thread is always notified.

Change-Id: I8b3003b47b6b181856faf82aca6e828ee014527c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2839

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1 more file in changeset.
[ASTERIXDB-2414][STO] Remove deleted component files from buffer cache

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When activating an LSM index, we remove files of components

that were merged into a bigger component but not cleaned up yet.

- However, we sometimes leave a file reference mapped in the buffer

cache even when the file is removed from disk.

- This change ensures that all files are removed from the buffer

cache as well.

Change-Id: If0f11bc222662e4b50c1b47b1dfa6b30d1463b2e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2822

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 7 more files in changeset.
[ASTERIXDB-2414][STO] Fix name of merge files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- A bug is found where merge file names are created incorrectly

where start and end components are reversed.

- The bug is fixed and an explicit check for the invariance was

added.

Change-Id: I861765bc0f293bdfdf0285f97884d536204fdb1e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2820

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 3 more files in changeset.
[NO ISSUE] Allow MetadataProvider config to store non String values

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- In many cases, we would like to associate a value with a key

in MetadataProvider to be accessed during the compilation

of jobs. However, currently, there is no place to store

such values, so we ended up storing them in the config

map.

- The config map is a <String, String> map and so, we would

write our values as a string and then parse them when needed.

- To avoid this, and to avoid introducing a new map, we simply

change the config stored in MetadataProvider from <String,String>

to <String, Object>.

Change-Id: I55b392ad199d74b0f3cffdc38b54593b12ec1a06

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2795

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

  1. … 23 more files in changeset.
[NO ISSUE] Report all BufferCache write failures.

- user model changes: no

- storage format changes: no

- interface changes: yes

+ IPageWriteFailureCallback: used to notify async

IO caller when something goes wrong.

Details:

- Before this change, it is possible for failures to

be lost and for bulkload operations to not be

aware of failure to write some pages. This can be

dangerous.

- To avoid this, when sending a page to be written

a PageWriteFailureCallback is associated with the

page to notify the caller that a failure took place.

Change-Id: I97fd3dccff85dab84d644359be6f66b15ee708ef

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2787

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 33 more files in changeset.
[ASTERIXDB-1130][STO]: JSON serialization for persisted objects

- user model changes: no

- storage format changes:

This change replaces the use of Java serialization for persisted objects

such as dataset/index metadata, checkpoints, ect...

This will break backward compatibly with any existing AsterixDB instance.

However, the change is needed to enable future backward compatibility support

for persisted objects.

- interface changes:

IJsonSerializable: contains API to serialize a class as a JsonNode.

IPersistedResourceRegistry: contains a mapping between an IJsonSerializable

class and a unique type id. An IPersistedResourceRegistry is responsible

for generating the class identifier in the JSON output.

The class identifier will always contain the following attributes:

@type: a unique type id that identifies the object type.

@version: the version of the serialized class.

@class: the serialized class full name.

Any registered class with PersistedResourceRegistry must provide

a static fromJson(IPersistedResourceRegistry, JsonNode) method for

deserialization. This is ensured during the class registration process.

Change-Id: I5b103e06eab6627dbfe9d531caae1a3ac4b296da

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2752

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 119 more files in changeset.
[NO ISSUE][STO] Skip deleting unallocated memory component

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- This change fixes the component delete logic

- It first ensures that memory is allocated for the index

- Then it checks whether the memory component is to be deleted

- This is important since there might be cases where primary

index has the memory allocated but not the secondary and

without allocating secondary and deleting it, we could end

up with memory components with different ids.

Change-Id: I0c6c7968830f3c9241bd036c0a330be1400349b4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2668

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -16
    • +23
    ./hyracks/storage/am/lsm/common/impls/LSMHarness.java
  1. … 1 more file in changeset.
[NO ISSUE][STO] Only delete allocated components

Change-Id: I372127a0dec21148efa1a94cf6c976818963d761

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2671

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -10
    • +18
    ./hyracks/storage/am/lsm/common/impls/LSMHarness.java
[NO ISSUE][STO] Improve logging when failing to schedule a flush

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Log partition info when failing to schedule a flush operation.

- Fix isCurrentMutableComponentEmpty() to consider the

UNREADABLE_UNWRITABLE state.

Change-Id: I9bbc0be36633b896becc16d3eeddeef980db5802

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2662

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1 more file in changeset.
[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- This change introduces some improvements to storage

operations.

- Local RecoveryManager is now extensible.

- Bulk loaders now call the IO callback similar to

Flushes, making them less special and creating a

unified lifecycle for adding an index component.

- As a result, The IndexCheckpointManager doesn't need

to have a special treatment for components loaded

through the bulk load operation.

- Component Id have been added to the index checkpoint

files.

- Cleanup for the code of local recovery for failed flush

operations.

- Ensure that after local recovery of flushes, primary

and secondary indexes have the same index for mutable

memory component.

- The use of WAIT logs to ensure in-flight flushes

are scheduled didn't work as expected. A new log type

WAIT_FOR_FLUSHES was introduced to acheive the expected

behavior.

- The local test framework was made Extensible to support

more use cases.

- Test cases were added for component ids in checkpoint files.

The following scenarios were covered:

- Primary and secondary both have values when a flush is

shceduled.

- Primary have values but not secondary when a flush is

scheduled.

- Primary is empty and an index is created through bulk

load.

- Primary has a single component and secondary is created

through bulk load.

- Primary has multiple components and secondary is created

through bulk load.

- Each primary opTracker now keeps a list of ongoing flushes.

- FlushDataset now waits only for flushes only and

not all io operations.

- Previously, we had many flushes scheduled on open datasets.

This was not detected but after this change, a failure

is thrown in such cases.

- Flush operations dont need to extend the comparable

interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2632

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 75 more files in changeset.