Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][OTH] Eliminate Object Creation on Thread Stats Lookup

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Use the thread object for thread stats lookup

rather than the thread id to avoid boxed objects

creation on stats lookup.

Change-Id: I5b8b26be9d3439889e21cf097ea86c413e973e1d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3763

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -185
    • +0
    ./AsyncFIFOPageQueueManager.java
    • -0
    • +40
    ./IPageWriteCallback.java
    • -0
    • +42
    ./NoOpPageWriteCallback.java
  1. … 128 more files in changeset.
[NO ISSUE][RT] Add Thread-Based Stats Collector

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add infra to allow collecting thread-based stats during

runtime for any thread that belongs to a task.

- Collect number of pinned pages per thread and report it

in the TaskProfile.

- Aggregate pinned pages counters from all job tasks and

report it as diskIoCount in the metrics field in the json

response. The plan is to move this stats to the profile

field when it is introduced.

- Collecting pinned pages stats is currently enabled by

default for any job with IndexSearchOperatorNodePushable.

The plan is to allow enabling/disabling as part of the

job profiling change.

- Add test case for diskIoCount metric.

- Remove unused IndexSearchOperatorNodePushable constructor.

Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 27 more files in changeset.
[ASTERIXDB-2599][STO] Cleanup compression LAFs

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Cleanup compression LAFs after merge.

- Delete LAF files when merged components contain

previous components.

- Make sure the recovery after non-graceful shutdown

(in the middle of cleanup) deletes the component and its LAF.

Change-Id: I17adb6145f7bf77470fd82f04321faf7a4007bf7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3498

Reviewed-by: Wail Alkowaileet <wael.y.k@gmail.com>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -7
    • +32
    ./AbstractBufferedFileIOManager.java
  1. … 7 more files in changeset.
[NO ISSUE] Make IOManager more configurable

Change-Id: I1c8ad11c2b8b983ef4bf7cf78c2f068accddfff4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3133

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Contrib: Michael Blow <mblow@apache.org>

  1. … 10 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 624 more files in changeset.
[ASTERIXDB-2422][STO] Introduce compressed storage

- user model changes: yes

- Add new configuration in the with-caluse to enable compression

- Add new nc configuration in the config file

- storage format changes: yes

- Pages of the primary index can be compressed

- Add a companion file (Look Aside File) with the compressed index

- Allow optional values in the LocalResource

- Add compression information in Metadata.Dataset

- interface changes: yes

- ICCApplicationContext:

- Add getCompressionManager()

- IBufferCache:

- Add getCompressedFileWriter(int fileId)

- ICachedPageInternal:

- Add setCompressedPageOffset(long offset)

- Add getCompressedPageOffset()

- Add setCompressedPageSize(int size)

- Add getCompressedPageSize()

Details:

- Add new integration test for this patch

- Fix ASTERIXDB-2464

- Add ddl-with-clause type validator

Additional details in the design document:

https://cwiki.apache.org/confluence/display/ASTERIXDB/Compression+in+AsterixDB

Change-Id: Idde6f37c810c30c7f1a5ee8bcbc1e3e5f4410031

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2857

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +287
    ./AbstractBufferedFileIOManager.java
    • -0
    • +90
    ./BufferCacheHeaderHelper.java
  1. … 186 more files in changeset.
[NO ISSUE][OTH] Log Level Fixes

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Ib5fb7e3140bf36cf4ba5857b9fda8e6cc4bdfa20

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2883

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 13 more files in changeset.
[NO ISSUE][OTH] Log Level Fixes

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Icbf7a3ee1c358a40918be7badc4bd00ac5d492a4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2879

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2398][STO] Ensure No Concurrent Access to FileMapManager

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently it is possible for a thread to access

FileMapManager without holding the proper lock

and therefore seeing a stale state of the map.

This change ensures that all access to the map

is synchronized on the same lock.

Change-Id: Iea7fdb0b891b4ba2aaa528b42eab47b6f841672d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2863

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 2 more files in changeset.
[NO ISSUE] Don't log interrupts as WARNings

Change-Id: I0e847b8197fa12f1ce235dde404df24196939a83

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2855

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 7 more files in changeset.
[NO ISSUE] Report all BufferCache write failures.

- user model changes: no

- storage format changes: no

- interface changes: yes

+ IPageWriteFailureCallback: used to notify async

IO caller when something goes wrong.

Details:

- Before this change, it is possible for failures to

be lost and for bulkload operations to not be

aware of failure to write some pages. This can be

dangerous.

- To avoid this, when sending a page to be written

a PageWriteFailureCallback is associated with the

page to notify the caller that a failure took place.

Change-Id: I97fd3dccff85dab84d644359be6f66b15ee708ef

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2787

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -20
    • +31
    ./AsyncFIFOPageQueueManager.java
    • -0
    • +44
    ./HaltOnFailureCallback.java
    • -0
    • +43
    ./IPageWriteFailureCallback.java
    • -0
    • +42
    ./PageWriteFailureCallback.java
  1. … 37 more files in changeset.
[NO ISSUE][STO] Limit BufferCacheHeaderHelper Pool Size

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Limit the BufferCacheHeaderHelper pool size to the IO

requests queue size.

Change-Id: I7914241110e9fe3e5e21f35cf03ba6ecdf132343

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2727

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 2 more files in changeset.
[NO ISSUE][STO] Improve error handling of writing deleted file

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When the bulkloader is aborted, the file would be deleted from the

buffer cache but there could still be unflushed pages in the write

queue. Instead of throwing exceptions, simply ignore the case when

writting to non-existing files (as before).

Change-Id: I3f693ef178e0ab68c2a28b0edc6ffeaaeca1c520

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2651

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2367][STO] Various Fixes for BufferCache

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure fileInfo is deleted from FileInfoMap when delete a file from

buffer cache

- When delete old components after merge, ensure components are first

purged to avoid sweep old pages

- When open/create/delete files from buffer cache, use file-level

synchronization to reduce blocking. Certain operations such as

createFile/deleteFile can take long time if the disk is busy.

Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2604

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 5 more files in changeset.
[NO ISSUE][STO] Add consistency to flush lifecycle

- user model changes: no

- storage format changes: yes

- renamed AbstractLSMIOOperationCallbackFactory

to LSMIOOperationCallbackFactory

- useless classes have been removed.

- LSMBTreeIOOperationCallbackFactory

- LSMBTreeWithBuddyIOOperationCallbackFactory

- LSMInvertedIndexIOOperationCallbackFactory

- LSMRTreeIOOperationCallbackFactory

- interface changes: yes

Details:

- Previously, flushes have different lifecycle depending

on the memory component state

- not allocated

- allocated

- modified

- In certain cases, flush operations are skipped alltogether

- IO Operation callbacks became complicated and difficult

to maintain since calls are done differently in different

cases.

- In certain cases, afterFinalize is called on the IO

Operation callbacks even if beforeOperation was never

called.

- In this change, flushes go through the same lifecycle

events regardless of the state of the memory component.

- In addition, primary and secondary memory components

would reside in different virtual buffer caches due

to skipped flushes, or due to having the secondary

index created when the primary index's memory component

is residing on the virtual buffer cache with index !=0.

- Moreover, when flushes are lagging and all memory

components are being flushed, search operations assumes

the oldest of the memory component is the newest and

produces incorrect results.

- In addition, in case of a failed flush of a component,

the IO scheduler would skip it and flush the next

component. This would produce a bad state on disk.

- In this change, a failed flush can be retried. otherwise,

all future flushes of the component fail due to the failure

of the previously failed flush.

- Previously, when a component fails to modify an index due

to flush failures, it assumes disk is full.

- With this change, the modification failure reports the

original cause of the failed flush.

Change-Id: I29f7992ec6c0f71c5b63d45800b2fb590d651e4b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2584

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

  1. … 159 more files in changeset.
[NO ISSUE][STO] Move the IO threads from BufferCache to IOManager

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Move the IO threads from BufferCache to IOManager to cover all

IO uses that go through the IOManager.

Change-Id: Ic02b456826ae7abc2619a7eec3f90b48717b0adb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2417

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 16 more files in changeset.
[NO ISSUE][STO] Fix deadlock in buffer cache

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- When two threads concurrently attempt to pin a page,

one as a new page, the other as an existing page, a deadlock

might happen.

- The deadlock is caused by the reader as a new page setting the

valid flag before the ReaderThread attempts to read the page.

- The fix is to have the reader thread ignore the read request

and have the reader as a new page notify all the other readers

that the page is valid.

Change-Id: I50cf18d96228a1ad78ce11f5258252e8d0107d86

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2407

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE] Correct import of InvokeUtil in BufferCache

- user-model changes: no

- storage format changes: no

- interface changes: no

Details:

- InvokeUtil was moved just when the previous change was submitted.

- That broke the build, this fixes it.

Change-Id: I4c9c1829db05ac691f7b0e6dc459d7646496ab6b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2396

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 1 more file in changeset.
[NO ISSUE][STO] Perform IO reads in uninterruptible threads

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Previously, IO reads are performed on the task threads.

- Task threads can be interrupted which can cause deadlocks

due to a JDK synchronization bug.

- After this change, 2 IO read threads are there per IO device.

- Threads will receive IO read requests and process them.

- Such threads are never interrupted and are killed through

the use of a poison pill.

Change-Id: Id28d57a222f42962284b24296cb9b91658e5dc77

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2387

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 2 more files in changeset.
[ASTERIXDB-2204][STO] Fix implementations and usages of IIndexCursor

- user model changes: no

- storage format changes: no

- interface changes: yes

- IIndexCursor.close() is now idempotent and can be called on

a closed cursor.

- IIndexCursor.destroy() is now idempotent and can be called

on a destroyed cursor.

- Add IIndexAccessor.destroy() letting the accessor know it is

safe to destroy its reusable cursors and operation contexts.

- Add IIndexOperationContext.destroy() letting the context

know that the user is done with it and allow it to release

resources

details:

- Previously, implementations of the IIndexCursor interface

didn't enforce the interface contract. This change enforces

the contract for all the implementations.

- With the enforcement of the contract, all the users of the

cursors are expected to follow and enforce the expected lifecycle.

- Test cases were added.

Change-Id: I98a7a8b931eb24dbe11bf2bdc61b754ca28ebdf9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2324

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 132 more files in changeset.
[ASTERIXDB-2270][STO] Fix Deadlock in LSMRTree

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- The LSMRTreeWithAntiMatterTuplesSearchCursor was

using the includeMutableComponent incorrectly to

determine whether memory components are included.

- This change ensures that memory components cursors

are always closed and destroyed as part of the

cursor lifecycle.

- The change also includes a check that fails if a

virtual file was deleted and some of its pages

are still latched.

Validate pages of deleted vbc files are not latched

Change-Id: If91ac3a2fa9265f929867e0dc75bbe0b7fa0aed1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2343

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 2 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -20
    • +23
    ./AsyncFIFOPageQueueManager.java
  1. … 979 more files in changeset.
[NO ISSUE][STO] Unpin pages when interrupted during reads

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- This change fixes a bug that happens when a thread pinning

a page that is not already in the buffer cache is interrupted.

- The fix is that if a failure happens during the read call,

the page is unpinned.

- A test case is added

Change-Id: I8d1c52fcf89ed90e8ef6019cd77842dd7468df49

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2274

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 389 more files in changeset.
[ASTERIXDB-1974][STO] Throw Exception on Interrupt

- user model changes: no

- storage format changes: no

- interface changes: yes

- Add HyracksDataException to IBufferCache

finishQueue.

Details:

- Re-interrupt thread and throw exception when

an interrupted thread is waiting for its pages

to be flushed in AsyncFIFOPageQueueManager.

Change-Id: I85197c1f334a5eb2d94d2a305817d280aa619209

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2133

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

[ASTERIXDB-1974][STO] Re-interrupt Thread on Interrupt in PageQueue

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Re-interrupt thread and throw exception when

a thread adding a page to the PageQueue is

interrupted.

Change-Id: Ia71e6623315af8eb1e9e8c58608e01ff4eb99b40

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2128

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[NO ISSUE][STO] Fix memory leaks in storage

- user model changes: no

- storage format changes: no

- interface changes: yes

- Added javadocs to:

-- IBufferCache

-- IExtraPageBlockHelper

- Moved IBufferCache.setPageDiskId -> ICachedPage.setDiskPageId

- Renamed:

-- IBufferCache.flushDirtyPage -> IBufferCache.flush

-- IBufferCache.getNumPages -> IBufferCache.getPageBudget

- Removed:

-- IBufferCache.adviseWontNeed [not used]

-- IBufferCache.tryPin [not used]

details:

- Previously, when adding a kv pair to the metadata of a memory

component, we add a new Pair item to the ArrayList. After

this change, we only update it if it exists.

- VirtualBufferCache used to leak pages when reclaiming pages

of a file after deletion. This has also been fixed.

- New tests for VirtualBufferCache added:

- Checks for memory budget after end of testDisjointPins

- Concurrent Users pinning pages concurrently

- Test for large pages and ensuring allocated large

pages are accounted for through removal of cached

free pages.

Change-Id: I4ae9736c9b5fdba5795245bdf835c023e3f73b15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2115

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 31 more files in changeset.
[NO ISSUE][HYR][*DB] Minor refactoring / address SonarQube comments

Change-Id: Icf10b6df0fdc006675d8f0da6fd06d50200c6b6a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2098

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 60 more files in changeset.
[ASTERIXDB-2046][STO] Retry Reading a Page on ClosedChannelException

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently when concurrent threads are reading from the same file

and one of them is interrupted, the file channel will be closed and

other threads will fail with AsynchronousCloseException if they were

reading or ClosedChannelException if they were about to read the file.

This change make the threads that were not interrupted retry the read

operation.

Change-Id: I4f9de9f51596314d17e4c6a4a58333e8fd6c03d1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2005

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.