Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][STO] Misc fixes of inverted index

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

This patch mainly improves the inverted index merge perf

in the following ways:

- Fix the bloom filter check on deleted btree

- Remove unnecessary btree search during inverted index range search

- Remove unnecessary page pins when merging inverted lists by

making the cursor remembering the last pinned page

- Fix close/destroy operations of range search cursor

Change-Id: I7bbf6fe2d332db0e0368a4ce0f1ab218ba6da81d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2514

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 3 more files in changeset.
[NO ISSUE][STO] Fix opening of inverted index deleted keys cursor

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- LSMInvertedIndexDeletedKeysBTreeMergeCursor.open used to open

their inner cursors twice. This change fixes it so they are open

only once.

Change-Id: Ib325348265d8cc203aea6cd22591a49221f95411

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2510

Reviewed-by: Luo Chen <cluo8@uci.edu>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[ASTERIXDB-2321][STO] Follow the contract in IIndexCursor.open calls

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The index cursor contract says that an open call which returns

successfully, leaves the cursor in the open state, otherwise,

the cursor remains in the closed state.

- The LSM cursors have many cursors inside. In the

case where one of the cursors fails to open, and an exception

is about to be thrown, we must close all previously open cursors

since the LSM cursor will be in the closed state

and close will not be called.

Change-Id: I19db2afd2d6ca4a2ca1056cd95ae504b2be69813

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2501

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 10 more files in changeset.
[ASTERIXDB-1952][TX][IDX] Filter logs pt.2

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Add a log type specifically for filters

- Only log change when filter actually widens

- Stop logging of index + filter tuple during modification

- Redo index and filter tuples separately via their logs

Change-Id: Ie9e7795d9c8c212e8610dcb9bb5d26ec9fbbee8a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1857

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 40 more files in changeset.
[NO ISSUE][RT] no latching on an inverted list during merge

- user-model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not conduct a latching on an inverted list during

a disk component merge of an inverted index since

a disk component is immutable.

Change-Id: I266135596bbd5158d5fcf5ce3d1735d407be6f0f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2444

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

[NO ISSUE][RT] Ensure an inverted list cursor.close()

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Ensure to always execute an inverted list cursor.close().

Change-Id: I78c7908830be810b1d40abffffbd5f1978818869

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2410

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

[ASTERIXDB-2083][COMP][RT][IDX][SITE] Budget-Constrained Inverted index search

- user-model changes: add text.searchmemory parameter

- storage format changes: no

- interface changes: IInvertedIndexSearcher, IInPlaceInvertedIndex,

IInvertedIndexAccessor, IInvertedListCursor

IObjectFactory, IPartitionedInvertedIndex,

IIndexAccessor

Details:

- Introduce text.searchmemory parameter in the configuration

to conduct budget-constrained inverted index search to prevent

a possible OOM exception

- Remove non-standard hyracks task context from the inverted-index-search

Change-Id: Ib2b2ef7c0b8c55ef66a5322be5d97ebbbf287bf5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2251

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +103
    ./storage/am/lsm/invertedindex/api/InvertedListCursor.java
  1. … 86 more files in changeset.
[NO ISSUE][RT] follow IFrameWriter protocol in SplitOperatorDescriptor

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Previously, the SplitOperatorDescriptor didn't follow the

IFrameWriter protocol in case of failure which lead to having

some open resources after the job.

- This caused so many failures in Cancellation tests.

- This change also increases the rate of cancellation during the

cancellation tests to ensure that similar problems are found.

Change-Id: I3166895589e1ab7355d689397f676f7da5c9809f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2399

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 13 more files in changeset.
[ASTERIXDB-1972][COMP][RT][TX] index-only plan

- user model changes: no

- storage format changes: no

- interface changes: IAccessMethod, ILSMIndexOperationContext,

IIndexAccessor

Details:

- Implement an index-only plan

- Add a SET option that disables the index-only plan

Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1866

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

  1. … 424 more files in changeset.
[ASTERIXDB-2204][STO] Fix implementations and usages of IIndexCursor

- user model changes: no

- storage format changes: no

- interface changes: yes

- IIndexCursor.close() is now idempotent and can be called on

a closed cursor.

- IIndexCursor.destroy() is now idempotent and can be called

on a destroyed cursor.

- Add IIndexAccessor.destroy() letting the accessor know it is

safe to destroy its reusable cursors and operation contexts.

- Add IIndexOperationContext.destroy() letting the context

know that the user is done with it and allow it to release

resources

details:

- Previously, implementations of the IIndexCursor interface

didn't enforce the interface contract. This change enforces

the contract for all the implementations.

- With the enforcement of the contract, all the users of the

cursors are expected to follow and enforce the expected lifecycle.

- Test cases were added.

Change-Id: I98a7a8b931eb24dbe11bf2bdc61b754ca28ebdf9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2324

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 118 more files in changeset.
[ASTERIXDB-2231][STO] Separate primary op tracker for each partition

- user model changes: no

- storage format changes: no.

- interface changes: yes.

Details:

- Separate primary index operation tracker for each partition, instead

of having a global one on each NC to achieve better scalability.

- As a coordinated change, separate component id generator for each

partition as well.

- Add partition to transaction context so that transaction operations

can operate on proper op tracker.

- Fixes [ASTERIXDB-2232] to calculate dataset partitions correctly.

Change-Id: I9eb3854d2343e45beeccb87b0d434e5f4efd69c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2263

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 62 more files in changeset.
[ASTERIXDB-2250] Clean up files after failed flush/merge

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently we didn't clean up component files if the flush/merge

operation fails. As a result, when a failure happens, the subsequent

retries must fail as well because the files already exist.

- This patch cleans up component files when there is exception thrown

during flush/merge operation

- Added a test case on failed merge

Change-Id: I94630613cfe68de9d5784e022ca3834de959aa02

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2300

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 12 more files in changeset.
[ASTERIXDB-2256] Hand-prepare two problematic files for source format

Change-Id: Ibcb24e807979629f25892d58bff1c44c061b42e3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2334

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Murtadha Hubail <mhubail@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 975 more files in changeset.
[ASTERIXDB-2183][STO] Remove On-disk Inverted List Latch

Change-Id: Idcf0df5119824f0ef6761d514ccbd82af678a797

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2328

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

[NO ISSUE][STO] Introduce ILSMIndexCursor

- user model changes: no

- storage format changes: no

- interface changes: yes

- introduce ILSMIndexCursor as a sub interface for IIndexCursor

- move getFilterMinTuple and getFilterMaxTuple to the

ILSMIndexCursor interface.

details:

- IIndexCursor.getFilterMinTuple and IIndexCursor.getFilterMaxTuple

don't make sense for any non LSM cursor and all the existing

implementations return null for those calls.

- In this change, we move those methods to a new interface for

LSM Index cursors.

Change-Id: I2fabe46e4c3a6370c48c2ccd9689fce507b37009

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2296

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 18 more files in changeset.
[ASTERIXDB-2184] Add Immutable DiskBTree

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a immutable DiskBTree to for LSM disk components. This DiskBTree

only supports two operations, i.e., search and bulkload. No concurrency

control is performed at all, since it's immutable.

- Change LSMBTree/InvertedIndex to use this DiskBTree

- Add a DiskBTree point search cursor to optimize point lookups

Change-Id: I8f2a9281478c4b8665589dc695769d0497af9961

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2193

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 21 more files in changeset.
[ASTERIXDB-2204][STO] Fix the IIndexCursor interface

- user model changes: no

- storage format changes: no

- interface changes: yes

- replace IIndexCursor.reset with close

- replace IIndexCursor.close with destroy

Details:

- This change is the first step towards fixing the behavior

of implementors/callers of the IIndexCursor interface

- In this change, we simply rename the reset -> close

and close -> destroy and we write down the javadocs

explaining the semantics of the interface.

- LSM Index Cursors don't implements ITreeIndexCursor

anymore.

Change-Id: I64cf8c0a5473268bdfd71fd560ee6b3bff219ce9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2238

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 55 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 390 more files in changeset.
[ASTERIXDB-2169][STO][TX] Unblock modifications during full scan

- user model changes: no

- storage format changes: no

- interface changes: yes

- added ILSMHarness.replaceMemoryComponentsWithDiskComponents

details:

- During a long running query aka full scan, two things block

incoming modifications:

1) Memory component gets full, is flushed but can't be recycled

because of the search operation inside the component.

2) Read latches on the memory component not being released and

the memory component search cursor is not advancing.

The two cases are addressed in this change for the LSMBTree but

not yet addressed for other indexes.

The proposed solution for case (1) is to poll memory components

states every n records during the search operation. If a memory

component was found to have been flushed, its cursor is moved

to the corresponding disk component allowing the memory

component to be recycled.

The proposed solution for case (2) is to check memory component

cursor every n records. If the cursor has not advanced and the

component has writers, then the latches over the leaf page are

released, and the cursor re-do the operation entering from the

tree root.

- Added a test case.

- Added performance traces for enter and exit components.

Change-Id: I37ba52f6324ed1c5a78465c3a8cbcd351f1ed5bc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2166

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

  1. … 49 more files in changeset.
[ASTERIXDB-2115] Add Component Ids to LSM Indexes

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add LSMComponentId to all LSM components. Component Ids are managed

through IO operation callbacks.

- For memory component, it's ID is reset every time it's recycled.

- For disk component, it's ID is copied from the source component(s)

during flush/merge

- For indexes of a dataset, we need to guarantee all their memory

components should recieve the same ID. This is achieved using a shared

component Id generator.

- Fix memory component recycled callback, make sure it's called only

when we've indeed recycled the memory component

A design wiki for this patch: https://cwiki.apache.org/confluence/display/

ASTERIXDB/Component+Id-based+secondary-to-primary+index+acceleration

Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2025

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 62 more files in changeset.
Merge pre-rc changes to master

Change-Id: I578ffd7ef17784034b14f3c0d23cd5094e39f6e2

  1. … 21 more files in changeset.
[NO ISSUE][RT][IDX] Simplify index.createAccessor()

- user model changes: no

- storage format changes: no

- interface change: yes

(changed) IIndex, ILSMIndex

(new) IIndexAccessParameters

details:

- Refactor index.createAccessor() method to accept

an instance of IIndexAccessParameters as its parameter

since currently only ModificationCallBack and

SearchOperationCallback can be passed. If an accessor

needs to have additional parameters, there was no way

to pass them.

Change-Id: Iae015c342e830c81d666428447b595280139740e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2120

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 65 more files in changeset.
[NO ISSUE][STO] Add a callback on recycling of memory components

- user model changes: no

- storage format changes: no

- interface change: yes

- ILSMIOOperationCallbackFactory.createIoOpCallback now takes

the ILSMIndex as a parameter.

- Remove ILSMIOOperationCallback.setNumOfMutableComponents

The callback can find out the number of mutable components

on instantiation since the lsm index is now passed.

- ILSMIOOperationCallback.allocated was added.

It gets called whenever a memory component is allocated.

- ILSMIOOperationCallback.recycled was added.

It gets called whenever a memory component is recycled.

- ILSMIndex.hasMemoryComponent is replaced with

ILSMIndex.getNumberOfMemoryComponents

Change-Id: I578ffd7ef17784034c94f3c0d23cd5094e39f6e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2126

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 99 more files in changeset.
[NO ISSUE][STO] Fix memory leaks in storage

- user model changes: no

- storage format changes: no

- interface changes: yes

- Added javadocs to:

-- IBufferCache

-- IExtraPageBlockHelper

- Moved IBufferCache.setPageDiskId -> ICachedPage.setDiskPageId

- Renamed:

-- IBufferCache.flushDirtyPage -> IBufferCache.flush

-- IBufferCache.getNumPages -> IBufferCache.getPageBudget

- Removed:

-- IBufferCache.adviseWontNeed [not used]

-- IBufferCache.tryPin [not used]

details:

- Previously, when adding a kv pair to the metadata of a memory

component, we add a new Pair item to the ArrayList. After

this change, we only update it if it exists.

- VirtualBufferCache used to leak pages when reclaiming pages

of a file after deletion. This has also been fixed.

- New tests for VirtualBufferCache added:

- Checks for memory budget after end of testDisjointPins

- Concurrent Users pinning pages concurrently

- Test for large pages and ensuring allocated large

pages are accounted for through removal of cached

free pages.

Change-Id: I4ae9736c9b5fdba5795245bdf835c023e3f73b15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2115

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 39 more files in changeset.
[ASTERIXDB-2128] Fix bloomfilter during index search

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The current use of bloomfilter during search is completely wrong

(including primary index point lookups, secondary index search with

deleted btrees). We call BTree.search before bloomfilter check,

which renders bloomfilter completely useless. This patch fixes this

serious problem.

Change-Id: I90ab0d0b2da0028e0ec3cfa94d21881318293ff7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2069

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 9 more files in changeset.
[NO ISSUE][STO][IDX] LSM storage cleanup

- user model changes: no

- storage format changes: no

- interface changes: yes

- Replaced component lifecycle-related fabric methods in AbstractLSMIndex

with direct method calls of lsmComponent functions.

- Extracted common lifecycle-related functionality from index-specific

disk/memory lsmComponents to interfaces.

- Introduced composable disk component bulkloader design which assembles the

proper bulkload pipeline from individual elements populating lsmFilters,

bloomFilters, buddyBTrees\deletedKeysBTrees, bTress\rTrees\invIndexes.

- Changed methods to return index-specific versions of objects (accessors,

components, index instances) to avoid nasty downcasting.

Change-Id: I6739d751b990e7a28e03e32a5de6e2b670d37a1e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2014

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 79 more files in changeset.
Revert "[ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy"

This reverts commit 21ed0f72681a20ccb6a654f9aa4d54b8d0ea9c5c.

Change-Id: I670545acd09c678f21be25313353ab306be86202

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2063

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 51 more files in changeset.
[ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Currently CorrelatedMergePolicy uses component Ids to ensure disk

components of primary and secondary indexes are merged together,

but without synchronization. However, this results in too many disk

components for secondary InvertedIndex. The reason is that secondary

index could miss some round of merges, if the merge policy finds out

the corresponding secondary components are not available (either being

merged or being flushed). Even though flow-control on secondary indexes

can guarantee the secondary index would catch up the next time, it is

still possible that the primary component is finialized, which leaves

the secondary components which miss this round of merge are never merged

again.

This patch fixes this bug by:

- Add the mechanism of depending operations to LSM IO operation. An

operation finishes only after all depending operations have finished.

- For correlated merge policy, the flush/merge of the primary index depends

on all flushes/merges of secondary indexes. This ensures when the

correlated policy schedules merge, all related components of all indexes

are available to merge.

Change-Id: Ib6c06ee23f3bfd16b758802388389c00e29780b1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2018

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

  1. … 51 more files in changeset.
Introduce ITracer

Change-Id: I1d41d9cf74f481ba26882cf2ca318d0d2b9607f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2050

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 24 more files in changeset.