Clone Tools
  • last updated 26 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2169][STO][TX] Unblock modifications during full scan

- user model changes: no

- storage format changes: no

- interface changes: yes

- added ILSMHarness.replaceMemoryComponentsWithDiskComponents

details:

- During a long running query aka full scan, two things block

incoming modifications:

1) Memory component gets full, is flushed but can't be recycled

because of the search operation inside the component.

2) Read latches on the memory component not being released and

the memory component search cursor is not advancing.

The two cases are addressed in this change for the LSMBTree but

not yet addressed for other indexes.

The proposed solution for case (1) is to poll memory components

states every n records during the search operation. If a memory

component was found to have been flushed, its cursor is moved

to the corresponding disk component allowing the memory

component to be recycled.

The proposed solution for case (2) is to check memory component

cursor every n records. If the cursor has not advanced and the

component has writers, then the latches over the leaf page are

released, and the cursor re-do the operation entering from the

tree root.

- Added a test case.

- Added performance traces for enter and exit components.

Change-Id: I37ba52f6324ed1c5a78465c3a8cbcd351f1ed5bc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2166

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

  1. … 49 more files in changeset.
[ASTERIXDB-2115] Add Component Ids to LSM Indexes

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add LSMComponentId to all LSM components. Component Ids are managed

through IO operation callbacks.

- For memory component, it's ID is reset every time it's recycled.

- For disk component, it's ID is copied from the source component(s)

during flush/merge

- For indexes of a dataset, we need to guarantee all their memory

components should recieve the same ID. This is achieved using a shared

component Id generator.

- Fix memory component recycled callback, make sure it's called only

when we've indeed recycled the memory component

A design wiki for this patch: https://cwiki.apache.org/confluence/display/

ASTERIXDB/Component+Id-based+secondary-to-primary+index+acceleration

Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2025

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 62 more files in changeset.
Merge pre-rc changes to master

Change-Id: I578ffd7ef17784034b14f3c0d23cd5094e39f6e2

  1. … 21 more files in changeset.
[NO ISSUE][RT][IDX] Simplify index.createAccessor()

- user model changes: no

- storage format changes: no

- interface change: yes

(changed) IIndex, ILSMIndex

(new) IIndexAccessParameters

details:

- Refactor index.createAccessor() method to accept

an instance of IIndexAccessParameters as its parameter

since currently only ModificationCallBack and

SearchOperationCallback can be passed. If an accessor

needs to have additional parameters, there was no way

to pass them.

Change-Id: Iae015c342e830c81d666428447b595280139740e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2120

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 65 more files in changeset.
[NO ISSUE][STO] Add a callback on recycling of memory components

- user model changes: no

- storage format changes: no

- interface change: yes

- ILSMIOOperationCallbackFactory.createIoOpCallback now takes

the ILSMIndex as a parameter.

- Remove ILSMIOOperationCallback.setNumOfMutableComponents

The callback can find out the number of mutable components

on instantiation since the lsm index is now passed.

- ILSMIOOperationCallback.allocated was added.

It gets called whenever a memory component is allocated.

- ILSMIOOperationCallback.recycled was added.

It gets called whenever a memory component is recycled.

- ILSMIndex.hasMemoryComponent is replaced with

ILSMIndex.getNumberOfMemoryComponents

Change-Id: I578ffd7ef17784034c94f3c0d23cd5094e39f6e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2126

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 99 more files in changeset.
[NO ISSUE][STO] Fix memory leaks in storage

- user model changes: no

- storage format changes: no

- interface changes: yes

- Added javadocs to:

-- IBufferCache

-- IExtraPageBlockHelper

- Moved IBufferCache.setPageDiskId -> ICachedPage.setDiskPageId

- Renamed:

-- IBufferCache.flushDirtyPage -> IBufferCache.flush

-- IBufferCache.getNumPages -> IBufferCache.getPageBudget

- Removed:

-- IBufferCache.adviseWontNeed [not used]

-- IBufferCache.tryPin [not used]

details:

- Previously, when adding a kv pair to the metadata of a memory

component, we add a new Pair item to the ArrayList. After

this change, we only update it if it exists.

- VirtualBufferCache used to leak pages when reclaiming pages

of a file after deletion. This has also been fixed.

- New tests for VirtualBufferCache added:

- Checks for memory budget after end of testDisjointPins

- Concurrent Users pinning pages concurrently

- Test for large pages and ensuring allocated large

pages are accounted for through removal of cached

free pages.

Change-Id: I4ae9736c9b5fdba5795245bdf835c023e3f73b15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2115

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 39 more files in changeset.
[ASTERIXDB-2128] Fix bloomfilter during index search

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The current use of bloomfilter during search is completely wrong

(including primary index point lookups, secondary index search with

deleted btrees). We call BTree.search before bloomfilter check,

which renders bloomfilter completely useless. This patch fixes this

serious problem.

Change-Id: I90ab0d0b2da0028e0ec3cfa94d21881318293ff7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2069

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 9 more files in changeset.
[NO ISSUE][STO][IDX] LSM storage cleanup

- user model changes: no

- storage format changes: no

- interface changes: yes

- Replaced component lifecycle-related fabric methods in AbstractLSMIndex

with direct method calls of lsmComponent functions.

- Extracted common lifecycle-related functionality from index-specific

disk/memory lsmComponents to interfaces.

- Introduced composable disk component bulkloader design which assembles the

proper bulkload pipeline from individual elements populating lsmFilters,

bloomFilters, buddyBTrees\deletedKeysBTrees, bTress\rTrees\invIndexes.

- Changed methods to return index-specific versions of objects (accessors,

components, index instances) to avoid nasty downcasting.

Change-Id: I6739d751b990e7a28e03e32a5de6e2b670d37a1e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2014

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 79 more files in changeset.
Revert "[ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy"

This reverts commit 21ed0f72681a20ccb6a654f9aa4d54b8d0ea9c5c.

Change-Id: I670545acd09c678f21be25313353ab306be86202

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2063

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 51 more files in changeset.
[ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

Currently CorrelatedMergePolicy uses component Ids to ensure disk

components of primary and secondary indexes are merged together,

but without synchronization. However, this results in too many disk

components for secondary InvertedIndex. The reason is that secondary

index could miss some round of merges, if the merge policy finds out

the corresponding secondary components are not available (either being

merged or being flushed). Even though flow-control on secondary indexes

can guarantee the secondary index would catch up the next time, it is

still possible that the primary component is finialized, which leaves

the secondary components which miss this round of merge are never merged

again.

This patch fixes this bug by:

- Add the mechanism of depending operations to LSM IO operation. An

operation finishes only after all depending operations have finished.

- For correlated merge policy, the flush/merge of the primary index depends

on all flushes/merges of secondary indexes. This ensures when the

correlated policy schedules merge, all related components of all indexes

are available to merge.

Change-Id: Ib6c06ee23f3bfd16b758802388389c00e29780b1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2018

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

  1. … 51 more files in changeset.
Introduce ITracer

Change-Id: I1d41d9cf74f481ba26882cf2ca318d0d2b9607f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2050

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 24 more files in changeset.
[NO ISSUE][STO] Component Deletes Through flushes and merges

- user model changes: no

- storage format changes: no

- interface changes: yes

- moved validation of component from the index:

- ILSMIndex and all of its implementations

to the component:

- ILSMDiskComponent and all of its implementations

details:

- This change enables component level deletes.

Change-Id: I178656207bfa1d15e6ae5ff2403a16df33940773

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2017

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 74 more files in changeset.
[NO ISSUE] Add support for tracing

- using a format that can can be visualized using chrome://tracing in

Chrome

Change-Id: Ifa83592870e7e38a84aa560e4edb7fa2ad2d5e74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2022

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 26 more files in changeset.
[STO][IDX] Eliminated excess antimatter in LSMBTree

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

A combination of some LSM operations (e.g. insert+delete) inserts a

record into the memory component of LSMBTree and then deletes it right

after leaving an antimatter entry. When no flush happens between two

operations this "tombstone" entry does not have any purpose and could

be eliminated during the flush without changing search semantics.

The fix introduces a new bit in record header which tracks if an entry

was inserted and then updated in-place. For secondary indexes this will

happen only when the record is changed from regular to antimatter.

The patch does not introduce changes in storage format because the bit

exists only for memory components.

In addition the patch refactored *TupleWriters, *TupleWriterFactories,

*TupleReferences, *Frames, *FrameFactories to return

index-specific types.

Change-Id: I12a67eff8431b52d1f9051b793a5a64b15c009e9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1538

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 86 more files in changeset.
[ASTERIXDB-2025][STO] Fix Merge Lifecycle

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Complete merge operation after deletion of old components

files.

Change-Id: I843de8b26c181205e43f4eabe22a7c43f3ebfcbc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1930

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2008][CLUS] Only add pending removal if node known

[ASTERIXDB-2023][ING] Introduce Enums instead of using bytes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Only nodes which are known to cluster manager are added

to the list of nodes pending removal. Other nodes are ignored

- Enums introduced:

- ActiveEvent.Kind

- ActivePartitionMessage.Event

- Remove AdapterRuntimeManager

- Remove AdapterExecutor

Change-Id: I7044896559798426c04a3f46861bc5335b25d140

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 101 more files in changeset.
[ASTERIXDB-1946][STO][IDX]Create RTree/InvertedIdx for Correlated Datasets

- user model changes: allow creating secondary rtree and inverted index

for datasets using correlated merge policy

- storage format changes: no

- interface changes: no

details:

- Implmeneted create secondary RTree and Inverted Index for datasets

using correlated merge policy.

- Add tests for RTree and Inverted Index creation job

Change-Id: I100fc0b86b8a6fa36a95d77806107bad0307544e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1845

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 167 more files in changeset.
[ASTERIXDB-1952][TX][IDX]Log incoming filter vals

- user model changes: no

- interface changes: yes, for txn context

- storage format changes: yes, to log

details:

- Prior to this patch the filter values were not correct

on recovery. The tuple that was logged came from within

the wrapped indexand contained only the values to be stored.

In filtered scenarios this differs with what is fed to

the LSM wrapper to a respective index. redo plays the log

to the LSM wrapped index, so the input was simply not the

same on redo as it was during live ingestion. Three are other

ways to remedy this but the most straightforward is to simply

log what is given on input, and this is what this patch does.

- There is also a small fix for the way filters are accessed for

2ndary to primary search with an rtree index

Change-Id: I9268fe0b60145545c5933bab698d651c324397d7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1798

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 103 more files in changeset.
[ASTERIXDB-1944][STO] Align LSM inverted index with other indexes

- user model changes: no

- storage format changes: no

- interface changes: yes

IInvertedIndexComponent was introduced to separate LSM inverted

index from its component and avoid multiple MethodNotSupported

exceptions

Change-Id: I0fb7b446edfb14de25ecaf99965ae0a7325101c9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1852

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +53
    ./storage/am/lsm/invertedindex/api/IInPlaceInvertedIndex.java
[ASTERIXDB-1953][STO] Cleanup LSM File Managers

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- use FileReference instead of String absolute path

- user error codes

Change-Id: I97bab76888790ca282ad9508ce8416f7c7a52fb7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1849

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 15 more files in changeset.
[ASTERIXDB-1945][STO] Cleanup Buffer Cache API

- user model changes: no

- storage format changes: no

- interface changes: yes

INcApplicationContext

- removed IFileMapProvider getFileMapManager();

to hide FileMapManager from other components;

IStorageManager

- IFileMapProvider getFileMapProvider(INCServiceContext ctx);

to hide FileMapManager from other components;

IFileHandle

- added FileReference getFileReference();

to avoid unnecessary casts;

IIOManager

- public void deleteWorkspaceFiles() throws

HyracksDataException;

added throws;

ILSMIndexFileManager

- void createDirs() throws HyracksDataException;

added throws;

IInvertedIndex

- added void purge() throws HyracksDataException;

a. InvertedIndexes don't implement the ITreeIndex interface.

b. when we deactivate a disk component, we need to purge it so

the buffer cache doesn't go through each page.

c. this need to be revisited, ASTERIXDB-1944

IFileMapManager

- int registerFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- FileReference unregisterFile(int fileId) throws

HyracksDataException;

return value added for future refernece of the file;

IBufferCache

- int createFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- void deleteFile(int fileId) throws HyracksDataException;

remove the dirty page flag since there's no dirty page;

- int openFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- added void deleteFile(FileReference file) throws

HyracksDataException;

we used to have this public methods in both BufferCache

and VirtualBufferCache. Now we lifted it into the interface.

AbstractLSMIndex

- removed protected abstract void

destroyMemoryComponent(ILSMMemoryComponent c)

throws HyracksDataException;

It is because turned out when we deactivate, we actually

destroy them. However, because of the not well defined API,

double destroy was okay and so we used to do double destroy.

Details:

This change fixes the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I0a973c2adb2e7fdcbbf18c7b888af3de5f0acc74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1843

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 151 more files in changeset.
[ASTERIXDB-1946][STO][IDX] Create BTreeIndex for Correlated Datasets

Implemented create seconary BTree index for datasets using

correlated merge policy. Instead of creating one component

for the new index, this change creates one component for each

component of the primary index to maintain the correlation.

The current implementation assumes when a secondary index is being

created, the dataset is locked with no modifications.

Change-Id: I2a3435e6720f07bd6a5092d4d9ce42e8d4b7894c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1813

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 120 more files in changeset.
Revert "ASTERIXDB-1945 [STO] Cleanup Buffer Cache API"

This reverts commit ae3daf6ef3397e583637360dc460c6391e03dc29.

Change-Id: I5e4e23f43a68e82c38fb8d1d7f4c0d01985c3a10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1842

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 151 more files in changeset.
ASTERIXDB-1945 [STO] Cleanup Buffer Cache API

Fix the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I15565b07afdc94ac74c608bfe4480fa09dcf8f1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1840

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 151 more files in changeset.
ASTERIXDB-1938: Keep empty disk components when merge/flush

During flush/merge, it is possible that the new disk component

is empty. However, the current LSMDiskComponentBulkLoader would

clean up empty disk component. This patch fix this behavior by

introducing an extra parameter cleanupEmptyComponent.

Change-Id: Ic4f7ec41f56a2f9124920d67657f88160634f0e7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1826

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 14 more files in changeset.
Add Disk Component Scan operation for primary LSM index

-Added disk component scan operation for primary LSMBTree index,

which would be used by creating new secondary index

-This operation scans all disk components of the primary index,

and return all tuples. Thus, tuples with the same primary key

in different in components would be returned separately.

-The returned tuple has an extra int field, which indicates

which component this tuple comes from, and a boolean flag,

which indicates whether this tuple is an anti-matter tuple or not.

Change-Id: I31b2c67c58cb0a440c1d2c26400af322e2f1c1e5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1791

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 16 more files in changeset.
Allow insert anti-matter tuples when bulk loading LSM index

Previously, when we bulk load an LSM index, we are not allowed to insert

anti-matter tuples to the disk component. However, creating secondary

index for correlated datasets requires anti-matter tuples to be inserted

as well. Thus, this patch mainly contains the following changes:

- When bulk loading LSM index, allow the user to switch between insert

mode and delete mode

- Extended the LSMDiskComponentBulkLoader with the delete method. For

LSM index with anti-matter tuples, the delete method simply sets the

TupleWriter to delete mode, and inserts the anti-matter tuple. For LSM

index with buddy btree, it simply inserts the deleted tuple into the

buddy btree.

- Since the LSMDiskComponentBulkLoader would have a delete method

anyway, added a new ILSMDiskComponentBulkLoader interface containing the

delete method.

Change-Id: I6665f56a5d2183697197298fa24824eeb827686a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1796

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 20 more files in changeset.
Add LSMDiskComponentBulkLoader

-Added LSMDiskComponentBulkLoader implementations, which are used to bulk

load an LSMDiskComponent with anti-matters

-Added LSMDiskComponentWithBuddyBTreeBulkLoader implementations,

which are used to bulk load an LSMDiskComponent with deleted-keys btrees

-Refactored LSM flush/merge/index bulk load operations to use

the LSMDiskComponentBulkLoader

Change-Id: I772a6d68761fcbb85982a1c9f72f2d186e1d1ffb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1773

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 14 more files in changeset.
Rename Component Filter Factory

Change-Id: I1b3d9bef3fb6118c66195630cec23a7e41fc5861

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1779

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 14 more files in changeset.
Unexpose enter and exit components

Change-Id: Ib4ef7b432bbe6ac9cf2bbe9244cfe2b406f4cb93

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1778

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 6 more files in changeset.