Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2541][STO] Introduce GreedyScheduler

- user model changes: yes.

Add new option: storage.io.scheduler (async/greedy)

- storage format changes: no.

- interface changes: yes.

Introduce IIndexCursorStats

Details:

- Introduce GreedyScheduler that always executes the merge

operation with the smallest number of remaining pages to minimize

the number of disk components

- Introduce IIndexCursorStats to collect the statistics of index scans.

This allows GreedyScheduler to know the remaning pages of merge

operations.

- Extend AbstractIoOperation so that GreedyScheduler can pause/resume

merge operations if needed.

Change-Id: I38fe394d1180d4e3f6796064c0e6c6630b6ad303

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3284

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 70 more files in changeset.
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 142 more files in changeset.
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 60 more files in changeset.
[NO ISSUE][RT] Improve PreclusteredGroupWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Modified PreclusteredGroupWriter to only save group fields

from a last tuple in a frame instead of the whole frame

- move PermutingFrameTupleReference and PermutingTupleReference

from 'hyracks-storage-am-common' to 'hyracks-dataflow-common'

Change-Id: Ic75de2e6b64d0aacaf48096ecc9d47fc8e95c9cf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3351

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

  1. … 37 more files in changeset.
[ASTERIXDB-2422][STO] Introduce compressed storage

- user model changes: yes

- Add new configuration in the with-caluse to enable compression

- Add new nc configuration in the config file

- storage format changes: yes

- Pages of the primary index can be compressed

- Add a companion file (Look Aside File) with the compressed index

- Allow optional values in the LocalResource

- Add compression information in Metadata.Dataset

- interface changes: yes

- ICCApplicationContext:

- Add getCompressionManager()

- IBufferCache:

- Add getCompressedFileWriter(int fileId)

- ICachedPageInternal:

- Add setCompressedPageOffset(long offset)

- Add getCompressedPageOffset()

- Add setCompressedPageSize(int size)

- Add getCompressedPageSize()

Details:

- Add new integration test for this patch

- Fix ASTERIXDB-2464

- Add ddl-with-clause type validator

Additional details in the design document:

https://cwiki.apache.org/confluence/display/ASTERIXDB/Compression+in+AsterixDB

Change-Id: Idde6f37c810c30c7f1a5ee8bcbc1e3e5f4410031

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2857

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 191 more files in changeset.
[NO ISSUE][OTH] Cleanup Redundant Type Cast

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: I9c93ce90ad4aa789ae46284431e172ae4777bf55

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2887

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 29 more files in changeset.
[NO ISSUE] Report all BufferCache write failures.

- user model changes: no

- storage format changes: no

- interface changes: yes

+ IPageWriteFailureCallback: used to notify async

IO caller when something goes wrong.

Details:

- Before this change, it is possible for failures to

be lost and for bulkload operations to not be

aware of failure to write some pages. This can be

dangerous.

- To avoid this, when sending a page to be written

a PageWriteFailureCallback is associated with the

page to notify the caller that a failure took place.

Change-Id: I97fd3dccff85dab84d644359be6f66b15ee708ef

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2787

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 47 more files in changeset.
[ASTERIXDB-2344] LIMIT pushdown for primary index

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

-Implement LIMIT pushdown for primary index scan/lookup. In case of

a select operator, we also push the select condition to the priamry

search operator to ensure correctness.

Change-Id: I824fcad79995325e12a1a81d629160025294b915

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2541

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 52 more files in changeset.
[ASTERIXDB-1952][TX][IDX] Filter logs pt.2

- user model changes: no

- storage format changes: yes

- interface changes: yes

Details:

- Add a log type specifically for filters

- Only log change when filter actually widens

- Stop logging of index + filter tuple during modification

- Redo index and filter tuples separately via their logs

Change-Id: Ie9e7795d9c8c212e8610dcb9bb5d26ec9fbbee8a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1857

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 44 more files in changeset.
[ASTERIXDB-2083][COMP][RT][IDX][SITE] Budget-Constrained Inverted index search

- user-model changes: add text.searchmemory parameter

- storage format changes: no

- interface changes: IInvertedIndexSearcher, IInPlaceInvertedIndex,

IInvertedIndexAccessor, IInvertedListCursor

IObjectFactory, IPartitionedInvertedIndex,

IIndexAccessor

Details:

- Introduce text.searchmemory parameter in the configuration

to conduct budget-constrained inverted index search to prevent

a possible OOM exception

- Remove non-standard hyracks task context from the inverted-index-search

Change-Id: Ib2b2ef7c0b8c55ef66a5322be5d97ebbbf287bf5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2251

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 100 more files in changeset.
[NO ISSUE][RT] follow IFrameWriter protocol in SplitOperatorDescriptor

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Previously, the SplitOperatorDescriptor didn't follow the

IFrameWriter protocol in case of failure which lead to having

some open resources after the job.

- This caused so many failures in Cancellation tests.

- This change also increases the rate of cancellation during the

cancellation tests to ensure that similar problems are found.

Change-Id: I3166895589e1ab7355d689397f676f7da5c9809f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2399

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 13 more files in changeset.
[ASTERIXDB-1972][COMP][RT][TX] index-only plan

- user model changes: no

- storage format changes: no

- interface changes: IAccessMethod, ILSMIndexOperationContext,

IIndexAccessor

Details:

- Implement an index-only plan

- Add a SET option that disables the index-only plan

Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1866

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

  1. … 423 more files in changeset.
[ASTERIXDB-2204][STO] Fix implementations and usages of IIndexCursor

- user model changes: no

- storage format changes: no

- interface changes: yes

- IIndexCursor.close() is now idempotent and can be called on

a closed cursor.

- IIndexCursor.destroy() is now idempotent and can be called

on a destroyed cursor.

- Add IIndexAccessor.destroy() letting the accessor know it is

safe to destroy its reusable cursors and operation contexts.

- Add IIndexOperationContext.destroy() letting the context

know that the user is done with it and allow it to release

resources

details:

- Previously, implementations of the IIndexCursor interface

didn't enforce the interface contract. This change enforces

the contract for all the implementations.

- With the enforcement of the contract, all the users of the

cursors are expected to follow and enforce the expected lifecycle.

- Test cases were added.

Change-Id: I98a7a8b931eb24dbe11bf2bdc61b754ca28ebdf9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2324

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 125 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 983 more files in changeset.
[NO ISSUE][STO] Introduce ILSMIndexCursor

- user model changes: no

- storage format changes: no

- interface changes: yes

- introduce ILSMIndexCursor as a sub interface for IIndexCursor

- move getFilterMinTuple and getFilterMaxTuple to the

ILSMIndexCursor interface.

details:

- IIndexCursor.getFilterMinTuple and IIndexCursor.getFilterMaxTuple

don't make sense for any non LSM cursor and all the existing

implementations return null for those calls.

- In this change, we move those methods to a new interface for

LSM Index cursors.

Change-Id: I2fabe46e4c3a6370c48c2ccd9689fce507b37009

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2296

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 16 more files in changeset.
[ASTERIXDB-2184] Add Immutable DiskBTree

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add a immutable DiskBTree to for LSM disk components. This DiskBTree

only supports two operations, i.e., search and bulkload. No concurrency

control is performed at all, since it's immutable.

- Change LSMBTree/InvertedIndex to use this DiskBTree

- Add a DiskBTree point search cursor to optimize point lookups

Change-Id: I8f2a9281478c4b8665589dc695769d0497af9961

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2193

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +289
    ./java/org/apache/hyracks/storage/am/btree/impls/DiskBTree.java
  1. … 15 more files in changeset.
[ASTERIXDB-2204][STO] Fix the IIndexCursor interface

- user model changes: no

- storage format changes: no

- interface changes: yes

- replace IIndexCursor.reset with close

- replace IIndexCursor.close with destroy

Details:

- This change is the first step towards fixing the behavior

of implementors/callers of the IIndexCursor interface

- In this change, we simply rename the reset -> close

and close -> destroy and we write down the javadocs

explaining the semantics of the interface.

- LSM Index Cursors don't implements ITreeIndexCursor

anymore.

Change-Id: I64cf8c0a5473268bdfd71fd560ee6b3bff219ce9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2238

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 61 more files in changeset.
[NO ISSUE][STO] Adapt Storage Structure To Rebalance

- user model changes: no

- storage format changes: no

- interface changes: yes

-- Added IResource#setPath to use for the resource

storage migration.

Details:

- Unify storage structure to support dataset rebalance:

Old format:

./storage/partition_#/dataverse/datasetName_idx_indexName

New format:

./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName

- Adapt recovery and replication to new storage structure.

- Add old structure -> new structure NC migration task.

- Add CompatibilityUtil to ensure NC can be upgraded during

NC startup.

- Centralize the logic for parsing file path to its components in

ResourceReference/DatasetResourceReference.

- Add storage structure migration test case.

- Add test case for recovery after rebalance.

Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2181

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 40 more files in changeset.
[NO ISSUE][RT][IDX] Simplify index.createAccessor()

- user model changes: no

- storage format changes: no

- interface change: yes

(changed) IIndex, ILSMIndex

(new) IIndexAccessParameters

details:

- Refactor index.createAccessor() method to accept

an instance of IIndexAccessParameters as its parameter

since currently only ModificationCallBack and

SearchOperationCallback can be passed. If an accessor

needs to have additional parameters, there was no way

to pass them.

Change-Id: Iae015c342e830c81d666428447b595280139740e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2120

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 74 more files in changeset.
[NO ISSUE][STO] Fix memory leaks in storage

- user model changes: no

- storage format changes: no

- interface changes: yes

- Added javadocs to:

-- IBufferCache

-- IExtraPageBlockHelper

- Moved IBufferCache.setPageDiskId -> ICachedPage.setDiskPageId

- Renamed:

-- IBufferCache.flushDirtyPage -> IBufferCache.flush

-- IBufferCache.getNumPages -> IBufferCache.getPageBudget

- Removed:

-- IBufferCache.adviseWontNeed [not used]

-- IBufferCache.tryPin [not used]

details:

- Previously, when adding a kv pair to the metadata of a memory

component, we add a new Pair item to the ArrayList. After

this change, we only update it if it exists.

- VirtualBufferCache used to leak pages when reclaiming pages

of a file after deletion. This has also been fixed.

- New tests for VirtualBufferCache added:

- Checks for memory budget after end of testDisjointPins

- Concurrent Users pinning pages concurrently

- Test for large pages and ensuring allocated large

pages are accounted for through removal of cached

free pages.

Change-Id: I4ae9736c9b5fdba5795245bdf835c023e3f73b15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2115

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 39 more files in changeset.
[ASTERIXDB-2128] Fix bloomfilter during index search

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The current use of bloomfilter during search is completely wrong

(including primary index point lookups, secondary index search with

deleted btrees). We call BTree.search before bloomfilter check,

which renders bloomfilter completely useless. This patch fixes this

serious problem.

Change-Id: I90ab0d0b2da0028e0ec3cfa94d21881318293ff7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2069

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 10 more files in changeset.
[NO ISSUE][STO][IDX] LSM storage cleanup

- user model changes: no

- storage format changes: no

- interface changes: yes

- Replaced component lifecycle-related fabric methods in AbstractLSMIndex

with direct method calls of lsmComponent functions.

- Extracted common lifecycle-related functionality from index-specific

disk/memory lsmComponents to interfaces.

- Introduced composable disk component bulkloader design which assembles the

proper bulkload pipeline from individual elements populating lsmFilters,

bloomFilters, buddyBTrees\deletedKeysBTrees, bTress\rTrees\invIndexes.

- Changed methods to return index-specific versions of objects (accessors,

components, index instances) to avoid nasty downcasting.

Change-Id: I6739d751b990e7a28e03e32a5de6e2b670d37a1e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2014

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 93 more files in changeset.
[STO][IDX] Eliminated excess antimatter in LSMBTree

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

A combination of some LSM operations (e.g. insert+delete) inserts a

record into the memory component of LSMBTree and then deletes it right

after leaving an antimatter entry. When no flush happens between two

operations this "tombstone" entry does not have any purpose and could

be eliminated during the flush without changing search semantics.

The fix introduces a new bit in record header which tracks if an entry

was inserted and then updated in-place. For secondary indexes this will

happen only when the record is changed from regular to antimatter.

The patch does not introduce changes in storage format because the bit

exists only for memory components.

In addition the patch refactored *TupleWriters, *TupleWriterFactories,

*TupleReferences, *Frames, *FrameFactories to return

index-specific types.

Change-Id: I12a67eff8431b52d1f9051b793a5a64b15c009e9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1538

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 75 more files in changeset.
[ASTERIXDB-1952][TX][IDX]Log incoming filter vals

- user model changes: no

- interface changes: yes, for txn context

- storage format changes: yes, to log

details:

- Prior to this patch the filter values were not correct

on recovery. The tuple that was logged came from within

the wrapped indexand contained only the values to be stored.

In filtered scenarios this differs with what is fed to

the LSM wrapper to a respective index. redo plays the log

to the LSM wrapped index, so the input was simply not the

same on redo as it was during live ingestion. Three are other

ways to remedy this but the most straightforward is to simply

log what is given on input, and this is what this patch does.

- There is also a small fix for the way filters are accessed for

2ndary to primary search with an rtree index

Change-Id: I9268fe0b60145545c5933bab698d651c324397d7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1798

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 107 more files in changeset.
[ASTERIXDB-1945][STO] Cleanup Buffer Cache API

- user model changes: no

- storage format changes: no

- interface changes: yes

INcApplicationContext

- removed IFileMapProvider getFileMapManager();

to hide FileMapManager from other components;

IStorageManager

- IFileMapProvider getFileMapProvider(INCServiceContext ctx);

to hide FileMapManager from other components;

IFileHandle

- added FileReference getFileReference();

to avoid unnecessary casts;

IIOManager

- public void deleteWorkspaceFiles() throws

HyracksDataException;

added throws;

ILSMIndexFileManager

- void createDirs() throws HyracksDataException;

added throws;

IInvertedIndex

- added void purge() throws HyracksDataException;

a. InvertedIndexes don't implement the ITreeIndex interface.

b. when we deactivate a disk component, we need to purge it so

the buffer cache doesn't go through each page.

c. this need to be revisited, ASTERIXDB-1944

IFileMapManager

- int registerFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- FileReference unregisterFile(int fileId) throws

HyracksDataException;

return value added for future refernece of the file;

IBufferCache

- int createFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- void deleteFile(int fileId) throws HyracksDataException;

remove the dirty page flag since there's no dirty page;

- int openFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- added void deleteFile(FileReference file) throws

HyracksDataException;

we used to have this public methods in both BufferCache

and VirtualBufferCache. Now we lifted it into the interface.

AbstractLSMIndex

- removed protected abstract void

destroyMemoryComponent(ILSMMemoryComponent c)

throws HyracksDataException;

It is because turned out when we deactivate, we actually

destroy them. However, because of the not well defined API,

double destroy was okay and so we used to do double destroy.

Details:

This change fixes the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I0a973c2adb2e7fdcbbf18c7b888af3de5f0acc74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1843

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 159 more files in changeset.
Revert "ASTERIXDB-1945 [STO] Cleanup Buffer Cache API"

This reverts commit ae3daf6ef3397e583637360dc460c6391e03dc29.

Change-Id: I5e4e23f43a68e82c38fb8d1d7f4c0d01985c3a10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1842

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 159 more files in changeset.
ASTERIXDB-1945 [STO] Cleanup Buffer Cache API

Fix the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I15565b07afdc94ac74c608bfe4480fa09dcf8f1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1840

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 159 more files in changeset.
Fix upsert deadlock and upsert with filtered primary only

This change fixes a deadlock that happens when 3 operations

an upsert, a search and a flush happen simulteniously.

If all the memory components are full, the upsert

gets blocked, the upsert could've obtained a lock on the

search key which would block the search not allowing it

to exit the components and not allowing the components

to be cleared and reused.

In addition, the change refactors common LSM index code.

Change-Id: I93fac0f27ab0b3cc071ff38aef90d850cbbce488

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1762

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 84 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 557 more files in changeset.
Carry filter in 2ndary-to-primary index search

Change-Id: I287f1dbd230aa649f1350114abf0a1d47e2bb53c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1720

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 52 more files in changeset.