Clone Tools
  • last updated 25 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[ASTERIXDB-2540][STO] Optimize Storage Disk I/O

- user model changes: yes. Add a new storage option:

storage.disk.force.bytes (default 16MB),

- storage format changes: no.

- interface changes: yes.

Introduced IPageWriteCallback to LSM indexes

Details:

- Bypass all queuing (from BufferCache and IOManager) for disk writes.

This queuing is unnecessary but destroys fairness among multiple

writers.

- Introduce IPageWriteCallback to control the behavior of disk page

writes. Currently, this interface is used to perform disk forces

regularly for each writer thread.

Change-Id: I1f618dc7c186623e860239b4d97640fe3528e75b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3285

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -2
    • +4
    ./IndexBulkLoadOperatorNodePushable.java
  1. … 142 more files in changeset.
[ASTERIXDB-2532][RT] per-operator profiling

Enables profiling in queries at the operator-level when the analyze

variable is set in a query.

Change-Id: Ie16f3901ae5b32920d8552d5fd1ec8bb6e2ec8ae

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3226

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -6
    • +10
    ./IndexSearchOperatorNodePushable.java
  1. … 60 more files in changeset.
[NO ISSUE][RT] Add Thread-Based Stats Collector

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add infra to allow collecting thread-based stats during

runtime for any thread that belongs to a task.

- Collect number of pinned pages per thread and report it

in the TaskProfile.

- Aggregate pinned pages counters from all job tasks and

report it as diskIoCount in the metrics field in the json

response. The plan is to move this stats to the profile

field when it is introduced.

- Collecting pinned pages stats is currently enabled by

default for any job with IndexSearchOperatorNodePushable.

The plan is to allow enabling/disabling as part of the

job profiling change.

- Add test case for diskIoCount metric.

- Remove unused IndexSearchOperatorNodePushable constructor.

Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -10
    • +9
    ./IndexSearchOperatorNodePushable.java
  1. … 27 more files in changeset.
[NO ISSUE][RT] Improve PreclusteredGroupWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Modified PreclusteredGroupWriter to only save group fields

from a last tuple in a frame instead of the whole frame

- move PermutingFrameTupleReference and PermutingTupleReference

from 'hyracks-storage-am-common' to 'hyracks-dataflow-common'

Change-Id: Ic75de2e6b64d0aacaf48096ecc9d47fc8e95c9cf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3351

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

    • -1
    • +1
    ./IndexBulkLoadOperatorNodePushable.java
    • -1
    • +1
    ./IndexInsertUpdateDeleteOperatorNodePushable.java
    • -1
    • +1
    ./IndexSearchOperatorNodePushable.java
  1. … 35 more files in changeset.
Merge commit '153b2ce970' from 'stabilization-f69489'

Change-Id: I5b0d2756d839b0bc2e7c59723306921e698a7630

    • -3
    • +19
    ./IndexBulkLoadOperatorNodePushable.java
  1. … 6 more files in changeset.
[NO ISSUE][RT] Fix Secondary Indexes Bulkload/Upsert

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Prevent null values from being upserted into

secondary indexes.

- Use secondary index tuple filter when bulkloading

to prevent null/missing values from being inserted

into indexes of optional fields.

Change-Id: I7b847aece06a4387cc41389b0566d0c0dc98487d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3304

Reviewed-by: Luo Chen <cluo8@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +20
    ./IndexBulkLoadOperatorNodePushable.java
    • -1
    • +12
    ./TreeIndexBulkLoadOperatorDescriptor.java
  1. … 10 more files in changeset.
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./IndexBulkLoadOperatorNodePushable.java
  1. … 625 more files in changeset.
[ASTERIXDB-2344] LIMIT pushdown for primary index

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

-Implement LIMIT pushdown for primary index scan/lookup. In case of

a select operator, we also push the select condition to the priamry

search operator to ensure correctness.

Change-Id: I824fcad79995325e12a1a81d629160025294b915

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2541

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

    • -7
    • +99
    ./IndexSearchOperatorNodePushable.java
  1. … 53 more files in changeset.
[NO ISSUE] Incremental cleanup of deprecated exception ctors

Change-Id: I1e7c3655828fc6530cef83ea502a6cfbf41acddf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2533

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -2
    • +2
    ./IndexInsertUpdateDeleteOperatorNodePushable.java
    • -1
    • +1
    ./IndexSearchOperatorNodePushable.java
    • -1
    • +1
    ./TreeIndexStatsOperatorNodePushable.java
  1. … 238 more files in changeset.
[ASTERIXDB-2083][COMP][RT][IDX][SITE] Budget-Constrained Inverted index search

- user-model changes: add text.searchmemory parameter

- storage format changes: no

- interface changes: IInvertedIndexSearcher, IInPlaceInvertedIndex,

IInvertedIndexAccessor, IInvertedListCursor

IObjectFactory, IPartitionedInvertedIndex,

IIndexAccessor

Details:

- Introduce text.searchmemory parameter in the configuration

to conduct budget-constrained inverted index search to prevent

a possible OOM exception

- Remove non-standard hyracks task context from the inverted-index-search

Change-Id: Ib2b2ef7c0b8c55ef66a5322be5d97ebbbf287bf5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2251

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +5
    ./IndexSearchOperatorNodePushable.java
  1. … 100 more files in changeset.
[NO ISSUE][RT] follow IFrameWriter protocol in SplitOperatorDescriptor

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Previously, the SplitOperatorDescriptor didn't follow the

IFrameWriter protocol in case of failure which lead to having

some open resources after the job.

- This caused so many failures in Cancellation tests.

- This change also increases the rate of cancellation during the

cancellation tests to ensure that similar problems are found.

Change-Id: I3166895589e1ab7355d689397f676f7da5c9809f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2399

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +3
    ./IndexSearchOperatorNodePushable.java
    • -2
    • +2
    ./TreeIndexDiskOrderScanOperatorNodePushable.java
  1. … 12 more files in changeset.
[ASTERIXDB-1972][COMP][RT][TX] index-only plan

- user model changes: no

- storage format changes: no

- interface changes: IAccessMethod, ILSMIndexOperationContext,

IIndexAccessor

Details:

- Implement an index-only plan

- Add a SET option that disables the index-only plan

Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1866

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

    • -1
    • +43
    ./IndexSearchOperatorNodePushable.java
  1. … 424 more files in changeset.
[ASTERIXDB-2204][STO] Fix implementations and usages of IIndexCursor

- user model changes: no

- storage format changes: no

- interface changes: yes

- IIndexCursor.close() is now idempotent and can be called on

a closed cursor.

- IIndexCursor.destroy() is now idempotent and can be called

on a destroyed cursor.

- Add IIndexAccessor.destroy() letting the accessor know it is

safe to destroy its reusable cursors and operation contexts.

- Add IIndexOperationContext.destroy() letting the context

know that the user is done with it and allow it to release

resources

details:

- Previously, implementations of the IIndexCursor interface

didn't enforce the interface contract. This change enforces

the contract for all the implementations.

- With the enforcement of the contract, all the users of the

cursors are expected to follow and enforce the expected lifecycle.

- Test cases were added.

Change-Id: I98a7a8b931eb24dbe11bf2bdc61b754ca28ebdf9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2324

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -35
    • +26
    ./IndexSearchOperatorNodePushable.java
    • -31
    • +48
    ./TreeIndexDiskOrderScanOperatorNodePushable.java
  1. … 130 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -2
    • +2
    ./IndexSearchOperatorNodePushable.java
  1. … 982 more files in changeset.
[NO ISSUE][STO] Introduce ILSMIndexCursor

- user model changes: no

- storage format changes: no

- interface changes: yes

- introduce ILSMIndexCursor as a sub interface for IIndexCursor

- move getFilterMinTuple and getFilterMaxTuple to the

ILSMIndexCursor interface.

details:

- IIndexCursor.getFilterMinTuple and IIndexCursor.getFilterMaxTuple

don't make sense for any non LSM cursor and all the existing

implementations return null for those calls.

- In this change, we move those methods to a new interface for

LSM Index cursors.

Change-Id: I2fabe46e4c3a6370c48c2ccd9689fce507b37009

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2296

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +3
    ./IndexSearchOperatorNodePushable.java
  1. … 20 more files in changeset.
[NO ISSUE][OTR] Logging Cleanup

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- No pretty printing for replica status.

- Log ignored exception in debug level.

Change-Id: I7e4a37e9621ef5bdb27105b17e883800212e594b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2276

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 1 more file in changeset.
[ASTERIXDB-2204][STO] Fix the IIndexCursor interface

- user model changes: no

- storage format changes: no

- interface changes: yes

- replace IIndexCursor.reset with close

- replace IIndexCursor.close with destroy

Details:

- This change is the first step towards fixing the behavior

of implementors/callers of the IIndexCursor interface

- In this change, we simply rename the reset -> close

and close -> destroy and we write down the javadocs

explaining the semantics of the interface.

- LSM Index Cursors don't implements ITreeIndexCursor

anymore.

Change-Id: I64cf8c0a5473268bdfd71fd560ee6b3bff219ce9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2238

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -2
    • +2
    ./IndexSearchOperatorNodePushable.java
    • -1
    • +1
    ./TreeIndexDiskOrderScanOperatorNodePushable.java
  1. … 61 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -4
    • +5
    ./IndexSearchOperatorNodePushable.java
  1. … 388 more files in changeset.
[ASTERIXDB-2115] Add Component Ids to LSM Indexes

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add LSMComponentId to all LSM components. Component Ids are managed

through IO operation callbacks.

- For memory component, it's ID is reset every time it's recycled.

- For disk component, it's ID is copied from the source component(s)

during flush/merge

- For indexes of a dataset, we need to guarantee all their memory

components should recieve the same ID. This is achieved using a shared

component Id generator.

- Fix memory component recycled callback, make sure it's called only

when we've indeed recycled the memory component

A design wiki for this patch: https://cwiki.apache.org/confluence/display/

ASTERIXDB/Component+Id-based+secondary-to-primary+index+acceleration

Change-Id: I8aec6261a84a0729ce35f4b1cb708be299ddb98d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2025

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +5
    ./IndexBulkLoadOperatorNodePushable.java
  1. … 62 more files in changeset.
[NO ISSUE][RT][IDX] Simplify index.createAccessor()

- user model changes: no

- storage format changes: no

- interface change: yes

(changed) IIndex, ILSMIndex

(new) IIndexAccessParameters

details:

- Refactor index.createAccessor() method to accept

an instance of IIndexAccessParameters as its parameter

since currently only ModificationCallBack and

SearchOperationCallback can be passed. If an accessor

needs to have additional parameters, there was no way

to pass them.

Change-Id: Iae015c342e830c81d666428447b595280139740e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2120

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +4
    ./IndexInsertUpdateDeleteOperatorNodePushable.java
    • -1
    • +4
    ./IndexSearchOperatorNodePushable.java
    • -2
    • +4
    ./TreeIndexDiskOrderScanOperatorNodePushable.java
  1. … 72 more files in changeset.
[ASTERIXDB-1954][STO][RT] Add Index Drop Options

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add options to allow drop index to ignore

index does not exist and retry on index in-use.

- Add test case for new index drop options.

Change-Id: Id6f8fa52489bbe64d2f48c5c3d0a07be60f30b1b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2121

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

    • -9
    • +60
    ./IndexDropOperatorNodePushable.java
  1. … 9 more files in changeset.
[ASTERIXDB-2108][API][RT] Add Processed Objects Metric

- user model changes: no

- storage format changes: no

- interface changes: yes

Introduced IOperatorStats and IStatsCollector APIs to collect

runtime stats.

Details:

- Introduce OperatorStats API to report operators runtime stats.

- Introduce StatsCollector API to report task runtime stats.

- Implement OperatorStats for IndexSearchOperatorNodePushable

(tuple counter only).

- Add "processedObjects" metric to QueryService API.

- Add Stats to ExecuteStatementResponseMessage to pass stats

from CC to NCQueryService.

- Add metrics test cases.

Change-Id: Ie4afe6a676ef0b8a31d36d7dafc13a4023ebf177

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2032

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -3
    • +11
    ./IndexSearchOperatorNodePushable.java
  1. … 37 more files in changeset.
[ASTERIXDB-2025][STO] Fix Merge Lifecycle

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Complete merge operation after deletion of old components

files.

Change-Id: I843de8b26c181205e43f4eabe22a7c43f3ebfcbc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1930

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 7 more files in changeset.
[ASTERIXDB-2008][CLUS] Only add pending removal if node known

[ASTERIXDB-2023][ING] Introduce Enums instead of using bytes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Only nodes which are known to cluster manager are added

to the list of nodes pending removal. Other nodes are ignored

- Enums introduced:

- ActiveEvent.Kind

- ActivePartitionMessage.Event

- Remove AdapterRuntimeManager

- Remove AdapterExecutor

Change-Id: I7044896559798426c04a3f46861bc5335b25d140

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -2
    • +2
    ./IndexSearchOperatorNodePushable.java
  1. … 102 more files in changeset.
[ASTERIXDB-1943][API][STO] Make rebalance idempotent.

- user model changes:

added rebalance cancellation HTTP API.

- storage format changes: no

- interface changes: no

Details:

- add a HTTP API for cancelling a rebalance request;

- clean up leftover states at the beginning of a

rebalance request;

- add tests for rebalance cancellation.

Change-Id: I0d14a07978e106cd497cc35538fafef318b2fcf7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1821

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +11
    ./IndexDropOperatorNodePushable.java
  1. … 16 more files in changeset.
[ASTERIXDB-1945][STO] Cleanup Buffer Cache API

- user model changes: no

- storage format changes: no

- interface changes: yes

INcApplicationContext

- removed IFileMapProvider getFileMapManager();

to hide FileMapManager from other components;

IStorageManager

- IFileMapProvider getFileMapProvider(INCServiceContext ctx);

to hide FileMapManager from other components;

IFileHandle

- added FileReference getFileReference();

to avoid unnecessary casts;

IIOManager

- public void deleteWorkspaceFiles() throws

HyracksDataException;

added throws;

ILSMIndexFileManager

- void createDirs() throws HyracksDataException;

added throws;

IInvertedIndex

- added void purge() throws HyracksDataException;

a. InvertedIndexes don't implement the ITreeIndex interface.

b. when we deactivate a disk component, we need to purge it so

the buffer cache doesn't go through each page.

c. this need to be revisited, ASTERIXDB-1944

IFileMapManager

- int registerFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- FileReference unregisterFile(int fileId) throws

HyracksDataException;

return value added for future refernece of the file;

IBufferCache

- int createFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- void deleteFile(int fileId) throws HyracksDataException;

remove the dirty page flag since there's no dirty page;

- int openFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- added void deleteFile(FileReference file) throws

HyracksDataException;

we used to have this public methods in both BufferCache

and VirtualBufferCache. Now we lifted it into the interface.

AbstractLSMIndex

- removed protected abstract void

destroyMemoryComponent(ILSMMemoryComponent c)

throws HyracksDataException;

It is because turned out when we deactivate, we actually

destroy them. However, because of the not well defined API,

double destroy was okay and so we used to do double destroy.

Details:

This change fixes the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I0a973c2adb2e7fdcbbf18c7b888af3de5f0acc74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1843

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -5
    • +1
    ./TreeIndexStatsOperatorNodePushable.java
  1. … 161 more files in changeset.
Revert "ASTERIXDB-1945 [STO] Cleanup Buffer Cache API"

This reverts commit ae3daf6ef3397e583637360dc460c6391e03dc29.

Change-Id: I5e4e23f43a68e82c38fb8d1d7f4c0d01985c3a10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1842

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -1
    • +5
    ./TreeIndexStatsOperatorNodePushable.java
  1. … 161 more files in changeset.
ASTERIXDB-1945 [STO] Cleanup Buffer Cache API

Fix the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I15565b07afdc94ac74c608bfe4480fa09dcf8f1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1840

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -5
    • +1
    ./TreeIndexStatsOperatorNodePushable.java
  1. … 161 more files in changeset.
Support IFrameWriter contract check.

- add a instance-level flag for injecting operators to

check IFrameWriter contract violations;

- check contract violations in runtime tests.

Change-Id: I9827b06f640858f27ec1bcca2a39991780bee3b1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1618

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -6
    • +11
    ./IndexSearchOperatorNodePushable.java
  1. … 43 more files in changeset.
Change DataflowHelperFactory not to require Task Context

Change-Id: I9dcd95dbefca131c4bbdb43306f00f6f8ea60800

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1758

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./IndexBulkLoadOperatorNodePushable.java
    • -1
    • +1
    ./IndexInsertUpdateDeleteOperatorNodePushable.java
    • -1
    • +1
    ./IndexSearchOperatorNodePushable.java
    • -1
    • +1
    ./TreeIndexDiskOrderScanOperatorNodePushable.java
    • -1
    • +1
    ./TreeIndexStatsOperatorNodePushable.java
  1. … 12 more files in changeset.