Clone Tools
  • last updated 13 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE][STO] Component Deletes Through flushes and merges

- user model changes: no

- storage format changes: no

- interface changes: yes

- moved validation of component from the index:

- ILSMIndex and all of its implementations

to the component:

- ILSMDiskComponent and all of its implementations

details:

- This change enables component level deletes.

Change-Id: I178656207bfa1d15e6ae5ff2403a16df33940773

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2017

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +3
    ./common/ophelpers/IndexOperation.java
  1. … 78 more files in changeset.
[ASTERIXDB-2108][API][RT] Add Processed Objects Metric

- user model changes: no

- storage format changes: no

- interface changes: yes

Introduced IOperatorStats and IStatsCollector APIs to collect

runtime stats.

Details:

- Introduce OperatorStats API to report operators runtime stats.

- Introduce StatsCollector API to report task runtime stats.

- Implement OperatorStats for IndexSearchOperatorNodePushable

(tuple counter only).

- Add "processedObjects" metric to QueryService API.

- Add Stats to ExecuteStatementResponseMessage to pass stats

from CC to NCQueryService.

- Add metrics test cases.

Change-Id: Ie4afe6a676ef0b8a31d36d7dafc13a4023ebf177

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2032

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -3
    • +11
    ./common/dataflow/IndexSearchOperatorNodePushable.java
  1. … 37 more files in changeset.
[STO][IDX] Eliminated excess antimatter in LSMBTree

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

A combination of some LSM operations (e.g. insert+delete) inserts a

record into the memory component of LSMBTree and then deletes it right

after leaving an antimatter entry. When no flush happens between two

operations this "tombstone" entry does not have any purpose and could

be eliminated during the flush without changing search semantics.

The fix introduces a new bit in record header which tracks if an entry

was inserted and then updated in-place. For secondary indexes this will

happen only when the record is changed from regular to antimatter.

The patch does not introduce changes in storage format because the bit

exists only for memory components.

In addition the patch refactored *TupleWriters, *TupleWriterFactories,

*TupleReferences, *Frames, *FrameFactories to return

index-specific types.

Change-Id: I12a67eff8431b52d1f9051b793a5a64b15c009e9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1538

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +35
    ./common/api/IBTreeIndexTupleReference.java
    • -0
    • +1
    ./common/api/ITreeIndexFrameFactory.java
    • -0
    • +10
    ./common/api/ITreeIndexTupleWriter.java
    • -1
    • +2
    ./common/api/ITreeIndexTupleWriterFactory.java
    • -4
    • +4
    ./common/frames/TreeIndexNSMFrame.java
    • -1
    • +2
    ./common/tuples/SimpleTupleReference.java
    • -2
    • +3
    ./common/tuples/SimpleTupleWriter.java
    • -4
    • +4
    ./common/tuples/TypeAwareTupleWriter.java
    • -4
    • +2
    ./common/tuples/TypeAwareTupleWriterFactory.java
    • -0
    • +38
    ./common/util/BitOperationUtils.java
  1. … 79 more files in changeset.
[ASTERIXDB-2025][STO] Fix Merge Lifecycle

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Complete merge operation after deletion of old components

files.

Change-Id: I843de8b26c181205e43f4eabe22a7c43f3ebfcbc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1930

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -0
    • +5
    ./common/dataflow/IndexDataflowHelper.java
  1. … 7 more files in changeset.
[ASTERIXDB-2008][CLUS] Only add pending removal if node known

[ASTERIXDB-2023][ING] Introduce Enums instead of using bytes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Only nodes which are known to cluster manager are added

to the list of nodes pending removal. Other nodes are ignored

- Enums introduced:

- ActiveEvent.Kind

- ActivePartitionMessage.Event

- Remove AdapterRuntimeManager

- Remove AdapterExecutor

Change-Id: I7044896559798426c04a3f46861bc5335b25d140

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -2
    • +2
    ./common/dataflow/IndexSearchOperatorNodePushable.java
  1. … 101 more files in changeset.
[ASTERIXDB-1943][API][STO] Make rebalance idempotent.

- user model changes:

added rebalance cancellation HTTP API.

- storage format changes: no

- interface changes: no

Details:

- add a HTTP API for cancelling a rebalance request;

- clean up leftover states at the beginning of a

rebalance request;

- add tests for rebalance cancellation.

Change-Id: I0d14a07978e106cd497cc35538fafef318b2fcf7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1821

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +11
    ./common/dataflow/IndexDropOperatorNodePushable.java
  1. … 16 more files in changeset.
[ASTERIXDB-1945][STO] Cleanup Buffer Cache API

- user model changes: no

- storage format changes: no

- interface changes: yes

INcApplicationContext

- removed IFileMapProvider getFileMapManager();

to hide FileMapManager from other components;

IStorageManager

- IFileMapProvider getFileMapProvider(INCServiceContext ctx);

to hide FileMapManager from other components;

IFileHandle

- added FileReference getFileReference();

to avoid unnecessary casts;

IIOManager

- public void deleteWorkspaceFiles() throws

HyracksDataException;

added throws;

ILSMIndexFileManager

- void createDirs() throws HyracksDataException;

added throws;

IInvertedIndex

- added void purge() throws HyracksDataException;

a. InvertedIndexes don't implement the ITreeIndex interface.

b. when we deactivate a disk component, we need to purge it so

the buffer cache doesn't go through each page.

c. this need to be revisited, ASTERIXDB-1944

IFileMapManager

- int registerFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- FileReference unregisterFile(int fileId) throws

HyracksDataException;

return value added for future refernece of the file;

IBufferCache

- int createFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- void deleteFile(int fileId) throws HyracksDataException;

remove the dirty page flag since there's no dirty page;

- int openFile(FileReference fileRef) throws

HyracksDataException;

return value added for future reference of the index file

inside BufferCache or VirtualBufferCache;

- added void deleteFile(FileReference file) throws

HyracksDataException;

we used to have this public methods in both BufferCache

and VirtualBufferCache. Now we lifted it into the interface.

AbstractLSMIndex

- removed protected abstract void

destroyMemoryComponent(ILSMMemoryComponent c)

throws HyracksDataException;

It is because turned out when we deactivate, we actually

destroy them. However, because of the not well defined API,

double destroy was okay and so we used to do double destroy.

Details:

This change fixes the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I0a973c2adb2e7fdcbbf18c7b888af3de5f0acc74

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1843

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -5
    • +1
    ./common/dataflow/TreeIndexStatsOperatorNodePushable.java
    • -71
    • +19
    ./common/impls/AbstractTreeIndex.java
    • -8
    • +8
    ./common/util/TreeIndexBufferCacheWarmup.java
    • -11
    • +8
    ./common/util/TreeIndexStatsGatherer.java
  1. … 157 more files in changeset.
Revert "ASTERIXDB-1945 [STO] Cleanup Buffer Cache API"

This reverts commit ae3daf6ef3397e583637360dc460c6391e03dc29.

Change-Id: I5e4e23f43a68e82c38fb8d1d7f4c0d01985c3a10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1842

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -1
    • +5
    ./common/dataflow/TreeIndexStatsOperatorNodePushable.java
    • -19
    • +71
    ./common/impls/AbstractTreeIndex.java
    • -8
    • +8
    ./common/util/TreeIndexBufferCacheWarmup.java
    • -8
    • +11
    ./common/util/TreeIndexStatsGatherer.java
  1. … 157 more files in changeset.
ASTERIXDB-1945 [STO] Cleanup Buffer Cache API

Fix the buffer cache to follow the API such that:

1. createFile creates the file.

2. deleteFile deletes the file.

3. openFile opens the file.

4. closeFile closes the file.

5. creates existing file is not allowed.

6. deletes deleted file is not allowed.

7. open non existing file is not allowed.

In addition, we hide the file map from all other components.

Change-Id: I15565b07afdc94ac74c608bfe4480fa09dcf8f1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1840

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -5
    • +1
    ./common/dataflow/TreeIndexStatsOperatorNodePushable.java
    • -71
    • +19
    ./common/impls/AbstractTreeIndex.java
    • -8
    • +8
    ./common/util/TreeIndexBufferCacheWarmup.java
    • -11
    • +8
    ./common/util/TreeIndexStatsGatherer.java
  1. … 157 more files in changeset.
Support IFrameWriter contract check.

- add a instance-level flag for injecting operators to

check IFrameWriter contract violations;

- check contract violations in runtime tests.

Change-Id: I9827b06f640858f27ec1bcca2a39991780bee3b1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1618

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -6
    • +11
    ./common/dataflow/IndexSearchOperatorNodePushable.java
  1. … 43 more files in changeset.
Change DataflowHelperFactory not to require Task Context

Change-Id: I9dcd95dbefca131c4bbdb43306f00f6f8ea60800

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1758

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./common/dataflow/IndexBulkLoadOperatorNodePushable.java
    • -1
    • +1
    ./common/dataflow/IndexInsertUpdateDeleteOperatorNodePushable.java
    • -1
    • +1
    ./common/dataflow/IndexSearchOperatorNodePushable.java
    • -1
    • +1
    ./common/dataflow/TreeIndexDiskOrderScanOperatorNodePushable.java
    • -1
    • +1
    ./common/dataflow/TreeIndexStatsOperatorNodePushable.java
  1. … 12 more files in changeset.
Add Disk Component Scan operation for primary LSM index

-Added disk component scan operation for primary LSMBTree index,

which would be used by creating new secondary index

-This operation scans all disk components of the primary index,

and return all tuples. Thus, tuples with the same primary key

in different in components would be returned separately.

-The returned tuple has an extra int field, which indicates

which component this tuple comes from, and a boolean flag,

which indicates whether this tuple is an anti-matter tuple or not.

Change-Id: I31b2c67c58cb0a440c1d2c26400af322e2f1c1e5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1791

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -1
    • +2
    ./common/ophelpers/IndexOperation.java
  1. … 16 more files in changeset.
Fix upsert deadlock and upsert with filtered primary only

This change fixes a deadlock that happens when 3 operations

an upsert, a search and a flush happen simulteniously.

If all the memory components are full, the upsert

gets blocked, the upsert could've obtained a lock on the

search key which would block the search not allowing it

to exit the components and not allowing the components

to be cleared and reused.

In addition, the change refactors common LSM index code.

Change-Id: I93fac0f27ab0b3cc071ff38aef90d850cbbce488

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1762

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -21
    • +28
    ./common/impls/AbstractTreeIndex.java
    • -11
    • +1
    ./common/impls/TreeIndexDiskOrderScanCursor.java
    • -1
    • +1
    ./common/tuples/DualTupleReference.java
  1. … 83 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -37
    • +0
    ./common/api/ICursorInitialState.java
    • -105
    • +0
    ./common/api/IIndexAccessor.java
    • -0
    • +35
    ./common/api/IIndexBuilder.java
    • -0
    • +29
    ./common/api/IIndexBuilderFactory.java
    • -13
    • +26
    ./common/api/IIndexDataflowHelper.java
    • -31
    • +0
    ./common/api/IIndexLifecycleManagerProvider.java
    • -52
    • +0
    ./common/api/IModificationOperationCallback.java
    • -1
    • +2
    ./common/api/IModificationOperationCallbackFactory.java
    • -83
    • +0
    ./common/api/IResourceLifecycleManager.java
    • -79
    • +0
    ./common/api/ISearchOperationCallback.java
    • -0
    • +1
    ./common/api/ISearchOperationCallbackFactory.java
  1. … 557 more files in changeset.
Carry filter in 2ndary-to-primary index search

Change-Id: I287f1dbd230aa649f1350114abf0a1d47e2bb53c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1720

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +2
    ./common/api/IIndexDataflowHelper.java
    • -32
    • +0
    ./common/dataflow/ITreeIndexOperatorDescriptor.java
    • -15
    • +53
    ./common/dataflow/IndexSearchOperatorNodePushable.java
    • -0
    • +11
    ./common/impls/TreeIndexDiskOrderScanCursor.java
  1. … 51 more files in changeset.
Cleanup storage exceptions

Change-Id: I020d2b4b1f4ae48fc2df0b720e70a1ce95867d34

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1619

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -1
    • +1
    ./common/api/ITwoPCIndexBulkLoader.java
    • -42
    • +0
    ./common/api/TreeIndexException.java
    • -35
    • +0
    ./common/api/UnsortedInputException.java
    • -7
    • +1
    ./common/dataflow/IndexBulkLoadOperatorNodePushable.java
    • -7
    • +12
    ./common/dataflow/IndexInsertUpdateDeleteOperatorNodePushable.java
    • -35
    • +0
    ./common/exceptions/TreeIndexNonExistentKeyException.java
    • -19
    • +11
    ./common/impls/AbstractTreeIndex.java
  1. … 175 more files in changeset.
Fix transaction logs and optimize upserts

Previously, Transaction logs didn't log previous image

which made it difficult to undo aborted transactions

correctly. This change fixes that by always recording

previous image.

In addition, Upsert was performed as a delete if found

followed by an insert with two logs. This change makes

it a single operation with a single transaction log.

Change-Id: Ice5296267033cd7debe76894c864c6411f761d83

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1554

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -14
    • +2
    ./common/api/IModificationOperationCallback.java
    • -5
    • +0
    ./common/impls/NoOpOperationCallback.java
  1. … 61 more files in changeset.
Add a REST endpoint for query cancellation.

This change also includes the following parts:

- Fix failure handling and potentially thread leakage in

MaterializingPipelinedPartition;

- Fix failure handling in PartitionDataWriter;

- Add a new test suite: SqlppExecutionWithCancellationTest.

Change-Id: I2936ac83f71bbef533e2695ed0a2b220c23fc483

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +5
    ./common/dataflow/TreeIndexStatsOperatorNodePushable.java
  1. … 45 more files in changeset.
Cleanup and bug fixes in Feeds pipeline

The bug fix is for the MessagingFrameTupleAppender. It used

to consume one extra byte per frame.

Cleanups are for feeds. These include:

1. Remove the use of feed dataflow marker. Feeds which

need to send progress can and should do that without

a marker thread.

2. Lock the memory component for feed commit to be able

to add information to the memory component's metadata

page safely.

In addition, this change introduces a frame level callback

for index operations.

Change-Id: Ie97b2133ebecb7380cf0ba336e60ed714d06f8ee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1523

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -1
    • +1
    ./common/api/IModificationOperationCallbackFactory.java
  1. … 71 more files in changeset.
Improve reading from and writing to Metadata pages

This change introduces a new interface IComponentMetadata.

Each LSM component is associated with a metadata object

which can be used to read and write arbitrary data to

the metadata pages of components. When flushing a

component, data in its metadata component is automatically

flushed to the disk component. For merge operations,

the IO Callback is responsible for merging the components'

metadata pages.

Change-Id: Id95ef33c0a0bc1abb3fc3ecdea5611ee4acd6dfa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1476

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Integration-Tests: Ian Maxon <imaxon@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -7
    • +10
    ./common/freepage/AppendOnlyLinkedMetadataPageManager.java
  1. … 152 more files in changeset.
RAT Cleanup, Resulting LICENSE fixes

Also:

- removed unused minimal javascript

- relocated storage test datagen package to hyracks-test-support

- eliminated MiniDFSCluster writes outside of target

- clean up .gitignore

- add line comments to PigletParser (taken from SQLPP.jj grammar)

Change-Id: I029dee65ac6b1d7ab7bddba66257a4857f682caa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1468

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -103
    • +0
    ./common/datagen/DataGenThread.java
    • -105
    • +0
    ./common/datagen/DocumentStringFieldValueGenerator.java
    • -39
    • +0
    ./common/datagen/DoubleFieldValueGenerator.java
    • -25
    • +0
    ./common/datagen/IFieldValueGenerator.java
    • -39
    • +0
    ./common/datagen/IntegerFieldValueGenerator.java
    • -103
    • +0
    ./common/datagen/PersonNameFieldValueGenerator.java
    • -68
    • +0
    ./common/datagen/ProbabilityHelper.java
    • -45
    • +0
    ./common/datagen/SortedDoubleFieldValueGenerator.java
    • -45
    • +0
    ./common/datagen/SortedFloatFieldValueGenerator.java
    • -45
    • +0
    ./common/datagen/SortedIntegerFieldValueGenerator.java
    • -52
    • +0
    ./common/datagen/StringFieldValueGenerator.java
    • -84
    • +0
    ./common/datagen/TupleGenerator.java
  1. … 57 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +5
    ./common/api/IIndexLifecycleManagerProvider.java
    • -0
    • +4
    ./common/api/IMetadataPageManagerFactory.java
    • -1
    • +12
    ./common/api/IPrimitiveValueProvider.java
    • -1
    • +8
    ./common/api/IPrimitiveValueProviderFactory.java
    • -4
    • +4
    ./common/dataflow/AbstractIndexOperatorDescriptor.java
    • -5
    • +6
    ./common/dataflow/AbstractTreeIndexOperatorDescriptor.java
    • -2
    • +2
    ./common/dataflow/TreeIndexBulkLoadOperatorDescriptor.java
    • -2
    • +2
    ./common/dataflow/TreeIndexCreateOperatorDescriptor.java
    • -2
    • +2
    ./common/dataflow/TreeIndexDiskOrderScanOperatorDescriptor.java
    • -2
    • +2
    ./common/dataflow/TreeIndexInsertUpdateDeleteOperatorDescriptor.java
    • -2
    • +2
    ./common/dataflow/TreeIndexStatsOperatorDescriptor.java
    • -0
    • +5
    ./common/freepage/AppendOnlyLinkedMetadataPageManagerFactory.java
    • -0
    • +1
    ./common/ophelpers/IndexOperation.java
  1. … 413 more files in changeset.
Change the API for writing and reading metadata pages

Change-Id: Iadad522ab5568677aa816c74fc1d63acad505380

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1396

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -35
    • +18
    ./common/api/IMetadataPageManager.java
    • -85
    • +0
    ./common/api/ITreeIndexMetaDataFrame.java
    • -24
    • +0
    ./common/api/ITreeIndexMetaDataFrameFactory.java
    • -0
    • +165
    ./common/api/ITreeIndexMetadataFrame.java
    • -0
    • +25
    ./common/api/ITreeIndexMetadataFrameFactory.java
    • -3
    • +1
    ./common/api/ITreeIndexTupleReference.java
    • -2
    • +2
    ./common/dataflow/IndexDataflowHelper.java
    • -92
    • +189
    ./common/frames/LIFOMetaDataFrame.java
    • -59
    • +58
    ./common/frames/TreeIndexNSMFrame.java
    • -133
    • +70
    ./common/freepage/AppendOnlyLinkedMetadataPageManager.java
    • -141
    • +51
    ./common/freepage/LinkedMetaDataPageManager.java
    • -0
    • +53
    ./common/freepage/MutableArrayValueReference.java
  1. … 84 more files in changeset.
Remove Append Only Flag

Change-Id: Id5d6917db8ab29aa01521596f556006e25a502fe

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1385

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -179
    • +0
    ./common/api/IMetaDataPageManager.java
    • -25
    • +0
    ./common/api/IMetadataManagerFactory.java
    • -0
    • +73
    ./common/api/IMetadataPageManager.java
    • -0
    • +27
    ./common/api/IMetadataPageManagerFactory.java
    • -0
    • +161
    ./common/api/IPageManager.java
    • -0
    • +28
    ./common/api/IPageManagerFactory.java
    • -2
    • +6
    ./common/api/ITreeIndexMetaDataFrame.java
    • -25
    • +0
    ./common/api/IVirtualMetaDataPageManager.java
    • -1
    • +10
    ./common/dataflow/AbstractIndexOperatorDescriptor.java
    • -2
    • +5
    ./common/dataflow/AbstractTreeIndexOperatorDescriptor.java
    • -12
    • +15
    ./common/dataflow/IIndexOperatorDescriptor.java
    • -0
    • +3
    ./common/dataflow/IndexDataflowHelper.java
  1. … 199 more files in changeset.
ASTERIXDB-1711: rename asterix_transaction_log

- s/asterix_transaction_log/transaction_log/

- update storage version

- remove some dead code

Change-Id: Ieed464a1993a6bbff798e18e4c94245755a447eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1368

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -2
    • +2
    ./common/api/ITreeIndexMetaDataFrame.java
  1. … 2 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +1
    ./common/dataflow/IndexDataflowHelper.java
    • -1
    • +1
    ./common/dataflow/TreeIndexStatsOperatorNodePushable.java
  1. … 251 more files in changeset.
Cleanup FileSplit and FileReference

This change gives FileSplit and FileReference specific meaning to

avoid confusion of an absolute vs relative, local vs global, inside

an IO device vs outside IO devices.

In addition, it enables better abstraction of global partitions and

delegate the responsibility of choosing which partition goes to which

IO device to the IO Manager through the introduction of FileDeviceComputer

In details:

Previously, the LocalResource in Hyracks had partition (storage partition)

and there is no such thing in Hyracks. This scope leak is bad. In addition

The local resource had a name and a path. they were always the same and so

the name was removed.

The storage partition was instead moved to asterixdb implementation of the

serialized object in the local resource.

With all of these changes, the cluster controller (compiler) only needs to

know about partitions and relative paths. It doesn't need to worry about

heterogenous Node setups and different io device configurations. For File

assignment to IO devices, a new interface (IFileDeviceComputer) was

introduced which can be overriden by applications to have their own

strategy for distributing files among IO devices.

Change-Id: I4fac508bf9af5a3bed41a3cf4464d2cbfecf2f61

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1352

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -14
    • +3
    ./common/api/IIndexDataflowHelper.java
    • -2
    • +4
    ./common/api/IModificationOperationCallbackFactory.java
    • -1
    • +1
    ./common/dataflow/IndexBulkLoadOperatorNodePushable.java
    • -1
    • +2
    ./common/dataflow/IndexCreateOperatorNodePushable.java
    • -48
    • +33
    ./common/dataflow/IndexDataflowHelper.java
    • -5
    • +7
    ./common/dataflow/IndexInsertUpdateDeleteOperatorNodePushable.java
    • -2
    • +3
    ./common/dataflow/IndexSearchOperatorNodePushable.java
    • -1
    • +2
    ./common/dataflow/TreeIndexBulkLoadOperatorDescriptor.java
    • -1
    • +2
    ./common/dataflow/TreeIndexCreateOperatorDescriptor.java
    • -1
    • +2
    ./common/dataflow/TreeIndexDiskOrderScanOperatorDescriptor.java
    • -2
    • +4
    ./common/dataflow/TreeIndexDiskOrderScanOperatorNodePushable.java
  1. … 273 more files in changeset.
Fix for ASTERIXDB-1725

This is tricky, the best method I could think to fix this involved simply

putting the root page number in the metadata page rather than trying to

calculate it somehow going backwards from the end of the file.

I also fixed a bug discovered by Mike B where getLSNOffset was not

taking into consideration page header sizes.

Change-Id: I9f21e96ab045d331e4fc1c77b5c73b975e6260fa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1331

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -0
    • +4
    ./common/api/IMetaDataPageManager.java
    • -1
    • +5
    ./common/api/ITreeIndexMetaDataFrame.java
    • -1
    • +13
    ./common/frames/LIFOMetaDataFrame.java
    • -16
    • +6
    ./common/impls/AbstractTreeIndex.java
  1. … 6 more files in changeset.
Small Cleanup Towards Fixing LifeCycle Issues

Before this change, dataset lifecycle manager was providing a set

of functionalities that are loosly related to management of datasets

and indexes. However, it was not clear what the possible states of

a dataset or an index and what is the responsibility of each object.

This change takes the first step towards fixing this area. Indexes

of a dataset are now grouped together under a single lifecycle class

A resource aka dataset must be created outside the lifecycle manager

and registered with it before it can be assigned resources (memory)

and before it can be used by any operation. This is still not the

case.

Change-Id: I84005a33837725f41ae63297a3711215dccce1d8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1148

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -38
    • +0
    ./common/api/IIndexLifecycleManager.java
    • -1
    • +1
    ./common/api/IIndexLifecycleManagerProvider.java
    • -0
    • +83
    ./common/api/IResourceLifecycleManager.java
    • -5
    • +6
    ./common/dataflow/IndexDataflowHelper.java
  1. … 65 more files in changeset.
Fixes related to ASTERIXDB-1534

- Cleanup boolean conditions and exceptions from SonarQube comments

- Fix issue where filter page in on-disk LSM components

can be confused with root page on restart

Change-Id: If51e0cd183f9d5ed6edaebef4a0568a6c67062e3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1111

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -12
    • +10
    ./common/freepage/LinkedMetaDataPageManager.java
    • -23
    • +28
    ./common/impls/AbstractTreeIndex.java