Murtadha Makki Al Hubail

[ASTERIXDB-2738][STO] Create Mask File Before Merge Operations

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before starting a merge operation, create a mask file

(.mask_C_startSeq_endSeq) for the merged component to

indicate that this component isn't valid yet.

- On the merge operation successful completion, delete the

merged component mask file.

- In the case of any unexpected failure during the merge

operation, all files of the failed merged component will

be deleted on node startup/shutdown, including the mask

file.

- Halt on any IO opeartion failure.

- Add a test case that ensures only masked merged components

are deleted but not the original components that were

supposed to be merged.

Change-Id: I476dd3be5e75468e83044b3aaf0f6c2d8beadf1c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6425

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Luo Chen <cluo8@uci.edu>

[ASTERIXDB-2730][STO] Avoid Double Flushes in GVBC

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure GVBC only flushes a primary index if it has a modified

memory component and there is no pending flush request.

Change-Id: Ib4c3c632c43d83c5e60960c2cdcce54f1216b851

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6305

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

[ASTERIXDB-2730][STO] Safely Update Flush Pointer in GVBC

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure at least one primary index is registered before

updating the flush pointer in GlobalVirtualBufferCache.

Otherwise, reset the pointer to zero.

Change-Id: Ib0b3c9c7cfcf7b6a3c2f404f42d1d4802cf12b28

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6285

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

[NO ISSUE][RT] Collect Tuple Stats in External Scan

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Report number of procssed tuples in external scan operator.

- Add test case.

Change-Id: I5dda25f1fc53581dcc5663f2516e79b9b66fd0a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/6224

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[NO ISSUE][RT] Ensure Fail is Called on RunFileWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When OptimizedHybridHashJoin fails, ensure that fail is

called on any RunFileWriter that was initialized. This

will ensure that any open run files are closed.

Change-Id: I27fa54367045e90540ef571a4cf33723aca66c53

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5924

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][FAIL] Improve Index Creation on Large Field Error Message

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Improve index creation on large field error message.

Change-Id: I4383e9b384f57367d422e9603abf247647a38701

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5843

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[NO ISSUE][STO] Account For Pages With Multiple Pages in LAFWriter

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Always ensure that the LAFWriter entry pages are allocated

for the main page as well as any extra pages.

- Account for null pages during backload failures to avoid NPEs.

Change-Id: Ib9a7a04fe97e4031001925134f7bfc14a97a6078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5764

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[NO ISSUE][TEST] Remove Spidersilk Integration

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Remove Spidersilk test framework since it has only a simple

test and the framework isn't currently being utilized.

Change-Id: I78eda6280fc20df1a91e17682785b2654473712d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5683

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -35
    • +0
    /asterixdb/asterix-spidersilk/config/cc.conf
    • -22
    • +0
    /asterixdb/asterix-spidersilk/docker/Dockerfile
    • -87
    • +0
    /asterixdb/asterix-spidersilk/pom.xml
[NO ISSUE][COMP] Introduce Index Only Compiler Property

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Introduce "compiler.indexonly" compiler property to

enable/disable index-only plans.

- Replace usage of 'noindexonly' query property by

the new compiler property.

- Update docs and test cases.

Change-Id: I82d05bbfacbaca17eb8d8a2b6d825aa180e9ab02

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5567

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 19 more files in changeset.
[NO ISSUE][MTD] Restrict Metadata Catalog Update to Synonym Dataset

- Only allow Metadata.Synonym datasets to be added to

an existing cluster. This is done to detect the case

when another metadata dataset that is expected to be

there but is not found.

Change-Id: I142d2537816cc302ef9e02e7faf18af03cb72350

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5604

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

[NO ISSUE][OTH] Remove Classad Parser

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Remove classad parser and related artifacts.

Change-Id: I2dc0f21b0f3b1ed1738b051d3b5c5c5882cba309

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5603

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 101 more files in changeset.
[NO ISSUE][NET] Allow Data Receivers To Report Errors

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- When an error is encountered while reading a result,

report an error to the node sending the data to allow

it to abort the operation.

- Allow FullFrameChannelWriteInterface to report errors

even when some data is still pending to be sent.

- Add test case to ensure result senders are terminated.

Change-Id: Ie7fba6760edb498b88112a7a68b1d0b9f08022b5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5323

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][OTH] Extensible Servlet Requests

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Allow extentions to create their own Servlet Requests

in http servers.

Change-Id: I71eeaaaf4fb30a51059b9cf6399c19bcec6b40ec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5306

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][NET] Ensure Channel Was Not Recycled Before Adding Credit

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before attempting to add credit to an existing channel,

ensure the channel slot was not recycled and reused by a

new channel. Without this, some additional credit might

erroneously be added to the new channel.

Change-Id: I0a5ec70ad4c2568477fac9eb98c030fc3d8b1ce9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4623

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Contrib: Michael Blow <mblow@apache.org>

[NO ISSUE][RT] Fail Fast When Interrupted on Frame Write Attempt

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before writing a frame to the next operator, check the thread

interrupted status and fail fast if the thread is interrupted.

This ensures that queries that were aborted will fail as soon

as they attempt to write a frame.

Change-Id: If4f154f11305e2f8286e0707b5b3adec905633a4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4203

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Merge branch 'stabilization-f69489' into 'mad-hatter'

Change-Id: I07cab31b76128dfa5cea9b49620b1c074b385c81

[NO ISSUE][STO] Force Checkpoints/Index Metadata Files to Disk

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure the system checkpoint file is forced to disk.

- Ensure index metadata files and checkpoints are forced

to disk.

Change-Id: I27ab8ed39e23760b514640d77532a1dc24a97992

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3943

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][NET] Exclude Warn Logs When No Buffer Factory is Used

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not warn about recycled partial frames when no buffer

factory is set in FullFrameChannelReadInterface as those

are valid cases that use a single frame to perform the

complete operation.

Change-Id: I80251c529eb15ee22d01c320883418ea80272c2c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3824

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[NO ISSUE][TEST] Do Not Print regexjson Actual Test Result

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not print actual test result when comparing the result

using regexjson.

Change-Id: Ia1381aa20d0ed47eb11ab8474c1acad1bd2988fb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3806

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

[NO ISSUE][STO] Persist Bloom Filter Existence in Index Metadata

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To clearly identify whether an index has a bloom filter or

not for BTree indexes, persist this information in the index's

metadata stored on each NC.

- For backward compatibility, when reading an index's metadata that

was created before adding the hasBloomFilter field, default its

value based on whether or not the index is a primary key index.

- Remove unused special readObject from LSMBTreeLocalResource.

Change-Id: Icec570d490987de401c036790ee9567238a60301

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3804

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 6 more files in changeset.
[NO ISSUE][RT] Report Disk IO Per Operator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not collect per thread stats when runtime

profiling is disabled.

- Include per operator disk IO in task profile.

- Remove aggregated disk IO count from the query

response metrics.

- Adapt test cases.

Change-Id: Iced174ae5d4fa9ca13f597325cbb66ff80819e71

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3783

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

  1. … 4 more files in changeset.
[NO ISSUE][OTH] Eliminate Object Creation on Thread Stats Lookup

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Use the thread object for thread stats lookup

rather than the thread id to avoid boxed objects

creation on stats lookup.

Change-Id: I5b8b26be9d3439889e21cf097ea86c413e973e1d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3763

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][NET] Log ChannelReadInterface Stats on Failure

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- When the read buffers of a FullFrameChannelReadInterface

are exceeded, log its stats to help in debugging the issue.

- Warn when a partial frame is being written over a full

frame write channel.

- Rename ThreadSafetyGuaranteedBy -> GuardedBy.

- Annotate networking calls with their synchronization

guards.

Change-Id: I89eed0c06dbf4b0e86747538bff286dc37853957

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3663

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Al Hubail <mhubail@uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][TXN] Remove Synchronization on TransactionContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently the flag indicating if a transaction timed out

synchronizes on the transaction context which makes it

a hot spot when multiple threads on the same transactions

attempting to access that flag. This change removes that

synchronization and replaces it with a volatile variable.

Change-Id: Ib14c07a051d27a5b47c62ff35769670e6ab01fd2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3579

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][TEST] Increase Test Buffer Cache Memory Budget

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Increase test buffer cache memory budget to 128MB

to avoid test failures due to small buffer cache.

Change-Id: I76d29bf7d81de189e2157d902382a60e6c9772e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3571

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Merge commit '82b067c' from stabilization-f69489

Change-Id: I4891035d517387148e86027a733afa4a21a5c75a

Merge commit '8930f0d' from stabilization-f69489

Change-Id: I64cf862bc5ebe19a24acbccf821f68556699222e

  1. … 9 more files in changeset.
[NO ISSUE][RT] Add Thread-Based Stats Collector

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add infra to allow collecting thread-based stats during

runtime for any thread that belongs to a task.

- Collect number of pinned pages per thread and report it

in the TaskProfile.

- Aggregate pinned pages counters from all job tasks and

report it as diskIoCount in the metrics field in the json

response. The plan is to move this stats to the profile

field when it is introduced.

- Collecting pinned pages stats is currently enabled by

default for any job with IndexSearchOperatorNodePushable.

The plan is to allow enabling/disabling as part of the

job profiling change.

- Add test case for diskIoCount metric.

- Remove unused IndexSearchOperatorNodePushable constructor.

Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 13 more files in changeset.
[NO ISSUE][STO] Increase Dataset Checkpoint Interval

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The current dataset checkpoint interval of 10 mins

is too small and can result in many unnecessary small

flushes. This change increases the interval to 60 mins.

Change-Id: Idd4d1d02e4df83833220bca05193c6a99393ac99

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3546

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[NO ISSUE][CLUS] Attempt To Get Node IP From Existing Nodes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When we fail to resolve a node's host to an IP on node

failure/removal, attempt to find its IP from the list

of existing nodes. This is done to prevent failing the

node removal operation when the node's host address

cannot be resolved anymore.

Change-Id: I8f12a3dbb84a4cb731f1379efd65ca50606a8b07

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3544

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>