Clone
 

murtadha hubail <mhubail@apache.org> in asterixdb

[NO ISSUE][RT] Fail Fast When Interrupted on Frame Write Attempt

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Before writing a frame to the next operator, check the thread

interrupted status and fail fast if the thread is interrupted.

This ensures that queries that were aborted will fail as soon

as they attempt to write a frame.

Change-Id: If4f154f11305e2f8286e0707b5b3adec905633a4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4203

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Merge branch 'stabilization-f69489' into 'mad-hatter'

Change-Id: I07cab31b76128dfa5cea9b49620b1c074b385c81

[NO ISSUE][STO] Force Checkpoints/Index Metadata Files to Disk

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure the system checkpoint file is forced to disk.

- Ensure index metadata files and checkpoints are forced

to disk.

Change-Id: I27ab8ed39e23760b514640d77532a1dc24a97992

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3943

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][NET] Exclude Warn Logs When No Buffer Factory is Used

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not warn about recycled partial frames when no buffer

factory is set in FullFrameChannelReadInterface as those

are valid cases that use a single frame to perform the

complete operation.

Change-Id: I80251c529eb15ee22d01c320883418ea80272c2c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3824

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

[NO ISSUE][TEST] Do Not Print regexjson Actual Test Result

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not print actual test result when comparing the result

using regexjson.

Change-Id: Ia1381aa20d0ed47eb11ab8474c1acad1bd2988fb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3806

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

[NO ISSUE][STO] Persist Bloom Filter Existence in Index Metadata

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To clearly identify whether an index has a bloom filter or

not for BTree indexes, persist this information in the index's

metadata stored on each NC.

- For backward compatibility, when reading an index's metadata that

was created before adding the hasBloomFilter field, default its

value based on whether or not the index is a primary key index.

- Remove unused special readObject from LSMBTreeLocalResource.

Change-Id: Icec570d490987de401c036790ee9567238a60301

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3804

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 6 more files in changeset.
[NO ISSUE][RT] Report Disk IO Per Operator

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not collect per thread stats when runtime

profiling is disabled.

- Include per operator disk IO in task profile.

- Remove aggregated disk IO count from the query

response metrics.

- Adapt test cases.

Change-Id: Iced174ae5d4fa9ca13f597325cbb66ff80819e71

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3783

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Murtadha Hubail <mhubail@apache.org>

  1. … 4 more files in changeset.
[NO ISSUE][OTH] Eliminate Object Creation on Thread Stats Lookup

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Use the thread object for thread stats lookup

rather than the thread id to avoid boxed objects

creation on stats lookup.

Change-Id: I5b8b26be9d3439889e21cf097ea86c413e973e1d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3763

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][NET] Log ChannelReadInterface Stats on Failure

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- When the read buffers of a FullFrameChannelReadInterface

are exceeded, log its stats to help in debugging the issue.

- Warn when a partial frame is being written over a full

frame write channel.

- Rename ThreadSafetyGuaranteedBy -> GuardedBy.

- Annotate networking calls with their synchronization

guards.

Change-Id: I89eed0c06dbf4b0e86747538bff286dc37853957

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/3663

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Al Hubail <mhubail@uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][TXN] Remove Synchronization on TransactionContext

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently the flag indicating if a transaction timed out

synchronizes on the transaction context which makes it

a hot spot when multiple threads on the same transactions

attempting to access that flag. This change removes that

synchronization and replaces it with a volatile variable.

Change-Id: Ib14c07a051d27a5b47c62ff35769670e6ab01fd2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3579

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][TEST] Increase Test Buffer Cache Memory Budget

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Increase test buffer cache memory budget to 128MB

to avoid test failures due to small buffer cache.

Change-Id: I76d29bf7d81de189e2157d902382a60e6c9772e0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3571

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

Merge commit '82b067c' from stabilization-f69489

Change-Id: I4891035d517387148e86027a733afa4a21a5c75a

Merge commit '8930f0d' from stabilization-f69489

Change-Id: I64cf862bc5ebe19a24acbccf821f68556699222e

  1. … 9 more files in changeset.
[NO ISSUE][RT] Add Thread-Based Stats Collector

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add infra to allow collecting thread-based stats during

runtime for any thread that belongs to a task.

- Collect number of pinned pages per thread and report it

in the TaskProfile.

- Aggregate pinned pages counters from all job tasks and

report it as diskIoCount in the metrics field in the json

response. The plan is to move this stats to the profile

field when it is introduced.

- Collecting pinned pages stats is currently enabled by

default for any job with IndexSearchOperatorNodePushable.

The plan is to allow enabling/disabling as part of the

job profiling change.

- Add test case for diskIoCount metric.

- Remove unused IndexSearchOperatorNodePushable constructor.

Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 13 more files in changeset.
[NO ISSUE][STO] Increase Dataset Checkpoint Interval

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The current dataset checkpoint interval of 10 mins

is too small and can result in many unnecessary small

flushes. This change increases the interval to 60 mins.

Change-Id: Idd4d1d02e4df83833220bca05193c6a99393ac99

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3546

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Hussain Towaileb <hussainht@gmail.com>

[NO ISSUE][CLUS] Attempt To Get Node IP From Existing Nodes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- When we fail to resolve a node's host to an IP on node

failure/removal, attempt to find its IP from the list

of existing nodes. This is done to prevent failing the

node removal operation when the node's host address

cannot be resolved anymore.

Change-Id: I8f12a3dbb84a4cb731f1379efd65ca50606a8b07

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3544

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][STO] Fix Time Unit Comparison For Periodic Flushes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- The dataset checkpoint interval is specified in the

configuration using seconds while the dataset last

flush time is tracked in nanos. This change fixes the

comparison by converting the dataset checkpoint

interval to nanos.

Change-Id: Id5a9b86207ea25ba44840af01324041525ac5bd9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3542

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][OTH] Make Active Recovery Task Extensible

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Allow active entity listeners to create their own

recovery tasks.

- Make lock acquisition in recovery task extensible.

Change-Id: I801eec74f7c1723e8243fe0f36db6148638bde35

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3538

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][NET] Ensure Recycling Buffer and Notifying Sender is Atomic

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To avoid synchronization issues that might occur

due to JVM reordering, ensure that both recycling

read buffers and notifying the sender of their

availability are done atomically before the next

buffer is received from the sender.

Change-Id: Ia3b1920f33bf7d4e7efbd2ea3405cbc4310a78c7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3520

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

(cherry picked from commit 32eed5f384c5851eae1c613fcb3b9532744ed595)

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3525

Reviewed-by: Michael Blow <mblow@apache.org>

[NO ISSUE][NET] Ensure Handshake Connections List is Thread-Safe

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure the list of connections that completed their asynchronous

SSL handshake is accessed in thread-safe manner.

Change-Id: Ibfd4fc569b59f39d730ef04f5d69b23d91713ed7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3523

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][NET] Ensure Recycling Buffer and Notifying Sender is Atomic

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- To avoid synchronization issues that might occur

due to JVM reordering, ensure that both recycling

read buffers and notifying the sender of their

availability are done atomically before the next

buffer is received from the sender.

Change-Id: Ia3b1920f33bf7d4e7efbd2ea3405cbc4310a78c7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3520

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Merge commit '8c99391' from stabilization-f69489

Change-Id: I139c265ba998c32ec049c8a8bbd7a5a213895d2a

[NO ISSUE][NET] Do Not Flush Partial Frames on Network Errors

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not flush partially read frames when a network error

is reported to avoid processing invalid partial frames

before the network failure is detected by the channel

reader.

Change-Id: Ia35b58412cc293426c18a7cd66d8d0a5962db4e2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3503

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

[NO ISSUE][OTH] Make Persisted Classes List Non-Static

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Make the registered classes list in PersistedResourceRegistry

non-static to prevent concurrent modification in environments

where multiple NCs share the same JVM.

- Use the proper NCApplication life cycle to configure the test

PersistedResourceRegistry rather than setting it after the

NCApplication is initialized.

Change-Id: Id89db7e89c90388ac7e5a1cee87be84e2f7d3c17

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3497

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

[NO ISSUE][STO] Tolerate Corrupted System Checkpoint Files

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Do not delete corrupted checkpoint files until a valid

checkpoint is persisted. This ensure a forged checkpoint

that will force recovery to start from the begining is

always used until a valid checkpoint is found.

- Attempt to read the latest checkpoint file right after

writing it and before attempting cleaning up invalid and

old checkpoint files.

- Use on disk files to determine next checkpoint id to

account for existing corrupted checkpoints.

- Maintain two older checkpoint files in addition to the

latest one.

- Catch all exceptions on checkpointing failures since it is

a try operation.

- Add test scenairo for the new checkpoints clean up behavior.

Change-Id: Iea689f5a644351491d9748273bb2158e8179f54d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3496

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Merge commit '0561d10' from stabilization-f69489

Change-Id: I18115329ce7ab3501e7fcc9c6dc06d0e87c97688

[NO ISSUE][REPL] Suspend Dataset Checkpointing on Replica Sync

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Before synchronizing replicas, stop datasets checkpointing to

prevent new files from being generated due to async IO operations

triggered by checkpointing.

- Instead of sync'ing current files to replicas then scheduling a flush

and sync'ing any newly generated files, just flush datasets before

the initial sync then sync all the files in one go.

Change-Id: I058fd48bc0fb89a1e16448ce516c3410bb4d681d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3469

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Merge commit 'fa6ce67' from stabilization-f69489

Change-Id: I01e1810f1c44d9ecaf569dd22577c33e8702c682

[ASTERIXDB-2598][RT] Add Support For Runtime Warnings

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Add the ability to add runtime warnings per task

and return the generated warnings as part of the

task profile on task completion.

- On successful job completion, aggregate warnings

from all task profiles of a job.

- Return the generated warnings in the query service

response as an array of "warnings" each with a code,

which is currently hard-coded to 1, and a message.

- Fix propagating source location to scalar aggregate

functions.

- Add a flag in test cases definition to indicate whether

or not to check for expected warnings and default it to

false.

- Generate warnings when min/max functions encounter

incomparable type or unsupported input.

- Add support in test framework to extract warnings

along with extracting the result field.

- Add support in test framework to validate generated

and expected warnings.

- Add test cases for min/max generated warnings.

Change-Id: I52fa5b807799487d62e67a8861068e1547aa629a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3451

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com>

  1. … 19 more files in changeset.
[NO ISSUE][REPL] Wait For Dataset IO Before Replica Sync

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Add API in DatasetLifecycleManager to wait for IO on datasets

matching a replication strategy.

- Before synchronizing the files on a replica, wait for any

on-going IO operations on replicated datasets to make sure

the set of files won't change while the replica is being

synchronized (e.g. a merge operation won't delete a file

that we wanted to send to a replica).

Change-Id: I01ed5c9379cf7ae249faeef624d5226ea699cf22

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3446

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>