Clone
 

murtadha hubail <mhubail@uci.edu> in asterixdb

[ASTERIXDB-1961][IDX] Prevent NPE in cursor during cancellation

- user model changes: no

- interface changes: no

- storage format changes: no

Details:

- Prevent NPE in cursor when an interruption happens during

cursors initialization.

- Keep track of interruption stack trace to help in diagnosing

future similar issues.

Change-Id: I6937d14bc79d6583bb62c1d7b726ab0f26a59d79

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1856

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

ASTERIXDB-1928: Regression test

Change-Id: I3e0c8702e6cdf27c2caed6a77bc182ea3dbbc3fd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1822

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

ASTERIXDB-1841: Fix Inverted Index Drop Operator

Change-Id: I2c0f52770fd791ab9293c661de810c0b3b9ba8fb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1600

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 3 more files in changeset.
Exceptions Cleanup for Replication/FaultTolerance Strategies

Change-Id: I8f28b8db42bf7c8537ff2da22cbd2a97e243f32f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1529

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 4 more files in changeset.
Introduce Strategy Based Replication and Fault-Tolerance

This change includes the following:

- Introduce new APIs for Replication and Fault-Tolerance Strategies.

- Add configuration in cluster description file for high-availability.

- Add built-in replication strategies (Metadata_Only, Chained_Declustering)

- Add built-in fault-tolerance strategies (Auto, Metadata_Node)

- Remove none cluster state functionality from ClusterStateManager.

- Add customizable NC startup sequence

Change-Id: I1d1012f5541ce786f127866efefb9f3db434fedd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1405

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 97 more files in changeset.
Replace Java serialization in checkpoints by Json

Change-Id: Id4353716267d45ce21580ac2856b6a552d05d1f7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1457

Reviewed-by: Till Westmann <tillw@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Introduce CheckpointManager API

This change includes the following:

- s/CheckpointObject/Checkpoint

- Add AsterixDB storage version to checkpoints.

- Prevent any txn log access when a storage version mismatch is detected.

- Introduce CheckpointManager API and CheckpointProperties.

- Properly stop checkpointing thread on instance shutdown.

- Separate checkpointing logic when replication enabled/disabled.

Change-Id: I36c00ca195b93bbe1e53f39bb4a3b5a344657f0d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1380

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 6 more files in changeset.
Fix indefinite wait time for replication Job ACK

Change-Id: I88d2d61270522c766441e16fd996ac975935594b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1372

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Enable Replication Tests on Multi-NC Setup

- Move replication IT from vagrant VMs to multi-NC setup.

- Add 'node' command to kill specific node process.

- Fix file path passing to replication manager.

Change-Id: I542b212e04469e2701690f464f821a5189b97f12

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1364

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 35 more files in changeset.
Fix sporadic failures in checkpoint test

Change-Id: Ic32b9221ba3f843e006931556bab52d184ab1d33

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1317

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Add Checkpoint Test

This change adds a unit test case which validates that

checkpoints do not delete log files that are still required

for recovery, and delete those that are no longer needed.

Change-Id: I4cb4743fe488deb5ad10f65604adc2231948795e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1270

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

ASTERIXDB-1701: Fix log file boundry check

Change-Id: I538a8b931f3d50a00b092f218887a9731d14e235

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1306

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

ASTERIXDB-1693: Fix Default dataverse types referencing

Change-Id: Iae7ed8a2de345531345e4df2f9e06aa0d7680ebc

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1284

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Separate static cluster properties from runtime state

This change includes the following:

- Remove cluster static properties from AsterixClusterProperties.

- Add new ClusterProperties class for static cluster properties.

- Rename AsterixClusterProperties to ClusterStateManager.

- Refactor cluster static properties.

Change-Id: I3da72f639476f081ba8a0ef21ec176d32ff8bdec

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1183

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 24 more files in changeset.
ASTERIXDB-1238: Refactor AqlMetadataProvider

Change-Id: If2720817c5659622e1f713653856825d612eb892

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1016

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Introduce MessagingNetworkManager for NC2NC AppMessaging

This change introduces MessagingNetworkManager to NodeControllerService.

The MessagingNetworkManager is used to open channels that are used for NC2NC

application messaging. The read and write interfaces of the messaging channels

are set by a MessagingChannelInterfaceFactory which is set by the application.

Change-Id: I5c0bd7c11c1e78954ebceff49cb274d8073a64bd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/897

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 60 more files in changeset.
ASTERIXDB-1363: Fix NPE on bulkload failure

Change-Id: Ic1626baa63371834cc2abbe30366df506d817da6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1033

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

ASTERIXDB-1497: Prevent ConcurrentModification in DefaultDeallocatableRegistry

This change prevents the possible ConcurrentModificationException

in DefaultDeallocatableRegistry.

Change-Id: I1189d74ca33cbe8abf5b964eb8ff334df03c4004

Reviewed-on: https://asterix-gerrit.ics.uci.edu/965

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

ASTERIXDB-1498: Validate DatasetLifeCycleManager State

This change adds appropriate error message when trying

to access a dataset/index after the DatasetLifeCycleManager

has been stopped.

Change-Id: Ie2c4a506103e0c04ec6b203253144603704e7e5c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/964

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

ASTERIXDB-1472: Fix LSMRTreeWithAntiMatterTuples merge filter setting

Change-Id: I742a6a0577c58963e3f1418caad0bfb27fbe2915

Reviewed-on: https://asterix-gerrit.ics.uci.edu/910

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Txn Log Replication Optimizations

- Add configurations for txn log replication.

- Establish handshake for continuous log replication.

- Replicate logs in batches instead of one by one.

- Eliminate HashMap iterator object creation per logs batch.

- Fix synchronization for jobs' ACKs from remote replicas.

- Bug fixes for big objects replication.

- Some SONAR fixes for old code.

Change-Id: I25b5b84eba0cd41ac8e87e71368072879fcf8582

Reviewed-on: https://asterix-gerrit.ics.uci.edu/883

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

ASTERIXDB-1467: Allow PrivilegedAccessor in Deployment Mode

Change-Id: I15130c84584e316cde62e62c6010639e58d5050f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/898

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

ASTERIXDB-1419: Fix type checking for CollectionType

Change-Id: Ibf11d6c59ae00fe6d21fed8d75f199ee4ac9852c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/886

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

ASTERIXDB-1462: Remove previous LSN from txn log record

Change-Id: I32509ebe476e5993609f6fd73d2227a7d5219558

Reviewed-on: https://asterix-gerrit.ics.uci.edu/879

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Fix log file EOF during recovery

Change-Id: If9adbdaea726dd21d15fc1287362375daf30bf16

Reviewed-on: https://asterix-gerrit.ics.uci.edu/878

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

ASTERIXDB-1425 & ASTERIXDB-1450: Fix LogReader random reads

- Fix random reads for truncated logs (ASTERIXDB-1425).

- Fix log file partition size boundary check (ASTERIXDB-1450).

- Fix deadlock between LogReader and LogFlusher.

- Prevent checkpoints from deleting log files being accessed by rollbacks.

- Make rollbacks start from LSN = max(txnFirstLSN, minMemoryLSN).

- Make default log partition size 256MB instead of 2GB.

Change-Id: I1c75ca4a7c8fe197451126392389d4baecbd7e45

Reviewed-on: https://asterix-gerrit.ics.uci.edu/867

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Remove Log-Based Remote Recovery

This change removes the code that supports log-based remote recovery.

Remote recovery was replaced by the failback process which depends on

copying LSM disk component instead of logs.

Change-Id: I86e3b5832b52207e36c8409a072ccbda564d78b5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/871

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Make job recovery memory configurable

- Make memory allocated per job during recovery configurable.

- Exclude bloom filter file from LSMBTree with no bloom filter.

Change-Id: Ief359eae2352408c0cbfd37b0e3a1e758c78e0dd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/852

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Remove Unnecessary Flush on Index Drop

Change-Id: I2e41b5d0aee13cb3b2d3a23f064c35d28837da10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/804

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Fix Triggering Flush on Temporary Datasets

Change-Id: I964566e7ba03bc674209b4d45d9abb82bb581240

Reviewed-on: https://asterix-gerrit.ics.uci.edu/785

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>