Clone Tools
  • last updated 11 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Exclude Temporary Resources From Replication

- Exclude temporary resources from replication.

- Remove flush logs from temporary datasets.

- Ignore takeover partitions request if NC is shutting down.

- Stop NCs on different threads to allow replica shutting down

notification to be sent when replication is enabled.

Change-Id: I9a52557bf1f3e7632dd826384280abdaa186f672

Reviewed-on: https://asterix-gerrit.ics.uci.edu/778

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 6 more files in changeset.
Allow Replication to be Enabled on Virtual Cluster

- Allow replication port assignment per NC.

- Allow replication to be enabled on virtual cluster.

- Wait for JOB_ABORT ACK from remote replicas.

- Fix LSM component mask file name.

- Fix index directory deletion on index drop.

- Eliminate multiple partition takeover requests.

- Free LogFlusher thread from sending replication ACKs.

- Fix possible deadlock between LogFlusher and Logs Replication Thread.

- Remove wait for FLUSH_LOG for replicated LSM components:

This wait is not needed since on node failure, complete remote recovery is done.

Change-Id: I34a38f59c4915a19242adb6a4eaa6ee1c82d2372

Reviewed-on: https://asterix-gerrit.ics.uci.edu/743

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 19 more files in changeset.
Fix Asterix Installer Test Cases

Many tests started to fail in installer after being

disabled for a long time. This change fixes the following:

1. asterix-installer: external library tests.

2. asterix-installer: recovery: query-after-restart.

Change-Id: Iefad5ac3ff754693025312f1e2fa01b08832514c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/737

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 34 more files in changeset.
ASTERIXDB-1337: Dataset Memory Management on Multi-Partition NC

As sugggested in ASTERIXDB-1337, this change is to maintain a per-

partition MultitenantVirtualBufferCache budget, as opposed to sharing

the budget across the dataset.

Change-Id: Ibbf08f532c1210c30be6a51c73570a789174213b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/705

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 10 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 315 more files in changeset.
Asterix NCs Failback Support

- Allow Failed NCs to failback and takeover their partitions.

- New cluster API servlet for cluster state description.

- Remove nodeId from txn logs except remote FLUSH_LOG.

- Add partition id in UPDATE and Entity_COMMIT logs.

- Adapt remote recovery to new logs format.

- Refactor RecoveryManager and split Analysis and Redo phase.

- Spill remote recover logs to temporary file.

- Replicate files to remote replicas for partitions of interest only.

- Introduce NC active/inactive partitions concept.

- Test case for failback.

Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24

Reviewed-on: https://asterix-gerrit.ics.uci.edu/613

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 78 more files in changeset.
Add Support for Upsert Operation

This change allows users to execute upsert commands which

couples delete if found with insert. It locks the primary

keys before doing the search ensuring consistency.

Change-Id: I8999000331795a5949d621d2dd003903e057a521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 238 more files in changeset.
remove end-of-line whitespace

Change-Id: I5c0415f47d4c3a9827574fbdab949b45718d9ea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/601

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 124 more files in changeset.
Asterix NCs Fault Tolerance

This change includes the following:

- Adapt replication to unique partitions storage.

- Implement auto failover for failing NCs.

- Implement auto failover for metadata node.

- Fix for ASTERIXDB-1251 using proper error message.

- Basic replication test cases using vagrant virtual cluster for:

1. LSM bulkload components replication.

2. LSM Memory components replication and recovery.

3. Metadata node takeover.

These test cases will be part of the cluster test profile.

Change-Id: Ice26d980912a315fcb3efdd571d6ce88717cfea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/573

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 69 more files in changeset.
Divide Cluster into Unique Partitions

The change includes the following:

- Fix passing NC stores to AsterixConfiguration.

- Unify storage direcotry name in the instance level rather than the node level.

- Divide the cluster into unique storage partitions based on the number of stores.

- Refactored FileSplits and moved out of AqlMetadataProvider.

- Make AsterixHyracksIntegrationUtil use the passed configuration file.

- Make File Splits pass relative index paths of partitions rather than absolute paths.

- Remove unused AqlCompiledMetadataDeclarations class.

Change-Id: I8c7fbca5113dd7ad569a46dfa2591addb5bf8655

Reviewed-on: https://asterix-gerrit.ics.uci.edu/564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 35 more files in changeset.
Adapt Replication to Append Only LSM Components

This change includes the following:

- Make LSN recording in a single tree in LSM components.

- Pass LSN byte offset with every tree index file being replicated.

- Add LSN to remote recovery logs to check which logs should be replayed.

- Update ILogRecord method names to better describe their operation.

Change-Id: I8bd2656746e1c293b981d5f43e80928314ccbad0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/561

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 15 more files in changeset.
Asterix MessageBroker implementation

This change includes the following:

- Add implementation for CC/NC MessageBroker.

- Implement GlobalResourceIdFactory using MessageBroker.

- Change resource id factory to GlobalResourceIdFactory.

- Refactor metadata indexes fixed properties.

- Use fixed resource ids for metadata indexes.

Change-Id: If4320e2c5a0130d2f86a4be6ae61f5cee43e30af

Reviewed-on: https://asterix-gerrit.ics.uci.edu/486

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 24 more files in changeset.
ASTERIXDB-1110 Fix count keeping in primary index opTracker

This change fixes the counter of number of active operations in primary

index opTracker. This is accomplished by sharing of counter between multiple

partitions. Each transaction will have that counter as part of its

context object

Change-Id: I0dc34b9a3aa69d39ac2eda11c17e7ad0ccc1a661

Reviewed-on: https://asterix-gerrit.ics.uci.edu/536

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 100 more files in changeset.
ASTERIXDB-1118: allow for lock conversion

Also improve debugability of ConcurrentLockManager and add new unit tests.

Change-Id: If49ed8d48fa8c71a52c880d4f42a2badbe6a57d7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/474

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
ASTERIXDB-1152: Delete storage data of old instances

Change-Id: Ibb6c6949bdf2ed6c3e491fa66a23491ff34fc830

Reviewed-on: https://asterix-gerrit.ics.uci.edu/469

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 6 more files in changeset.
Introducing Data Replication To AsterixDB

This change includes the following:

- Add data replication properties to cluster properties and Managix validate command.

- Introduce Data Replication components.

- Add data replication required fields to LogRecord.

- Specialized LogManager for data replication.

- Fix for invalid cluster state on nodes failure.

- ASTERIXDB-139: Fix for cleaning workspace files on startup/shutdown.

- Fix for temp datasets storage reclamation.

- Allow MetadataNode rebinding with CC.

- Add flag to checkpoint to identify sharp checkpoints.

- ASTERIXDB-1170: Fix shutdown sequence

Change-Id: I729fdd1144dbc9ff039b4bc414494860d7553810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/338

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 64 more files in changeset.
ASTERIXDB-1058: use ResourceHeapBufferAllocator for dataset memory components.

Change-Id: Ifd90fabc79e61f84370d415c38917b998db41466

Reviewed-on: https://asterix-gerrit.ics.uci.edu/481

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 5 more files in changeset.
ASTERIXDB-1058: make Asterix compatible with lazy LSM memory allocation

- Adapt memory budget calculation to lazy LSM memory allocation.

- Add IDatasetLifecycleManager interface.

Change-Id: I4ea1eb129fe3043d43b077473dc29d17a97dfcc2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/408

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 18 more files in changeset.
ASTERIXDB-1160: Break deadlock between LogManager and LogFlusher

This change introduces a new thread in LogManager to log FLUSH logs.

This will prevent LogFlusher thread from waiting on itself until FLUSH logs are flushed to disk.

Change-Id: I0b414f5ab92e1c68c8aafbaab859c644ba399dcb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/475

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 1 more file in changeset.
ASTERIXDB-969: Redesigned recovery analysis phase to spill to disk

Change-Id: Ide2b346c2ad498d7595e71bae890362c2143d301

Reviewed-on: https://asterix-gerrit.ics.uci.edu/458

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 3 more files in changeset.
ASTERIXDB-1045: fix log file reading during recovery

Change-Id: Ic83ee1dd2d7ba88180c25f4ec6c7aa8d0a5a7162

Reviewed-on: https://asterix-gerrit.ics.uci.edu/465

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Updated LSMOperationType references to LSM API package

Change-Id: Ie3fca99e7019424f2b2a3ca43180cf6390feb761

Reviewed-on: https://asterix-gerrit.ics.uci.edu/448

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 7 more files in changeset.
ASTERIXDB-1045: Log analysis fixes

-Avoid using exceptions for control flow in LogRecord

-Rename LogPage and ilk to LogBuffer

-Busywait on read() to fill entire buffer for fillLogBuffer rather than failing

-Distinguish between log truncation and checksum corruption

TODOs:

- Log IO and parsing still happen in lock-step.

- Busywaiting for read to return something other than 0 is unfortunate

Change-Id: I1658e938eb0f199f748407361ffee4833aac661c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/289

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 4 more files in changeset.
ASTERIXDB-1053: Allow lazy loading for persistent local resources

- Allow indexes to be accessed by name.

- Allow lazy loading for persistent local resources.

- Caching for local resources.

Change-Id: I48b9260a3280750145f6ddb3783673a299055910

Reviewed-on: https://asterix-gerrit.ics.uci.edu/344

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 8 more files in changeset.
Change License Headers

Also tweak the NOTICE file with some extras.

Change-Id:I09bc388089e515d7f51fd39c31bfbbc9f00cf84f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/388

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2004 more files in changeset.
Change Java package from edu.uci.ics to org.apache

Change-Id: I2f01d2b5614e9e9c94fda4bf1294a8eba6a26c54

Reviewed-on: https://asterix-gerrit.ics.uci.edu/309

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2590 more files in changeset.
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: Iefd2a576415ebc1416cba2a3334d2b64f042ba92

Reviewed-on: https://asterix-gerrit.ics.uci.edu/306

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 3781 more files in changeset.
Fix for issue 923

Change-Id: I87053315fc7650682fcbedd573b1155c17810073

Reviewed-on: https://asterix-gerrit.ics.uci.edu/347

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Add Apache RAT License Auditor plugin, and fix missing licenses

Change-Id: Ic0fa0b7836e8a69d0d8beed83f9c3b41c168afd4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/263

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1530 more files in changeset.