Clone Tools
  • last updated 24 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Adapt Replication to Append Only LSM Components

This change includes the following:

- Make LSN recording in a single tree in LSM components.

- Pass LSN byte offset with every tree index file being replicated.

- Add LSN to remote recovery logs to check which logs should be replayed.

- Update ILogRecord method names to better describe their operation.

Change-Id: I8bd2656746e1c293b981d5f43e80928314ccbad0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/561

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 15 more files in changeset.
Asterix MessageBroker implementation

This change includes the following:

- Add implementation for CC/NC MessageBroker.

- Implement GlobalResourceIdFactory using MessageBroker.

- Change resource id factory to GlobalResourceIdFactory.

- Refactor metadata indexes fixed properties.

- Use fixed resource ids for metadata indexes.

Change-Id: If4320e2c5a0130d2f86a4be6ae61f5cee43e30af

Reviewed-on: https://asterix-gerrit.ics.uci.edu/486

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 24 more files in changeset.
ASTERIXDB-1110 Fix count keeping in primary index opTracker

This change fixes the counter of number of active operations in primary

index opTracker. This is accomplished by sharing of counter between multiple

partitions. Each transaction will have that counter as part of its

context object

Change-Id: I0dc34b9a3aa69d39ac2eda11c17e7ad0ccc1a661

Reviewed-on: https://asterix-gerrit.ics.uci.edu/536

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 100 more files in changeset.
ASTERIXDB-1118: allow for lock conversion

Also improve debugability of ConcurrentLockManager and add new unit tests.

Change-Id: If49ed8d48fa8c71a52c880d4f42a2badbe6a57d7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/474

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 4 more files in changeset.
ASTERIXDB-1152: Delete storage data of old instances

Change-Id: Ibb6c6949bdf2ed6c3e491fa66a23491ff34fc830

Reviewed-on: https://asterix-gerrit.ics.uci.edu/469

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 6 more files in changeset.
Introducing Data Replication To AsterixDB

This change includes the following:

- Add data replication properties to cluster properties and Managix validate command.

- Introduce Data Replication components.

- Add data replication required fields to LogRecord.

- Specialized LogManager for data replication.

- Fix for invalid cluster state on nodes failure.

- ASTERIXDB-139: Fix for cleaning workspace files on startup/shutdown.

- Fix for temp datasets storage reclamation.

- Allow MetadataNode rebinding with CC.

- Add flag to checkpoint to identify sharp checkpoints.

- ASTERIXDB-1170: Fix shutdown sequence

Change-Id: I729fdd1144dbc9ff039b4bc414494860d7553810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/338

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 64 more files in changeset.
ASTERIXDB-1058: use ResourceHeapBufferAllocator for dataset memory components.

Change-Id: Ifd90fabc79e61f84370d415c38917b998db41466

Reviewed-on: https://asterix-gerrit.ics.uci.edu/481

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 5 more files in changeset.
ASTERIXDB-1058: make Asterix compatible with lazy LSM memory allocation

- Adapt memory budget calculation to lazy LSM memory allocation.

- Add IDatasetLifecycleManager interface.

Change-Id: I4ea1eb129fe3043d43b077473dc29d17a97dfcc2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/408

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 18 more files in changeset.
ASTERIXDB-1160: Break deadlock between LogManager and LogFlusher

This change introduces a new thread in LogManager to log FLUSH logs.

This will prevent LogFlusher thread from waiting on itself until FLUSH logs are flushed to disk.

Change-Id: I0b414f5ab92e1c68c8aafbaab859c644ba399dcb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/475

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 1 more file in changeset.
ASTERIXDB-969: Redesigned recovery analysis phase to spill to disk

Change-Id: Ide2b346c2ad498d7595e71bae890362c2143d301

Reviewed-on: https://asterix-gerrit.ics.uci.edu/458

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 3 more files in changeset.
ASTERIXDB-1045: fix log file reading during recovery

Change-Id: Ic83ee1dd2d7ba88180c25f4ec6c7aa8d0a5a7162

Reviewed-on: https://asterix-gerrit.ics.uci.edu/465

Reviewed-by: Ian Maxon <imaxon@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Updated LSMOperationType references to LSM API package

Change-Id: Ie3fca99e7019424f2b2a3ca43180cf6390feb761

Reviewed-on: https://asterix-gerrit.ics.uci.edu/448

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 7 more files in changeset.
ASTERIXDB-1045: Log analysis fixes

-Avoid using exceptions for control flow in LogRecord

-Rename LogPage and ilk to LogBuffer

-Busywait on read() to fill entire buffer for fillLogBuffer rather than failing

-Distinguish between log truncation and checksum corruption

TODOs:

- Log IO and parsing still happen in lock-step.

- Busywaiting for read to return something other than 0 is unfortunate

Change-Id: I1658e938eb0f199f748407361ffee4833aac661c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/289

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 4 more files in changeset.
ASTERIXDB-1053: Allow lazy loading for persistent local resources

- Allow indexes to be accessed by name.

- Allow lazy loading for persistent local resources.

- Caching for local resources.

Change-Id: I48b9260a3280750145f6ddb3783673a299055910

Reviewed-on: https://asterix-gerrit.ics.uci.edu/344

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 8 more files in changeset.
Change License Headers

Also tweak the NOTICE file with some extras.

Change-Id:I09bc388089e515d7f51fd39c31bfbbc9f00cf84f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/388

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2004 more files in changeset.
Change Java package from edu.uci.ics to org.apache

Change-Id: I2f01d2b5614e9e9c94fda4bf1294a8eba6a26c54

Reviewed-on: https://asterix-gerrit.ics.uci.edu/309

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2590 more files in changeset.
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: Iefd2a576415ebc1416cba2a3334d2b64f042ba92

Reviewed-on: https://asterix-gerrit.ics.uci.edu/306

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 3781 more files in changeset.
Fix for issue 923

Change-Id: I87053315fc7650682fcbedd573b1155c17810073

Reviewed-on: https://asterix-gerrit.ics.uci.edu/347

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Add Apache RAT License Auditor plugin, and fix missing licenses

Change-Id: Ic0fa0b7836e8a69d0d8beed83f9c3b41c168afd4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/263

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1530 more files in changeset.
Added a flag to identify first NC start during creation

Change-Id: If1b76ec83ae19f9c643a223e44712228ed7037f5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/272

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 7 more files in changeset.
Temp dataset support: 1. DDLs for creating a temporary dataset 2. Garbage collection for temporary dataset 3. Reading, inserting, and deleting data from (to) a temporary dataset is locking-free (except metadata locks) and logging-free (except flush/merge/job-commit logs).

Change-Id: Id0960acd1b4dfbfa7685dba227634572ca50b126

Reviewed-on: https://asterix-gerrit.ics.uci.edu/241

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 241 more files in changeset.
- Fixed Type Casting issue - Reorganized duplicated internal class in the DelimitedDataParser and DelimitedDataParserFactory - Prevented a user from creating an inverted index on a dataset with a variable-length PK - INT64 is now the default type - Issue 852 fixed

Change-Id: I2d71e8a21da4f709c3259a3d3f678c640f9e1160

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/192

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 847 more files in changeset.
Modified locking on DatasetLifeCycleManager

Change-Id: Ia5ab435f53879ba1d08b6dee24eb4969c5ad16e3

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/206

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 5 more files in changeset.
Added flush logs and enabled non-sharp checkpoints

The following commits from your working branch will be included:

commit c11a22cfd916041c806409c6d8e66dd6ea162b7b

Author: hubailmor@gmail.com <mhubail@uci.edu>

Date: Tue Nov 18 23:12:19 2014 -0800

Defined Startup LSN

commit 95a8752072e53ef62a11c39799f296209a3c622c

Author: hubailmor@gmail.com <mhubail@uci.edu>

Date: Tue Nov 18 20:22:08 2014 -0800

Revised Checkpoint Logic

commit ea4baa22089ba5d3eaff3d5c787828efcbff8ad9

Author: hubailmor@gmail.com <mhubail@uci.edu>

Date: Thu Nov 13 10:25:45 2014 -0800

added comments

commit 8991554f840f50d800a619fc2ac327f1c024e838

Author: hubailmor@gmail.com <mhubail@uci.edu>

Date: Wed Nov 12 23:34:25 2014 -0800

Added a method to the interface

commit a31e36b0b8234fe876b42e0cdc4543e4e73314a1

Author: hubailmor@gmail.com <mhubail@uci.edu>

Date: Wed Nov 12 22:57:51 2014 -0800

Refactored getMinFirstLSN code

commit 25f62d585b2f279e78711efd2d327c777919f62c

Author: hubailmor@gmail.com <mhubail@uci.edu>

Date: Tue Nov 11 14:05:34 2014 -0800

Added flush logs

Change-Id: I13fb61c04b6b510b6e8af68d2a0e7e14859f519d

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/182

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 19 more files in changeset.
763: Omit wrapper "results" JSON object in HTTP API output for JSON results; Wrap JSON output in outer [array]; consolidate output logic into ResultUtils; use UTF-8 explicitly when converting byte[] from Hyracks to Strings

Change-Id: Idbeac3ebbf397b0ef73cc54b7d901a21ac855932

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/123

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Keren-Audrey Ouaknine <kereno@gmail.com>

  1. … 5 more files in changeset.
Changes to fix issue793

Change-Id: I54281f8cccd2e9322c4a8f93a9aca573aea2c552

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/111

Reviewed-by: Till Westmann <westmann@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1 more file in changeset.
Added a new merge policy, called correlated-prefix.

Change-Id: I8f961eb28b4ea460f6326c8b4acaea02b50819e9

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/108

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 24 more files in changeset.
Added filters to AsterixDB's primary and secondary indexes.

Change-Id: If16b2758bb25392c8dd0405d7d59ae503efad3f0

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/82

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 114 more files in changeset.
Add external indexes

This change include the following:

1. additional data parser for external data that parse hdfs records using Hive serdes.

2. allow users to create external data. this includes:

a) changes in metadata external dataset details.

b) addition of a new metadata index to store external file's statuses.

c) the pipeline for building the B-Tree and R-Tree indexes.

d) hyracks operators to fetch records with their RIDs using different formats.

e) hyracks operators to lookup and parse external records.

f) test cases for indexing and index access of different hdfs file formats.

g) exposing the secondary indexes over external data to the compiler.

3. adding a new aql command to refresh external datasets. this includes

a) global recovery on system startup.

b) changes in the aql parser.

c) construction of bulk modify pipelines and additional operators to perform local commit and abort operations (using 2PC protocol).

4. Added copyright header to all new files

5. Added additional test cases to test left outer join on external data

Change-Id: I1065a473299f6027eb073aeeba3a56d137f6f98e

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/70

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 146 more files in changeset.