Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Asterix NCs Fault Tolerance

This change includes the following:

- Adapt replication to unique partitions storage.

- Implement auto failover for failing NCs.

- Implement auto failover for metadata node.

- Fix for ASTERIXDB-1251 using proper error message.

- Basic replication test cases using vagrant virtual cluster for:

1. LSM bulkload components replication.

2. LSM Memory components replication and recovery.

3. Metadata node takeover.

These test cases will be part of the cluster test profile.

Change-Id: Ice26d980912a315fcb3efdd571d6ce88717cfea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/573

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +257
    ./org/apache/asterix/installer/test/ReplicationIT.java
  1. … 71 more files in changeset.
First stage of external data cleanup

In this change, different parts of external data were refactored.

The goal was to make it more modular, easier to maintain and allow

higher flexibility for extension in addition to reducing code redundancy.

Change-Id: I04a8c4e494d8d1363992b6fe0bdbe6b2b3b7b767

Reviewed-on: https://asterix-gerrit.ics.uci.edu/566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 338 more files in changeset.
Run ExecutionTest w/integration test framework

I changed the default local.xml to use 2 NCs with

2 partitions each like AsterixHyracksIntegrationUtil

so that the results will match despite using simple

string compare to determine correctness.

There is also one test in particular (big-object-join)

that I had to add an artificial order by clause to

to deal with different results.

Issues outstanding:

- HDFS in ClusterExecutionIT needs to be thought out

Change-Id: I423f2a7c77839b999d466dd5cace302574d956c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/525

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

    • -0
    • +116
    ./org/apache/asterix/installer/test/AbstractExecutionIT.java
    • -0
    • +107
    ./org/apache/asterix/installer/test/ClusterExecutionIT.java
    • -0
    • +68
    ./org/apache/asterix/installer/test/ManagixSqlppExecutionIT.java
  1. … 3861 more files in changeset.
Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 109 more files in changeset.
ASTERIXDB-1126 Correct comparators and type traits for external index

In case of external data, the type definition does not contain information

about primary keys and we should get them based on the data input format.

Change-Id: I71d924d7e2b7a7e6c752bc97679e612946afc17c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/504

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 15 more files in changeset.
This change fixes the sporadic connection refused errors

This can mostly be seen during asterix-installer tests.

The fix basically lets managix wait and listens

through zookeeper for a signal from the cluster controller that

the cluster is up and ready.

Once the cluster controller sends the signal, Managix can proceed.

Change-Id: Ib730f50ab2fb492f3cf973d1cf2f03b34e24e5b3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/366

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 11 more files in changeset.
Change License Headers

Also tweak the NOTICE file with some extras.

Change-Id:I09bc388089e515d7f51fd39c31bfbbc9f00cf84f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/388

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 2013 more files in changeset.
Change Java package from edu.uci.ics to org.apache

Change-Id: I2f01d2b5614e9e9c94fda4bf1294a8eba6a26c54

Reviewed-on: https://asterix-gerrit.ics.uci.edu/309

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2599 more files in changeset.
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: Iefd2a576415ebc1416cba2a3334d2b64f042ba92

Reviewed-on: https://asterix-gerrit.ics.uci.edu/306

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +78
    ./org/apache/asterix/installer/test/AsterixExternalLibraryIT.java
    • -0
    • +144
    ./org/apache/asterix/installer/test/AsterixLifecycleIT.java
    • -0
    • +130
    ./org/apache/asterix/installer/transaction/DmlRecoveryIT.java
  1. … 3784 more files in changeset.
Add Apache RAT License Auditor plugin, and fix missing licenses

Change-Id: Ic0fa0b7836e8a69d0d8beed83f9c3b41c168afd4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/263

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1540 more files in changeset.
YARN integration for AsterixDB

This is an initial version of YARN integration for AsterixDB.

- Uses static assignment of CC and NC nodes to NM locations

- Stores state locally on each NM, outside of HDFS

- "All or nothing" container allocation. We don't attempt to

move or rellocate containers the RM may kill (yet).

- Retains feature parity with managix.

Change-Id: I49c849179d17fc7faa446b9be57a0695df6836ab

Reviewed-on: https://asterix-gerrit.ics.uci.edu/161

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 65 more files in changeset.
Introduces Feeds 2.0

commit c3f577861fc705d848c1641605689cadd6973bae

Merge: ebc4cae fc0c2c0

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Fri Jun 26 13:04:05 2015 -0700

Merge branch 'raman/feeds_2_release' of https://code.google.com/p/asterixdb-sandbox into raman/feeds_2_release

Conflicts:

asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/FeedServlet.java

asterix-external-data/src/main/java/edu/uci/ics/asterix/external/library/java/JObjectAccessors.java

commit ebc4cae21a7302869f953df1ebda601e798d12d2

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Sat Jun 20 17:14:45 2015 -0700

Introduces Feeds 2.0

Some of the prominent chnages introduced are as follows

a) Support for building a cascade network of feeds (via secondary feeds feature)

b) Feed Management Console for tracking active feeds and associated metrics

c) Support for elastic runtime for data ingestion

d) Improved fault-tolerance with support for logging of failed records

Documentation has been added at asterix-doc/src/site/markdown/feeds/

commit fc0c2c0549a6ee8b202e57607d2e110478cd57bb

Author: ramangrover29 <ramangrover29@gmail.com>

Date: Sat Jun 20 17:14:45 2015 -0700

Introduces Feeds 2.0

Some of the prominent chnages introduced are as follows

a) Support for building a cascade network of feeds (via secondary feeds feature)

b) Feed Management Console for tracking active feeds and associated metrics

c) Support for elastic runtime for data ingestion

d) Improved fault-tolerance with support for logging of failed records

Documentation has been added at asterix-doc/src/site/markdown/feeds/

Change-Id: I498f01c591a229aaf51cec43ab20f3e5c4f072f4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/297

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Steven Jacobs <sjaco002@ucr.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 383 more files in changeset.
Potential fix for sporadic issues with AsterixClusterLifeCycleIT.tearDown()

I think there was a race between these two adjacent methods in tearDown(), the latter of which looks like it isn't

even doing what I intended it to do at the moment. I haven't been able to observe this sporadic failure after changing

it so the latter one goes after the printing of the output of the former, which blocks and therefore would stop any

issues of this type.

Change-Id: I57ac2117c42a0b798f3249846f0b49927c64d870

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/91

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Basis for Cluster integration testing

This branch adds cluster testing via Vagrant.

Requires my branch of the vagrant-maven plugin to work,

which can be sourced here:

https://github.com/parshimers/vagrant-maven-plugin

It is enabled with -DclusterTest=true in mvn verify.

A virtualized cluster with 4 nodes is started, and

then Asterix is started via managix on this cluster,

and then stopped.

Change-Id: I7e3cdcd4162ada19ee1e15f532be7447b4f34367

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/31

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Reviewed-by: Zachary Heilbron <zheilbron@gmail.com>

Reviewed-by: Till Westmann <westmann@gmail.com>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

  1. … 7 more files in changeset.
some cleanup

added test cases for comparator issues

  1. … 15 more files in changeset.
DML_RECOVER_ON, DDL_RECOVER_ON flags added

secondary indices in use; comment addressed;

  1. … 8 more files in changeset.
modified test case to use adaptor alias instead of class name

  1. … 10 more files in changeset.
added test for installation and use of an external adaptor

  1. … 14 more files in changeset.
dml recovery added

    • -0
    • +117
    ./edu/uci/ics/asterix/installer/transaction/DmlRecoveryIT.java
  1. … 9 more files in changeset.
minor changes to pull based twitter adapter

  1. … 6 more files in changeset.
added an additional test for feed ingestion

  1. … 14 more files in changeset.
Merge branch 'master' into zheilbron/asterix_msr_demo

Conflicts:

asterix-app/src/main/java/edu/uci/ics/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java

asterix-app/src/test/resources/metadata/results/basic/meta17/meta17.1.adm

asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm

asterix-common/src/main/java/edu/uci/ics/asterix/common/config/AsterixExternalProperties.java

asterix-common/src/main/java/edu/uci/ics/asterix/common/transactions/ILockManager.java

asterix-metadata/src/main/java/edu/uci/ics/asterix/metadata/MetadataManager.java

asterix-metadata/src/main/java/edu/uci/ics/asterix/metadata/MetadataNode.java

asterix-metadata/src/main/java/edu/uci/ics/asterix/metadata/entitytupletranslators/DatasetTupleTranslator.java

  1. … 18 more files in changeset.
all refined expect secondary indices

  1. … 41 more files in changeset.
merge from master

  1. … 17 more files in changeset.
copy once

  1. … 11 more files in changeset.
after merge master

  1. … 1 more file in changeset.
revert some modifications

  1. … 2 more files in changeset.
three test cases on indexes

  1. … 43 more files in changeset.