Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
ASTERIXDB-1337: Dataset Memory Management on Multi-Partition NC

As sugggested in ASTERIXDB-1337, this change is to maintain a per-

partition MultitenantVirtualBufferCache budget, as opposed to sharing

the budget across the dataset.

Change-Id: Ibbf08f532c1210c30be6a51c73570a789174213b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/705

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -9
    • +3
    ./common/AsterixAppRuntimeContext.java
    • -109
    • +0
    ./common/AsterixAppRuntimeContextProdiverForRecovery.java
    • -0
    • +101
    ./common/AsterixAppRuntimeContextProviderForRecovery.java
  1. … 16 more files in changeset.
ASTERIXDB-1340: fix node group nodes order

Change-Id: I6e93aedb6cdc57d98f83991ab910476b82bacb15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/708

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -2
    • +2
    ./common/AsterixHyracksIntegrationUtil.java
  1. … 4 more files in changeset.
patch 8

  1. … 73 more files in changeset.
ASTERIXDB-1302 ASTERIXDB-1301 Fix Socket Feed Connection

A bug causes a read lock to never be released when a feed is

connected with "wait-for-completion" set to false. The bug

was fixed and a test case was added.

Another bug was causing the socket feed to not receive

connections correctly. The bug was fixed and a test case

was added.

Additionally, this change ensures that adapters have absolute

partitions to ensure consistency with regards to feed log

manager.

Change-Id: I8f6e982440d3577343f2479c3779653a9c3db614

Reviewed-on: https://asterix-gerrit.ics.uci.edu/660

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

    • -2
    • +3
    ./common/AsterixHyracksIntegrationUtil.java
  1. … 49 more files in changeset.
Enabled Feed Tests and Added External Library tests

Feed tests had been switched off for a while due to having too many

sporadic failures. Now, we are switching them back on.

In addition, a new set of tests have been added to test that external

library works as expected.

Change-Id: Idd1fccd136fa2645b2707bbf7c04e60991ae8d4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/625

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +1
    ./http/servlet/ConnectorAPIServlet.java
  1. … 202 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 326 more files in changeset.
Asterix NCs Failback Support

- Allow Failed NCs to failback and takeover their partitions.

- New cluster API servlet for cluster state description.

- Remove nodeId from txn logs except remote FLUSH_LOG.

- Add partition id in UPDATE and Entity_COMMIT logs.

- Adapt remote recovery to new logs format.

- Refactor RecoveryManager and split Analysis and Redo phase.

- Spill remote recover logs to temporary file.

- Replicate files to remote replicas for partitions of interest only.

- Introduce NC active/inactive partitions concept.

- Test case for failback.

Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24

Reviewed-on: https://asterix-gerrit.ics.uci.edu/613

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -11
    • +40
    ./common/AsterixAppRuntimeContext.java
    • -0
    • +51
    ./http/servlet/ClusterAPIServlet.java
  1. … 91 more files in changeset.
Add Support for Upsert Operation

This change allows users to execute upsert commands which

couples delete if found with insert. It locks the primary

keys before doing the search ensuring consistency.

Change-Id: I8999000331795a5949d621d2dd003903e057a521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 251 more files in changeset.
Return ParseExceptions Stacktrace in RESTAPIServlet

Change-Id: Ia22a275aa328c54254c3d0193a472460995f80c5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/599

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 1 more file in changeset.
Add QueryServiceServlet

Adds a new improved HTTP endpoint for queries. Also introduces

initial stats gathering.

Change-Id: Ia494c54f7252445ce38903c0b58fc4e23c324e6e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/597

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

    • -0
    • +1
    ./common/AsterixHyracksIntegrationUtil.java
    • -0
    • +132
    ./http/servlet/JSONUtil.java
    • -4
    • +3
    ./http/servlet/QueryResultAPIServlet.java
    • -0
    • +342
    ./http/servlet/QueryServiceServlet.java
  1. … 4 more files in changeset.
Asterix NCs Fault Tolerance

This change includes the following:

- Adapt replication to unique partitions storage.

- Implement auto failover for failing NCs.

- Implement auto failover for metadata node.

- Fix for ASTERIXDB-1251 using proper error message.

- Basic replication test cases using vagrant virtual cluster for:

1. LSM bulkload components replication.

2. LSM Memory components replication and recovery.

3. Metadata node takeover.

These test cases will be part of the cluster test profile.

Change-Id: Ice26d980912a315fcb3efdd571d6ce88717cfea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/573

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -9
    • +50
    ./common/AsterixAppRuntimeContext.java
  1. … 71 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -2
    • +2
    ./common/AsterixAppRuntimeContext.java
    • -205
    • +0
    ./common/FeedWorkCollection.java
    • -1
    • +1
    ./http/servlet/ConnectorAPIServlet.java
  1. … 413 more files in changeset.
Enabled Datasets to use Datatypes from foreign Dataverses

Removed broken Metadata Secondary Indexes

Added DatatypeDataverse field to Dataset

Cleaned up Metadata Tests

Added tests for dropping used Datatypes

User-facing changes are:

1) Allow qualified datatype when creating dataset

2) Metadata queries for datasets will now

include the type dataverse

Change-Id: I24dbc04dcb2a4126fc8361ebe3104877a0d1f2bb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/558

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -7
    • +7
    ./http/servlet/ConnectorAPIServlet.java
  1. … 139 more files in changeset.
Improve the Testing Framework

This change includes the following improvements to the test framework

1. Checking each expected failure against the failure exception.

2. Checking against multiple causes of failure in case of multiple

expected errors.

3. Check if a test that is expected to fail pass.

4. Make temp datasets use different storage path.

Change-Id: I957ecf19bf7209981e010e0e50fb882442a525dd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/537

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -1
    • +1
    ./common/AsterixHyracksIntegrationUtil.java
  1. … 28 more files in changeset.
Divide Cluster into Unique Partitions

The change includes the following:

- Fix passing NC stores to AsterixConfiguration.

- Unify storage direcotry name in the instance level rather than the node level.

- Divide the cluster into unique storage partitions based on the number of stores.

- Refactored FileSplits and moved out of AqlMetadataProvider.

- Make AsterixHyracksIntegrationUtil use the passed configuration file.

- Make File Splits pass relative index paths of partitions rather than absolute paths.

- Remove unused AqlCompiledMetadataDeclarations class.

Change-Id: I8c7fbca5113dd7ad569a46dfa2591addb5bf8655

Reviewed-on: https://asterix-gerrit.ics.uci.edu/564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -3
    • +3
    ./common/AsterixAppRuntimeContext.java
    • -34
    • +33
    ./common/AsterixHyracksIntegrationUtil.java
  1. … 40 more files in changeset.
Asterix MessageBroker implementation

This change includes the following:

- Add implementation for CC/NC MessageBroker.

- Implement GlobalResourceIdFactory using MessageBroker.

- Change resource id factory to GlobalResourceIdFactory.

- Refactor metadata indexes fixed properties.

- Use fixed resource ids for metadata indexes.

Change-Id: If4320e2c5a0130d2f86a4be6ae61f5cee43e30af

Reviewed-on: https://asterix-gerrit.ics.uci.edu/486

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -11
    • +13
    ./common/AsterixAppRuntimeContext.java
    • -2
    • +2
    ./common/AsterixAppRuntimeContextProdiverForRecovery.java
  1. … 25 more files in changeset.
patch 2

  1. … 155 more files in changeset.
Run ExecutionTest w/integration test framework

I changed the default local.xml to use 2 NCs with

2 partitions each like AsterixHyracksIntegrationUtil

so that the results will match despite using simple

string compare to determine correctness.

There is also one test in particular (big-object-join)

that I had to add an artificial order by clause to

to deal with different results.

Issues outstanding:

- HDFS in ClusterExecutionIT needs to be thought out

Change-Id: I423f2a7c77839b999d466dd5cace302574d956c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/525

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

    • -2
    • +2
    ./common/AsterixHyracksIntegrationUtil.java
  1. … 3866 more files in changeset.
Fix ASTERIXDB-1194,ASTERIXDB-1195,ASTERIXDB-1196,ASTERIXDB-1197.

Change-Id: I7d167b64bf9ec754182b5b2fe44dfc7e5908c686

Reviewed-on: https://asterix-gerrit.ics.uci.edu/325

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -6
    • +44
    ./common/AsterixHyracksIntegrationUtil.java
    • -5
    • +20
    ./http/servlet/ConnectorAPIServlet.java
  1. … 22 more files in changeset.
Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -14
    • +3
    ./http/servlet/ShutdownAPIServlet.java
    • -12
    • +8
    ./http/servlet/VersionAPIServlet.java
  1. … 106 more files in changeset.
ASTERIXDB-1152: Delete storage data of old instances

Change-Id: Ibb6c6949bdf2ed6c3e491fa66a23491ff34fc830

Reviewed-on: https://asterix-gerrit.ics.uci.edu/469

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -6
    • +14
    ./common/AsterixAppRuntimeContext.java
  1. … 6 more files in changeset.
SQL++ support in AsterixDB: 1. implemented SQL++ expression to logical plan translator; 2. refactored REST API to be agnostic of query languages; 3. disabled fuzzy join queries for SQL++ runtime tests; 4. fixed several rewriting rules.

Change-Id: I82919c4527b304325059519d819a2c30cf2902a9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/479

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -17
    • +26
    ./http/servlet/RESTAPIServlet.java
  1. … 1371 more files in changeset.
Verify that BufferCache is clean after tests

Change-Id: I5610e903be3347893e676c915b98316b06073c26

Reviewed-on: https://asterix-gerrit.ics.uci.edu/498

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -3
    • +3
    ./common/AsterixHyracksIntegrationUtil.java
  1. … 2 more files in changeset.
Introducing Data Replication To AsterixDB

This change includes the following:

- Add data replication properties to cluster properties and Managix validate command.

- Introduce Data Replication components.

- Add data replication required fields to LogRecord.

- Specialized LogManager for data replication.

- Fix for invalid cluster state on nodes failure.

- ASTERIXDB-139: Fix for cleaning workspace files on startup/shutdown.

- Fix for temp datasets storage reclamation.

- Allow MetadataNode rebinding with CC.

- Add flag to checkpoint to identify sharp checkpoints.

- ASTERIXDB-1170: Fix shutdown sequence

Change-Id: I729fdd1144dbc9ff039b4bc414494860d7553810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/338

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

    • -7
    • +101
    ./common/AsterixAppRuntimeContext.java
  1. … 73 more files in changeset.
ASTERIXDB-1058: make Asterix compatible with lazy LSM memory allocation

- Adapt memory budget calculation to lazy LSM memory allocation.

- Add IDatasetLifecycleManager interface.

Change-Id: I4ea1eb129fe3043d43b077473dc29d17a97dfcc2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/408

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

    • -14
    • +11
    ./common/AsterixAppRuntimeContext.java
    • -3
    • +3
    ./common/AsterixAppRuntimeContextProdiverForRecovery.java
  1. … 31 more files in changeset.
ASTERIXDB-1148: Selectable array-wrapping of results

Introduce "wrapper-array" parameter to HTTP API which selects (for ADM and

JSON) whether to wrap the result sequence in a generated outer array. For

JSON this defaults to "true" as before. For ADM this defaults to false,

resulting in a large number of expected-results changes.

Also introduce ability to have AQL tests which provide HTTP parameters.

Change-Id: I3122f136ff9ca8a2c2268238c57bb5eddab7b27e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/473

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

    • -12
    • +34
    ./http/servlet/RESTAPIServlet.java
  1. … 1438 more files in changeset.
SQL++ parser: 1. refactored asterix-aql to become asterix-lang-common and asterix-lang-aql, where the former is the common part for different languages; 2. added asterix-lang-sqlpp on top of asterix-lang-common; 3. ported parser tests, optimizer tests and runtime tests in asterix-app to their sql++ version, and added parser tests for all the queries.

Change-Id: Ie5af4e3b692ca017ec047a1ba3b404a51beb3a2e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/466

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -22
    • +17
    ./http/servlet/RESTAPIServlet.java
  1. … 10215 more files in changeset.
ASTERIXDB-1153: Determine build information at runtime

Very simple API. Just do a GET against /admin/version and get all of the build-time git info in JSON.

Change-Id: Ie392eb0cdbd25f2f4679fba12aae4c7a496e9637

Reviewed-on: https://asterix-gerrit.ics.uci.edu/468

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +8
    ./common/AsterixAppRuntimeContext.java
    • -0
    • +53
    ./http/servlet/VersionAPIServlet.java
  1. … 8 more files in changeset.
ASTERIXDB-1053: Allow lazy loading for persistent local resources

- Allow indexes to be accessed by name.

- Allow lazy loading for persistent local resources.

- Caching for local resources.

Change-Id: I48b9260a3280750145f6ddb3783673a299055910

Reviewed-on: https://asterix-gerrit.ics.uci.edu/344

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -6
    • +5
    ./common/AsterixAppRuntimeContext.java
  1. … 15 more files in changeset.
ASTERIXDB-1083: Fixed non-query statements' plans display on WebUI

Change-Id: I52dd69062b2aaf89798ebb8e0e58a1941ac4119e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/390

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 2 more files in changeset.