Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
remove unused member

Change-Id: I8d0660faacc39aa3a0fca50f4813669a0179c408

Reviewed-on: https://asterix-gerrit.ics.uci.edu/653

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Enabled Feed Tests and Added External Library tests

Feed tests had been switched off for a while due to having too many

sporadic failures. Now, we are switching them back on.

In addition, a new set of tests have been added to test that external

library works as expected.

Change-Id: Idd1fccd136fa2645b2707bbf7c04e60991ae8d4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/625

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -10
    • +6
    ./metadata/feeds/FeedMetadataUtil.java
  1. … 203 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -1
    • +1
    ./metadata/bootstrap/MetadataIndex.java
    • -1
    • +1
    ./metadata/bootstrap/MetadataIndexImmutableProperties.java
    • -1
    • +1
    ./metadata/entities/CompactionPolicy.java
    • -1
    • +1
    ./metadata/entities/FeedPolicyEntity.java
  1. … 314 more files in changeset.
Asterix NCs Failback Support

- Allow Failed NCs to failback and takeover their partitions.

- New cluster API servlet for cluster state description.

- Remove nodeId from txn logs except remote FLUSH_LOG.

- Add partition id in UPDATE and Entity_COMMIT logs.

- Adapt remote recovery to new logs format.

- Refactor RecoveryManager and split Analysis and Redo phase.

- Spill remote recover logs to temporary file.

- Replicate files to remote replicas for partitions of interest only.

- Introduce NC active/inactive partitions concept.

- Test case for failback.

Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24

Reviewed-on: https://asterix-gerrit.ics.uci.edu/613

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 92 more files in changeset.
Support Sending Messages Alongside Frame Data

This change supports sending messages with records. The tuple Appender

reserves 100 bytes for a message. Before sending the frame, it appends

The message in the last tuple position. The message is read from the

task context as the shared object between different operators in the

pipeline. The first use of this feature will be within feeds to request

acks for at least once semantics.

Change-Id: Iaa23e9f8a909ddcafc1c3ee95181092eb04ee1ad

Reviewed-on: https://asterix-gerrit.ics.uci.edu/605

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +6
    ./metadata/feeds/FeedMetadataUtil.java
  1. … 13 more files in changeset.
Some exception cleanup

- Use HyracksDataException and IOException for errors ingesting external

data (instead of a mixture of AlgebricksException, AsterixException, and

HyracksDataException).

- Avoid unnecessary wrapping of exceptions.

- Wrap exception at boundaries (e.g. HiveRecordParser)

- ADMLexerException and ADMDataParser.ParseException extend

HyracksDataException

- Tighter exception declarations and handling for internal methods.

Change-Id: Icf9591046c44f5fa2281874ff0c98d780e741267

Reviewed-on: https://asterix-gerrit.ics.uci.edu/603

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -1
    • +1
    ./metadata/MetadataTransactionContext.java
  1. … 50 more files in changeset.
Add Support for Upsert Operation

This change allows users to execute upsert commands which

couples delete if found with insert. It locks the primary

keys before doing the search ensuring consistency.

Change-Id: I8999000331795a5949d621d2dd003903e057a521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +2
    ./metadata/utils/MetadataLockManager.java
  1. … 250 more files in changeset.
Add flush() to IFrameWriter

This method is expected to be used with feeds to push

frames all the way to storage when needed. As of now, it is

needed in two cases:

1. No activities in ingestion node and need to push content

so it can be stored.

2. When the ingestion node needs to move the checkpoint ahead

if the at least once semantics are used.

Two feeds make use of this function. The filesystem feed and

couchbase feed which was introduced as well in this change.

Change-Id: Id862ce9e9b1360864c6976f2aea2137092f51203

Reviewed-on: https://asterix-gerrit.ics.uci.edu/585

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -1
    • +2
    ./metadata/feeds/FeedMetadataUtil.java
  1. … 102 more files in changeset.
remove end-of-line whitespace

Change-Id: I5c0415f47d4c3a9827574fbdab949b45718d9ea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/601

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -3
    • +3
    ./metadata/api/IMetadataEntityTupleTranslator.java
    • -38
    • +38
    ./metadata/api/IMetadataNode.java
    • -1
    • +1
    ./metadata/cluster/AbstractClusterManagementWork.java
    • -1
    • +1
    ./metadata/cluster/ClusterManagementWorkResponse.java
    • -1
    • +1
    ./metadata/dataset/hints/DatasetHints.java
    • -1
    • +1
    ./metadata/utils/ExternalDatasetsRegistry.java
  1. … 130 more files in changeset.
ASTERIXDB-1005, ASTERIXDB-1263: Clean up subplan flattening: 1. Inline NestedTupleSource and remove SubplanOperator for special cases that join operators inside the SubplanOperator can be re-targeted for correlations; 2. Blindly inline NestedTupleSource and remove SubplanOperator for general cases where the condition of the special case is not met.

Change-Id: I4dd130a25f3c81272cc23f844ea20e376e990612

Reviewed-on: https://asterix-gerrit.ics.uci.edu/579

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -1
    • +1
    ./metadata/feeds/FeedMetadataUtil.java
  1. … 84 more files in changeset.
Add QueryServiceServlet

Adds a new improved HTTP endpoint for queries. Also introduces

initial stats gathering.

Change-Id: Ia494c54f7252445ce38903c0b58fc4e23c324e6e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/597

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

  1. … 8 more files in changeset.
Asterix NCs Fault Tolerance

This change includes the following:

- Adapt replication to unique partitions storage.

- Implement auto failover for failing NCs.

- Implement auto failover for metadata node.

- Fix for ASTERIXDB-1251 using proper error message.

- Basic replication test cases using vagrant virtual cluster for:

1. LSM bulkload components replication.

2. LSM Memory components replication and recovery.

3. Metadata node takeover.

These test cases will be part of the cluster test profile.

Change-Id: Ice26d980912a315fcb3efdd571d6ce88717cfea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/573

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +6
    ./metadata/utils/SplitsAndConstraintsUtil.java
  1. … 70 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -15
    • +8
    ./metadata/MetadataTransactionContext.java
    • -37
    • +4
    ./metadata/api/IMetadataManager.java
    • -0
    • +1
    ./metadata/bootstrap/MetadataIndex.java
    • -0
    • +2
    ./metadata/bootstrap/MetadataIndexImmutableProperties.java
    • -6
    • +11
    ./metadata/declared/FeedDataSource.java
    • -3
    • +3
    ./metadata/entities/CompactionPolicy.java
  1. … 403 more files in changeset.
Enabled Datasets to use Datatypes from foreign Dataverses

Removed broken Metadata Secondary Indexes

Added DatatypeDataverse field to Dataset

Cleaned up Metadata Tests

Added tests for dropping used Datatypes

User-facing changes are:

1) Allow qualified datatype when creating dataset

2) Metadata queries for datasets will now

include the type dataverse

Change-Id: I24dbc04dcb2a4126fc8361ebe3104877a0d1f2bb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/558

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -165
    • +124
    ./metadata/MetadataNode.java
    • -1
    • +1
    ./metadata/MetadataTransactionContext.java
    • -4
    • +1
    ./metadata/bootstrap/MetadataIndexImmutableProperties.java
    • -8
    • +20
    ./metadata/utils/MetadataLockManager.java
  1. … 126 more files in changeset.
Improve the Testing Framework

This change includes the following improvements to the test framework

1. Checking each expected failure against the failure exception.

2. Checking against multiple causes of failure in case of multiple

expected errors.

3. Check if a test that is expected to fail pass.

4. Make temp datasets use different storage path.

Change-Id: I957ecf19bf7209981e010e0e50fb882442a525dd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/537

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +4
    ./metadata/utils/SplitsAndConstraintsUtil.java
  1. … 28 more files in changeset.
some cleanup

- remove redundant code around DataSources

- fix typo in constant name

Change-Id: I48602e1acbbc22642d4467ab37f27f337de20c5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/583

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -18
    • +10
    ./metadata/declared/AqlDataSource.java
    • -18
    • +2
    ./metadata/declared/DatasetDataSource.java
    • -12
    • +2
    ./metadata/declared/FeedDataSource.java
  1. … 3 more files in changeset.
First stage of external data cleanup

In this change, different parts of external data were refactored.

The goal was to make it more modular, easier to maintain and allow

higher flexibility for extension in addition to reducing code redundancy.

Change-Id: I04a8c4e494d8d1363992b6fe0bdbe6b2b3b7b767

Reviewed-on: https://asterix-gerrit.ics.uci.edu/566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -168
    • +55
    ./metadata/declared/AqlMetadataProvider.java
    • -170
    • +0
    ./metadata/declared/FieldExtractingAdapter.java
    • -86
    • +0
    ./metadata/declared/FieldExtractingAdapterFactory.java
    • -86
    • +0
    ./metadata/declared/PKGeneratingAdapterFactory.java
    • -2
    • +2
    ./metadata/feeds/AbstractDatasourceAdapter.java
    • -2
    • +2
    ./metadata/feeds/AbstractFeedDatasourceAdapter.java
    • -3
    • +3
    ./metadata/feeds/ExternalDataScanOperatorDescriptor.java
    • -10
    • +10
    ./metadata/feeds/FeedIntakeOperatorDescriptor.java
    • -10
    • +11
    ./metadata/feeds/FeedIntakeOperatorNodePushable.java
    • -1
    • +1
    ./metadata/feeds/ITypedAdapterFactory.java
  1. … 329 more files in changeset.
Divide Cluster into Unique Partitions

The change includes the following:

- Fix passing NC stores to AsterixConfiguration.

- Unify storage direcotry name in the instance level rather than the node level.

- Divide the cluster into unique storage partitions based on the number of stores.

- Refactored FileSplits and moved out of AqlMetadataProvider.

- Make AsterixHyracksIntegrationUtil use the passed configuration file.

- Make File Splits pass relative index paths of partitions rather than absolute paths.

- Remove unused AqlCompiledMetadataDeclarations class.

Change-Id: I8c7fbca5113dd7ad569a46dfa2591addb5bf8655

Reviewed-on: https://asterix-gerrit.ics.uci.edu/564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -276
    • +0
    ./metadata/declared/AqlCompiledMetadataDeclarations.java
    • -15
    • +1
    ./metadata/declared/AqlLogicalPlanAndMetadataImpl.java
    • -154
    • +10
    ./metadata/declared/AqlMetadataProvider.java
    • -0
    • +173
    ./metadata/utils/SplitsAndConstraintsUtil.java
  1. … 37 more files in changeset.
Refactored External Data

This change re-arrange asterix module's order. asterix-

external-data is moved in front of asterix-metadata.

Change-Id: I46b60b5e1cc37fd59adc0dd89f374d96502091b2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/559

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -159
    • +243
    ./metadata/declared/AqlMetadataProvider.java
    • -2
    • +2
    ./metadata/declared/FieldExtractingAdapterFactory.java
    • -146
    • +0
    ./metadata/entities/ExternalFile.java
    • -148
    • +0
    ./metadata/external/ExternalFileIndexAccessor.java
    • -137
    • +0
    ./metadata/external/ExternalLoopkupOperatorDiscriptor.java
    • -98
    • +0
    ./metadata/external/IAdapterFactory.java
  1. … 79 more files in changeset.
Asterix MessageBroker implementation

This change includes the following:

- Add implementation for CC/NC MessageBroker.

- Implement GlobalResourceIdFactory using MessageBroker.

- Change resource id factory to GlobalResourceIdFactory.

- Refactor metadata indexes fixed properties.

- Use fixed resource ids for metadata indexes.

Change-Id: If4320e2c5a0130d2f86a4be6ae61f5cee43e30af

Reviewed-on: https://asterix-gerrit.ics.uci.edu/486

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -10
    • +7
    ./metadata/bootstrap/MetadataIndex.java
    • -0
    • +79
    ./metadata/bootstrap/MetadataIndexImmutableProperties.java
    • -13
    • +14
    ./metadata/bootstrap/MetadataSecondaryIndexes.java
  1. … 21 more files in changeset.
Changed the IFrameWriter Contract

Updated existing operators to conform to the new contract.

These operators are either index operators or Feed operators.

The rest of the operator already follow the new contract.

Change-Id: Ibcebe876340a25be0f561945582a95211c140e10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/552

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -24
    • +32
    ./metadata/external/ExternalLoopkupOperatorDiscriptor.java
    • -5
    • +5
    ./metadata/feeds/ExternalDataScanOperatorDescriptor.java
    • -10
    • +11
    ./metadata/feeds/FeedMetaComputeNodePushable.java
  1. … 6 more files in changeset.
ASTERIX-1211: Malformed metadata record for enforced open index - Changed type on the open olist field in index metadata to [ ANY ] - Fixed heterogeneous list deserialization

Change-Id: Ib587a138f59f37c369ba01e627a1ef9dd5a4989e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/532

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 9 more files in changeset.
ASTERIXDB-1198: make ISerializerDeserializer implementations in AsterixDB stateless.

Change-Id: I808b6583db6b48a0e9000dcf0ac88c03c336c3a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/522

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

  1. … 59 more files in changeset.
Fix ASTERIXDB-1194,ASTERIXDB-1195,ASTERIXDB-1196,ASTERIXDB-1197.

Change-Id: I7d167b64bf9ec754182b5b2fe44dfc7e5908c686

Reviewed-on: https://asterix-gerrit.ics.uci.edu/325

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -28
    • +6
    ./metadata/utils/MetadataLockManager.java
  1. … 24 more files in changeset.
Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -15
    • +15
    ./metadata/feeds/FeedIntakeOperatorNodePushable.java
    • -7
    • +11
    ./metadata/feeds/FeedMessageOperatorNodePushable.java
    • -49
    • +1
    ./metadata/feeds/RemoteSocketMessageListener.java
  1. … 100 more files in changeset.
Clean up asterix-om to eliminate compilation warnings.

Change-Id: Ib1c562d087097d9cc749f1e04f1f06106c7f1625

Reviewed-on: https://asterix-gerrit.ics.uci.edu/503

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -42
    • +15
    ./metadata/functions/ExternalFunctionCompilerUtil.java
    • -0
    • +27
    ./metadata/functions/ExternalTypeComputer.java
  1. … 41 more files in changeset.
ASTERIXDB-1126 Correct comparators and type traits for external index

In case of external data, the type definition does not contain information

about primary keys and we should get them based on the data input format.

Change-Id: I71d924d7e2b7a7e6c752bc97679e612946afc17c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/504

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 15 more files in changeset.
ASTERIXDB-1159: make ARecordType read-only and separate its mutable state to RuntimeRecordTypeInfo.

Change-Id: Ibbfa73907ae9422a4ad85dc4db5fa568855beb17

Reviewed-on: https://asterix-gerrit.ics.uci.edu/501

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 20 more files in changeset.
ASTERIXDB-1176: remove unused feeds code

Change-Id: I2de2d7c7fd816ddbd53a80c855f64923c02efe35

Reviewed-on: https://asterix-gerrit.ics.uci.edu/494

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -42
    • +0
    ./metadata/feeds/EndFeedMessage.java
    • -65
    • +0
    ./metadata/feeds/FeedManagerElectMessage.java
  1. … 3 more files in changeset.
Introducing Data Replication To AsterixDB

This change includes the following:

- Add data replication properties to cluster properties and Managix validate command.

- Introduce Data Replication components.

- Add data replication required fields to LogRecord.

- Specialized LogManager for data replication.

- Fix for invalid cluster state on nodes failure.

- ASTERIXDB-139: Fix for cleaning workspace files on startup/shutdown.

- Fix for temp datasets storage reclamation.

- Allow MetadataNode rebinding with CC.

- Add flag to checkpoint to identify sharp checkpoints.

- ASTERIXDB-1170: Fix shutdown sequence

Change-Id: I729fdd1144dbc9ff039b4bc414494860d7553810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/338

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 72 more files in changeset.