Clone Tools
  • last updated 19 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Add Support for Upsert Operation

This change allows users to execute upsert commands which

couples delete if found with insert. It locks the primary

keys before doing the search ensuring consistency.

Change-Id: I8999000331795a5949d621d2dd003903e057a521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +2
    ./metadata/utils/MetadataLockManager.java
  1. … 250 more files in changeset.
Add flush() to IFrameWriter

This method is expected to be used with feeds to push

frames all the way to storage when needed. As of now, it is

needed in two cases:

1. No activities in ingestion node and need to push content

so it can be stored.

2. When the ingestion node needs to move the checkpoint ahead

if the at least once semantics are used.

Two feeds make use of this function. The filesystem feed and

couchbase feed which was introduced as well in this change.

Change-Id: Id862ce9e9b1360864c6976f2aea2137092f51203

Reviewed-on: https://asterix-gerrit.ics.uci.edu/585

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -1
    • +2
    ./metadata/feeds/FeedMetadataUtil.java
  1. … 102 more files in changeset.
remove end-of-line whitespace

Change-Id: I5c0415f47d4c3a9827574fbdab949b45718d9ea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/601

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -3
    • +3
    ./metadata/api/IMetadataEntityTupleTranslator.java
    • -38
    • +38
    ./metadata/api/IMetadataNode.java
    • -1
    • +1
    ./metadata/cluster/AbstractClusterManagementWork.java
    • -1
    • +1
    ./metadata/cluster/ClusterManagementWorkResponse.java
    • -1
    • +1
    ./metadata/dataset/hints/DatasetHints.java
    • -1
    • +1
    ./metadata/utils/ExternalDatasetsRegistry.java
  1. … 130 more files in changeset.
ASTERIXDB-1005, ASTERIXDB-1263: Clean up subplan flattening: 1. Inline NestedTupleSource and remove SubplanOperator for special cases that join operators inside the SubplanOperator can be re-targeted for correlations; 2. Blindly inline NestedTupleSource and remove SubplanOperator for general cases where the condition of the special case is not met.

Change-Id: I4dd130a25f3c81272cc23f844ea20e376e990612

Reviewed-on: https://asterix-gerrit.ics.uci.edu/579

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -1
    • +1
    ./metadata/feeds/FeedMetadataUtil.java
  1. … 84 more files in changeset.
Add QueryServiceServlet

Adds a new improved HTTP endpoint for queries. Also introduces

initial stats gathering.

Change-Id: Ia494c54f7252445ce38903c0b58fc4e23c324e6e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/597

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

  1. … 8 more files in changeset.
Asterix NCs Fault Tolerance

This change includes the following:

- Adapt replication to unique partitions storage.

- Implement auto failover for failing NCs.

- Implement auto failover for metadata node.

- Fix for ASTERIXDB-1251 using proper error message.

- Basic replication test cases using vagrant virtual cluster for:

1. LSM bulkload components replication.

2. LSM Memory components replication and recovery.

3. Metadata node takeover.

These test cases will be part of the cluster test profile.

Change-Id: Ice26d980912a315fcb3efdd571d6ce88717cfea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/573

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +6
    ./metadata/utils/SplitsAndConstraintsUtil.java
  1. … 70 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -15
    • +8
    ./metadata/MetadataTransactionContext.java
    • -37
    • +4
    ./metadata/api/IMetadataManager.java
    • -0
    • +1
    ./metadata/bootstrap/MetadataIndex.java
    • -0
    • +2
    ./metadata/bootstrap/MetadataIndexImmutableProperties.java
    • -6
    • +11
    ./metadata/declared/FeedDataSource.java
    • -3
    • +3
    ./metadata/entities/CompactionPolicy.java
  1. … 403 more files in changeset.
Enabled Datasets to use Datatypes from foreign Dataverses

Removed broken Metadata Secondary Indexes

Added DatatypeDataverse field to Dataset

Cleaned up Metadata Tests

Added tests for dropping used Datatypes

User-facing changes are:

1) Allow qualified datatype when creating dataset

2) Metadata queries for datasets will now

include the type dataverse

Change-Id: I24dbc04dcb2a4126fc8361ebe3104877a0d1f2bb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/558

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -165
    • +124
    ./metadata/MetadataNode.java
    • -1
    • +1
    ./metadata/MetadataTransactionContext.java
    • -4
    • +1
    ./metadata/bootstrap/MetadataIndexImmutableProperties.java
    • -8
    • +20
    ./metadata/utils/MetadataLockManager.java
  1. … 126 more files in changeset.
Improve the Testing Framework

This change includes the following improvements to the test framework

1. Checking each expected failure against the failure exception.

2. Checking against multiple causes of failure in case of multiple

expected errors.

3. Check if a test that is expected to fail pass.

4. Make temp datasets use different storage path.

Change-Id: I957ecf19bf7209981e010e0e50fb882442a525dd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/537

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +4
    ./metadata/utils/SplitsAndConstraintsUtil.java
  1. … 28 more files in changeset.
some cleanup

- remove redundant code around DataSources

- fix typo in constant name

Change-Id: I48602e1acbbc22642d4467ab37f27f337de20c5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/583

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -18
    • +10
    ./metadata/declared/AqlDataSource.java
    • -18
    • +2
    ./metadata/declared/DatasetDataSource.java
    • -12
    • +2
    ./metadata/declared/FeedDataSource.java
  1. … 3 more files in changeset.
First stage of external data cleanup

In this change, different parts of external data were refactored.

The goal was to make it more modular, easier to maintain and allow

higher flexibility for extension in addition to reducing code redundancy.

Change-Id: I04a8c4e494d8d1363992b6fe0bdbe6b2b3b7b767

Reviewed-on: https://asterix-gerrit.ics.uci.edu/566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -168
    • +55
    ./metadata/declared/AqlMetadataProvider.java
    • -170
    • +0
    ./metadata/declared/FieldExtractingAdapter.java
    • -86
    • +0
    ./metadata/declared/FieldExtractingAdapterFactory.java
    • -86
    • +0
    ./metadata/declared/PKGeneratingAdapterFactory.java
    • -2
    • +2
    ./metadata/feeds/AbstractDatasourceAdapter.java
    • -2
    • +2
    ./metadata/feeds/AbstractFeedDatasourceAdapter.java
    • -3
    • +3
    ./metadata/feeds/ExternalDataScanOperatorDescriptor.java
    • -10
    • +10
    ./metadata/feeds/FeedIntakeOperatorDescriptor.java
    • -10
    • +11
    ./metadata/feeds/FeedIntakeOperatorNodePushable.java
    • -1
    • +1
    ./metadata/feeds/ITypedAdapterFactory.java
  1. … 329 more files in changeset.
Divide Cluster into Unique Partitions

The change includes the following:

- Fix passing NC stores to AsterixConfiguration.

- Unify storage direcotry name in the instance level rather than the node level.

- Divide the cluster into unique storage partitions based on the number of stores.

- Refactored FileSplits and moved out of AqlMetadataProvider.

- Make AsterixHyracksIntegrationUtil use the passed configuration file.

- Make File Splits pass relative index paths of partitions rather than absolute paths.

- Remove unused AqlCompiledMetadataDeclarations class.

Change-Id: I8c7fbca5113dd7ad569a46dfa2591addb5bf8655

Reviewed-on: https://asterix-gerrit.ics.uci.edu/564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -276
    • +0
    ./metadata/declared/AqlCompiledMetadataDeclarations.java
    • -15
    • +1
    ./metadata/declared/AqlLogicalPlanAndMetadataImpl.java
    • -154
    • +10
    ./metadata/declared/AqlMetadataProvider.java
    • -0
    • +173
    ./metadata/utils/SplitsAndConstraintsUtil.java
  1. … 37 more files in changeset.
Refactored External Data

This change re-arrange asterix module's order. asterix-

external-data is moved in front of asterix-metadata.

Change-Id: I46b60b5e1cc37fd59adc0dd89f374d96502091b2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/559

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -159
    • +243
    ./metadata/declared/AqlMetadataProvider.java
    • -2
    • +2
    ./metadata/declared/FieldExtractingAdapterFactory.java
    • -146
    • +0
    ./metadata/entities/ExternalFile.java
    • -148
    • +0
    ./metadata/external/ExternalFileIndexAccessor.java
    • -137
    • +0
    ./metadata/external/ExternalLoopkupOperatorDiscriptor.java
    • -98
    • +0
    ./metadata/external/IAdapterFactory.java
  1. … 79 more files in changeset.
Asterix MessageBroker implementation

This change includes the following:

- Add implementation for CC/NC MessageBroker.

- Implement GlobalResourceIdFactory using MessageBroker.

- Change resource id factory to GlobalResourceIdFactory.

- Refactor metadata indexes fixed properties.

- Use fixed resource ids for metadata indexes.

Change-Id: If4320e2c5a0130d2f86a4be6ae61f5cee43e30af

Reviewed-on: https://asterix-gerrit.ics.uci.edu/486

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -10
    • +7
    ./metadata/bootstrap/MetadataIndex.java
    • -0
    • +79
    ./metadata/bootstrap/MetadataIndexImmutableProperties.java
    • -13
    • +14
    ./metadata/bootstrap/MetadataSecondaryIndexes.java
  1. … 21 more files in changeset.
Changed the IFrameWriter Contract

Updated existing operators to conform to the new contract.

These operators are either index operators or Feed operators.

The rest of the operator already follow the new contract.

Change-Id: Ibcebe876340a25be0f561945582a95211c140e10

Reviewed-on: https://asterix-gerrit.ics.uci.edu/552

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -24
    • +32
    ./metadata/external/ExternalLoopkupOperatorDiscriptor.java
    • -5
    • +5
    ./metadata/feeds/ExternalDataScanOperatorDescriptor.java
    • -10
    • +11
    ./metadata/feeds/FeedMetaComputeNodePushable.java
  1. … 6 more files in changeset.
ASTERIX-1211: Malformed metadata record for enforced open index - Changed type on the open olist field in index metadata to [ ANY ] - Fixed heterogeneous list deserialization

Change-Id: Ib587a138f59f37c369ba01e627a1ef9dd5a4989e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/532

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 9 more files in changeset.
ASTERIXDB-1198: make ISerializerDeserializer implementations in AsterixDB stateless.

Change-Id: I808b6583db6b48a0e9000dcf0ac88c03c336c3a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/522

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

  1. … 59 more files in changeset.
Fix ASTERIXDB-1194,ASTERIXDB-1195,ASTERIXDB-1196,ASTERIXDB-1197.

Change-Id: I7d167b64bf9ec754182b5b2fe44dfc7e5908c686

Reviewed-on: https://asterix-gerrit.ics.uci.edu/325

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -28
    • +6
    ./metadata/utils/MetadataLockManager.java
  1. … 24 more files in changeset.
Clean up compilation warnings.

Change-Id: Idbfcd9c67f91d373c5f7269125778a5681021227

Reviewed-on: https://asterix-gerrit.ics.uci.edu/505

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -15
    • +15
    ./metadata/feeds/FeedIntakeOperatorNodePushable.java
    • -7
    • +11
    ./metadata/feeds/FeedMessageOperatorNodePushable.java
    • -49
    • +1
    ./metadata/feeds/RemoteSocketMessageListener.java
  1. … 100 more files in changeset.
Clean up asterix-om to eliminate compilation warnings.

Change-Id: Ib1c562d087097d9cc749f1e04f1f06106c7f1625

Reviewed-on: https://asterix-gerrit.ics.uci.edu/503

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -42
    • +15
    ./metadata/functions/ExternalFunctionCompilerUtil.java
    • -0
    • +27
    ./metadata/functions/ExternalTypeComputer.java
  1. … 41 more files in changeset.
ASTERIXDB-1126 Correct comparators and type traits for external index

In case of external data, the type definition does not contain information

about primary keys and we should get them based on the data input format.

Change-Id: I71d924d7e2b7a7e6c752bc97679e612946afc17c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/504

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 15 more files in changeset.
ASTERIXDB-1159: make ARecordType read-only and separate its mutable state to RuntimeRecordTypeInfo.

Change-Id: Ibbfa73907ae9422a4ad85dc4db5fa568855beb17

Reviewed-on: https://asterix-gerrit.ics.uci.edu/501

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 20 more files in changeset.
ASTERIXDB-1176: remove unused feeds code

Change-Id: I2de2d7c7fd816ddbd53a80c855f64923c02efe35

Reviewed-on: https://asterix-gerrit.ics.uci.edu/494

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -42
    • +0
    ./metadata/feeds/EndFeedMessage.java
    • -65
    • +0
    ./metadata/feeds/FeedManagerElectMessage.java
  1. … 3 more files in changeset.
Introducing Data Replication To AsterixDB

This change includes the following:

- Add data replication properties to cluster properties and Managix validate command.

- Introduce Data Replication components.

- Add data replication required fields to LogRecord.

- Specialized LogManager for data replication.

- Fix for invalid cluster state on nodes failure.

- ASTERIXDB-139: Fix for cleaning workspace files on startup/shutdown.

- Fix for temp datasets storage reclamation.

- Allow MetadataNode rebinding with CC.

- Add flag to checkpoint to identify sharp checkpoints.

- ASTERIXDB-1170: Fix shutdown sequence

Change-Id: I729fdd1144dbc9ff039b4bc414494860d7553810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/338

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 72 more files in changeset.
ASTERIXDB-1058: use ResourceHeapBufferAllocator for dataset memory components.

Change-Id: Ifd90fabc79e61f84370d415c38917b998db41466

Reviewed-on: https://asterix-gerrit.ics.uci.edu/481

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 11 more files in changeset.
ASTERIXDB-1058: make Asterix compatible with lazy LSM memory allocation

- Adapt memory budget calculation to lazy LSM memory allocation.

- Add IDatasetLifecycleManager interface.

Change-Id: I4ea1eb129fe3043d43b077473dc29d17a97dfcc2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/408

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 30 more files in changeset.
This change fixes the sporadic connection refused errors

This can mostly be seen during asterix-installer tests.

The fix basically lets managix wait and listens

through zookeeper for a signal from the cluster controller that

the cluster is up and ready.

Once the cluster controller sends the signal, Managix can proceed.

Change-Id: Ib730f50ab2fb492f3cf973d1cf2f03b34e24e5b3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/366

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -3
    • +6
    ./metadata/cluster/ClusterManager.java
  1. … 11 more files in changeset.
ASTERIXDB-1102: VarSize Encoding to store length of String and ByteArray

This patch is to change the encoding format that stores the length value

of

the variable length type (e.g. String, ByteArray) from fix-size encoding

(2bytes) to variable-size encoding ( 1 to 5bytes)

It will solve the issue 1102 to enable us to store a String that longer

than 64K. Also for the common case of storing the short string ( <=

127), it will save one byte per string.

Some important changes include:

1. The UTF8StringSerDer and ByteArraySerDer is not Singleton instance

any more. I need some state to speedup the serialization and avoid the

object creatation. Luckily, 99% percent of Serializer were used as

factory way. The other 1% has been fixed.

A separate Test support, the ExcutionTest now can produce the only.xml

which stores the previous failed runtime test.xml. It can speedup the

debug process.

Change-Id: I41fff780f5c071742ef10129d83c8f945d5886d7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/450

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

    • -3
    • +2
    ./metadata/external/ExternalLoopkupOperatorDiscriptor.java
  1. … 317 more files in changeset.
prevented secondary index insert operator in feed from dropping frames

Change-Id: I5a22b7a56e476b0cb4535ba4323419ef7eb69fb1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/446

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +7
    ./metadata/feeds/FeedMetaStoreNodePushable.java
  1. … 2 more files in changeset.
ASTERIXDB-1053: Allow lazy loading for persistent local resources

- Allow indexes to be accessed by name.

- Allow lazy loading for persistent local resources.

- Caching for local resources.

Change-Id: I48b9260a3280750145f6ddb3783673a299055910

Reviewed-on: https://asterix-gerrit.ics.uci.edu/344

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 14 more files in changeset.