Clone Tools
  • last updated 21 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
ASTERIXDB-1378 Fix NPE on Feed Connect After Restart

This issue was caused by the way type traits are computed for each

index. since we have duplicate code to create the index dataflow

helper, we had to fix the way type traits are created in all of

these different places. We propably need to do further refactoring

to have common code for creating index dataflow helper instances.

Change-Id: If8f8696d252868a8cce0afdbaeda0dd046f99186

Reviewed-on: https://asterix-gerrit.ics.uci.edu/766

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 17 more files in changeset.
Improve Error Handling in Local Directory Feeds

This change improves handling of two error types for filesystem

based feeds. The first one is the handling of IO Errors which

causes the input stream to be closed, and the second one is

reacting to missed filesystem events. In both cases, we scan the

directory and compare it with the history we have in order to

resume from where we last left off.

In addition, this change includes some refactoring in external

data. Particularly, we get rid of the stream provider layer and

instead, stream factories create input streams directly. This

is consistent with record reader factories which create readers

directly without reader providers.

Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/720

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 143 more files in changeset.
ASTERIXDB-1233: Fixed the bulk-loading with an inverted index on an open-type field

Change-Id: If58f594c0a7b6f4bca45b13ceaef07b605d2fe22

Reviewed-on: https://asterix-gerrit.ics.uci.edu/740

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

  1. … 5 more files in changeset.
Merge branch 'release-0.8.8' into master

Change-Id: I8fcf41c009f9faf51bc8bccc0c3c7217b7769149

  1. … 36 more files in changeset.
ASTERIXDB-1343: support queries over nodegroup-based datasets.

Fixed operators to have the right NodeDomain.

Added several regerssion tests.

Change-Id: I776b80e5b824c83bcdf43c95fff99bb151506f84

Reviewed-on: https://asterix-gerrit.ics.uci.edu/719

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 32 more files in changeset.
Support Change Feeds and Ingestion of Records with MetaData

This change allows feeds to perform upserts and deletes

in order to perform replication of an external data source.

The change does so by performing the following:

1. The adapter produces [PK][Record]. (Record == null --> delete)

2. The insert is replaced by an upsert operator.

Change-Id: If136a03d424970132dfb09f0dda56e160d4c0078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/621

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 268 more files in changeset.
ASTERIXDB-1168: use either primary key or generated id for decorrelation.

Change-Id: Ib5e232f3ba99018cd1aedfa4f8bb2f98affa0f55

Reviewed-on: https://asterix-gerrit.ics.uci.edu/696

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 31 more files in changeset.
Supprt querying meta record with meta().

Change-Id: Ie65417b6baf209ca0ab413cfa4a5f7fc5156ca63

Reviewed-on: https://asterix-gerrit.ics.uci.edu/685

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 62 more files in changeset.
Add DDL support for optionally associating meta record with dataset record.

-Fixed the TestExecutor.runScriptAndCompareWithResult() to compare fields

after "Time"-prefixed fields.

Change-Id: I613316985dbd7f553d946f95c5b1a1df56914902

Reviewed-on: https://asterix-gerrit.ics.uci.edu/666

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 68 more files in changeset.
remove unused member

Change-Id: I8d0660faacc39aa3a0fca50f4813669a0179c408

Reviewed-on: https://asterix-gerrit.ics.uci.edu/653

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Enabled Feed Tests and Added External Library tests

Feed tests had been switched off for a while due to having too many

sporadic failures. Now, we are switching them back on.

In addition, a new set of tests have been added to test that external

library works as expected.

Change-Id: Idd1fccd136fa2645b2707bbf7c04e60991ae8d4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/625

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 204 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 326 more files in changeset.
Add Support for Upsert Operation

This change allows users to execute upsert commands which

couples delete if found with insert. It locks the primary

keys before doing the search ensuring consistency.

Change-Id: I8999000331795a5949d621d2dd003903e057a521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 251 more files in changeset.
Add flush() to IFrameWriter

This method is expected to be used with feeds to push

frames all the way to storage when needed. As of now, it is

needed in two cases:

1. No activities in ingestion node and need to push content

so it can be stored.

2. When the ingestion node needs to move the checkpoint ahead

if the at least once semantics are used.

Two feeds make use of this function. The filesystem feed and

couchbase feed which was introduced as well in this change.

Change-Id: Id862ce9e9b1360864c6976f2aea2137092f51203

Reviewed-on: https://asterix-gerrit.ics.uci.edu/585

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 103 more files in changeset.
Feed Fixes and Cleanup

1. Introduce filesystem feed data source.

2. Fix the order of closing feed stages on disconnection.

3. Added Twitter feed to the compatibility utility to allow

using it with its alias.

4. First part of the feed log space.

5. Fixed the handling of duplicate key exception.

Change-Id: I4e8db26a810efd1fbaa52ceeb3efd0c8328ab070

Reviewed-on: https://asterix-gerrit.ics.uci.edu/574

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 416 more files in changeset.
Enabled Datasets to use Datatypes from foreign Dataverses

Removed broken Metadata Secondary Indexes

Added DatatypeDataverse field to Dataset

Cleaned up Metadata Tests

Added tests for dropping used Datatypes

User-facing changes are:

1) Allow qualified datatype when creating dataset

2) Metadata queries for datasets will now

include the type dataverse

Change-Id: I24dbc04dcb2a4126fc8361ebe3104877a0d1f2bb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/558

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 139 more files in changeset.
some cleanup

- remove redundant code around DataSources

- fix typo in constant name

Change-Id: I48602e1acbbc22642d4467ab37f27f337de20c5e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/583

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 3 more files in changeset.
First stage of external data cleanup

In this change, different parts of external data were refactored.

The goal was to make it more modular, easier to maintain and allow

higher flexibility for extension in addition to reducing code redundancy.

Change-Id: I04a8c4e494d8d1363992b6fe0bdbe6b2b3b7b767

Reviewed-on: https://asterix-gerrit.ics.uci.edu/566

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -86
    • +0
    ./FieldExtractingAdapterFactory.java
  1. … 339 more files in changeset.
Divide Cluster into Unique Partitions

The change includes the following:

- Fix passing NC stores to AsterixConfiguration.

- Unify storage direcotry name in the instance level rather than the node level.

- Divide the cluster into unique storage partitions based on the number of stores.

- Refactored FileSplits and moved out of AqlMetadataProvider.

- Make AsterixHyracksIntegrationUtil use the passed configuration file.

- Make File Splits pass relative index paths of partitions rather than absolute paths.

- Remove unused AqlCompiledMetadataDeclarations class.

Change-Id: I8c7fbca5113dd7ad569a46dfa2591addb5bf8655

Reviewed-on: https://asterix-gerrit.ics.uci.edu/564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -276
    • +0
    ./AqlCompiledMetadataDeclarations.java
    • -15
    • +1
    ./AqlLogicalPlanAndMetadataImpl.java
  1. … 39 more files in changeset.
Refactored External Data

This change re-arrange asterix module's order. asterix-

external-data is moved in front of asterix-metadata.

Change-Id: I46b60b5e1cc37fd59adc0dd89f374d96502091b2

Reviewed-on: https://asterix-gerrit.ics.uci.edu/559

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -159
    • +243
    ./AqlMetadataProvider.java
  1. … 91 more files in changeset.
ASTERIXDB-1126 Correct comparators and type traits for external index

In case of external data, the type definition does not contain information

about primary keys and we should get them based on the data input format.

Change-Id: I71d924d7e2b7a7e6c752bc97679e612946afc17c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/504

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 15 more files in changeset.
Introducing Data Replication To AsterixDB

This change includes the following:

- Add data replication properties to cluster properties and Managix validate command.

- Introduce Data Replication components.

- Add data replication required fields to LogRecord.

- Specialized LogManager for data replication.

- Fix for invalid cluster state on nodes failure.

- ASTERIXDB-139: Fix for cleaning workspace files on startup/shutdown.

- Fix for temp datasets storage reclamation.

- Allow MetadataNode rebinding with CC.

- Add flag to checkpoint to identify sharp checkpoints.

- ASTERIXDB-1170: Fix shutdown sequence

Change-Id: I729fdd1144dbc9ff039b4bc414494860d7553810

Reviewed-on: https://asterix-gerrit.ics.uci.edu/338

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Young-Seok Kim <kisskys@gmail.com>

  1. … 73 more files in changeset.
ASTERIXDB-1115 fixed: Incorrect comparators are given for a secondary btree index when AsterixDB instance with a preloaded/inserted data stop/starts and sends a query without having any dml operation

Note: this fix is created by Young-Seok Kim and submitted by Taewoo Kim on behalf of him.

Change-Id: I648d808b77910582a6b52a0e730d7271f9d2c213

Reviewed-on: https://asterix-gerrit.ics.uci.edu/412

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

Change License Headers

Also tweak the NOTICE file with some extras.

Change-Id:I09bc388089e515d7f51fd39c31bfbbc9f00cf84f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/388

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -10
    • +14
    ./AqlCompiledMetadataDeclarations.java
    • -10
    • +14
    ./AqlLogicalPlanAndMetadataImpl.java
    • -10
    • +14
    ./FieldExtractingAdapterFactory.java
  1. … 2004 more files in changeset.
Change Java package from edu.uci.ics to org.apache

Change-Id: I2f01d2b5614e9e9c94fda4bf1294a8eba6a26c54

Reviewed-on: https://asterix-gerrit.ics.uci.edu/309

Reviewed-by: Till Westmann <tillw@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -26
    • +26
    ./AqlCompiledMetadataDeclarations.java
    • -163
    • +163
    ./AqlMetadataProvider.java
  1. … 2590 more files in changeset.
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: Iefd2a576415ebc1416cba2a3334d2b64f042ba92

Reviewed-on: https://asterix-gerrit.ics.uci.edu/306

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +272
    ./AqlCompiledMetadataDeclarations.java
    • -0
    • +205
    ./AqlDataSource.java
    • -0
    • +60
    ./AqlLogicalPlanAndMetadataImpl.java
    • -0
    • +28
    ./AqlMetadataImplConfig.java
    • -0
    • +2367
    ./AqlMetadataProvider.java
    • -0
    • +120
    ./DatasetDataSource.java
    • -0
    • +124
    ./FeedDataSource.java
    • -0
    • +166
    ./FieldExtractingAdapter.java
    • -0
    • +82
    ./FieldExtractingAdapterFactory.java
    • -0
    • +46
    ./FileSplitDataSink.java
    • -0
    • +37
    ./FileSplitSinkId.java
    • -0
    • +138
    ./LoadableDataSource.java
    • -0
    • +180
    ./PKGeneratingAdapter.java
  1. … 3781 more files in changeset.