Clone Tools
  • last updated 11 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Control Number of Readers for LocalFS Data

Change-Id: Ib9d5ece656220d5f562cc385f882c5ddfd3283a6

Reviewed-on: https://asterix-gerrit.ics.uci.edu/776

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -1
    • +1
    ./org/apache/asterix/file/DataverseOperations.java
  1. … 23 more files in changeset.
Exclude Temporary Resources From Replication

- Exclude temporary resources from replication.

- Remove flush logs from temporary datasets.

- Ignore takeover partitions request if NC is shutting down.

- Stop NCs on different threads to allow replica shutting down

notification to be sent when replication is enabled.

Change-Id: I9a52557bf1f3e7632dd826384280abdaa186f672

Reviewed-on: https://asterix-gerrit.ics.uci.edu/778

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 5 more files in changeset.
ASTERIXDB-1378 Fix NPE on Feed Connect After Restart

This issue was caused by the way type traits are computed for each

index. since we have duplicate code to create the index dataflow

helper, we had to fix the way type traits are created in all of

these different places. We propably need to do further refactoring

to have common code for creating index dataflow helper instances.

Change-Id: If8f8696d252868a8cce0afdbaeda0dd046f99186

Reviewed-on: https://asterix-gerrit.ics.uci.edu/766

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 17 more files in changeset.
ASTERIXDB-1360 Fix Error Message for Unknown Format

This change includes some refactoring for external

data. This refactoring makes the code less error prone

and separate data source selection from data parser

selection. It also fixes issue ASTERIXDB-1366 and adds

a test case for it as well.

Change-Id: Ib4aac833e30bd7c5a7706f5c8116383c2362c964

Reviewed-on: https://asterix-gerrit.ics.uci.edu/767

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 33 more files in changeset.
Move AsterixDB to Subfolder

    • -51
    • +0
    ./org/apache/asterix/api/common/Job.java
  1. … 44144 more files in changeset.
Improve Error Handling in Local Directory Feeds

This change improves handling of two error types for filesystem

based feeds. The first one is the handling of IO Errors which

causes the input stream to be closed, and the second one is

reacting to missed filesystem events. In both cases, we scan the

directory and compare it with the history we have in order to

resume from where we last left off.

In addition, this change includes some refactoring in external

data. Particularly, we get rid of the stream provider layer and

instead, stream factories create input streams directly. This

is consistent with record reader factories which create readers

directly without reader providers.

Change-Id: I08d89229e33c91532b1038ba9f7a372f7ca1fdb5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/720

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 143 more files in changeset.
ASTERIXDB-1361: Meaningful error when missing output-record-type

Change-Id: Id5ca7d634b2be5164af82ad128ff577c0bdbd457

Reviewed-on: https://asterix-gerrit.ics.uci.edu/741

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 3 more files in changeset.
add parameter to QueryServiceServlet to indent JSON

Change-Id: Id197d3ad6aa17c9a36bb3845bd3ca75a695ba6d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/750

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Fixed Feed Connect Statement

This change includes two fixes:

1. Feed connect doesn't return until the connection is complete.

2. When using wait for completion, it waits until all the jobs

complete.

Change-Id: I416bf4917b1f5cea687d1202c435f7183136cf1f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/726

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 36 more files in changeset.
Allow Replication to be Enabled on Virtual Cluster

- Allow replication port assignment per NC.

- Allow replication to be enabled on virtual cluster.

- Wait for JOB_ABORT ACK from remote replicas.

- Fix LSM component mask file name.

- Fix index directory deletion on index drop.

- Eliminate multiple partition takeover requests.

- Free LogFlusher thread from sending replication ACKs.

- Fix possible deadlock between LogFlusher and Logs Replication Thread.

- Remove wait for FLUSH_LOG for replicated LSM components:

This wait is not needed since on node failure, complete remote recovery is done.

Change-Id: I34a38f59c4915a19242adb6a4eaa6ee1c82d2372

Reviewed-on: https://asterix-gerrit.ics.uci.edu/743

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 19 more files in changeset.
ASTERIXDB-1343: support queries over nodegroup-based datasets.

Fixed operators to have the right NodeDomain.

Added several regerssion tests.

Change-Id: I776b80e5b824c83bcdf43c95fff99bb151506f84

Reviewed-on: https://asterix-gerrit.ics.uci.edu/719

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 36 more files in changeset.
Support Change Feeds and Ingestion of Records with MetaData

This change allows feeds to perform upserts and deletes

in order to perform replication of an external data source.

The change does so by performing the following:

1. The adapter produces [PK][Record]. (Record == null --> delete)

2. The insert is replaced by an upsert operator.

Change-Id: If136a03d424970132dfb09f0dda56e160d4c0078

Reviewed-on: https://asterix-gerrit.ics.uci.edu/621

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 268 more files in changeset.
Eliminate Duplicate Hyracks Classes From AsterixDB

The following hyracks classes (package org/apache/hyracks/dataflow/std/

misc) are duplicated in asterixdb repo:

- ConstantTupleSourceOperatorDescriptor

- ConstantTupleSourceOperatorNodePushable

This eliminates the copies in asterixdb repo.

Change-Id: I64bded72def25fa34ea1545d4bf0f01218d04b9f

Reviewed-on: https://asterix-gerrit.ics.uci.edu/722

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Coverity Fixes

FeedMessageOperatorNodePushable

- CID 68721 (#1 of 1): Useless call (USELESS_CALL)

FeedLoadManager

- CID 68719 (#1 of 1): Useless call (USELESS_CALL)

RuntimeRecordTypeInfo

- CID 68717 (#1 of 1): Unused value (UNUSED_VALUE)

HelpCommand

- CID 68591 (#1 of 1): Missing break in switch (MISSING_BREAK)

- CID 68716 (#1 of 1): Unused value (UNUSED_VALUE)

SimilarityJaccardEvaluator

- CID 68714 (#1 of 1): Arguments in wrong order (SWAPPED_ARGUMENTS)

DatasetLifecycleManager

- CID 68713 (#1 of 1): Dereference before null check (REVERSE_INULL)

FeedMetaComputeNodePushable

- CID 68712 (#1 of 2): Dereference before null check (REVERSE_INULL)

- CID 68712 (#2 of 2): Dereference before null check (REVERSE_INULL)

FeedRuntimeInputHandler

- CID 68709 (#1 of 1): Dereference before null check (REVERSE_INULL)

DataflowControllerProvider

- CID 68708 (#1 of 1): Resource leak on an exceptional path

(RESOURCE_LEAK)

TokenLoad

- CID 68697 (#1 of 1): Resource leak (RESOURCE_LEAK)

RecordGroupLengthRange

- CID 68696 (#1 of 1): Resource leak (RESOURCE_LEAK)

RecordManagerGeneratorMojo

- CID 68691 (#1 of 1): Resource leak (RESOURCE_LEAK)

APIServlet

- CID 68687 (#1 of 1): Resource leak (RESOURCE_LEAK)

Change-Id: I8927126f505481e83dd5b9e6e5ccd75bb67d8188

Reviewed-on: https://asterix-gerrit.ics.uci.edu/715

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 11 more files in changeset.
- Reapplied fix for ASTERIXDB-1109 on merged master - Cleaned up runtime testsuite, made it validate against XSD

Change-Id: I8fcf41c009f9faf51bc8bccc0c3c7217b7769121

Reviewed-on: https://asterix-gerrit.ics.uci.edu/712

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@apache.org>

  1. … 26 more files in changeset.
ASTERIXDB-1337: Dataset Memory Management on Multi-Partition NC

As sugggested in ASTERIXDB-1337, this change is to maintain a per-

partition MultitenantVirtualBufferCache budget, as opposed to sharing

the budget across the dataset.

Change-Id: Ibbf08f532c1210c30be6a51c73570a789174213b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/705

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 16 more files in changeset.
ASTERIXDB-1340: fix node group nodes order

Change-Id: I6e93aedb6cdc57d98f83991ab910476b82bacb15

Reviewed-on: https://asterix-gerrit.ics.uci.edu/708

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 3 more files in changeset.
Revert "ASTERIXDB-1109: Fixed deletion of records from open secondary index"

This reverts commit a05f71826a4ab2c0e7148009e0b7e269b15e0a2f.

Change-Id: I8fcf41c009f9faf51bc8bccc0c3c7217b7769148

Reviewed-on: https://asterix-gerrit.ics.uci.edu/698

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

    • -157
    • +177
    ./org/apache/asterix/aql/translator/AqlTranslator.java
  1. … 37 more files in changeset.
patch 8

  1. … 73 more files in changeset.
Supprt querying meta record with meta().

Change-Id: Ie65417b6baf209ca0ab413cfa4a5f7fc5156ca63

Reviewed-on: https://asterix-gerrit.ics.uci.edu/685

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 63 more files in changeset.
Add an integer ordered list to dataset "Dataset" and "Index" optinally.

For the case the primary or secondary index key contains fields from the meta

record associated with a dataset record, we need an integer ordered

list to indicate which record a key field comes from.

Change-Id: I979c642ebd60e53213369f1e2070146b4c26e805

Reviewed-on: https://asterix-gerrit.ics.uci.edu/683

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 50 more files in changeset.
Add DDL support for optionally associating meta record with dataset record.

-Fixed the TestExecutor.runScriptAndCompareWithResult() to compare fields

after "Time"-prefixed fields.

Change-Id: I613316985dbd7f553d946f95c5b1a1df56914902

Reviewed-on: https://asterix-gerrit.ics.uci.edu/666

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 72 more files in changeset.
ASTERIXDB-1302 ASTERIXDB-1301 Fix Socket Feed Connection

A bug causes a read lock to never be released when a feed is

connected with "wait-for-completion" set to false. The bug

was fixed and a test case was added.

Another bug was causing the socket feed to not receive

connections correctly. The bug was fixed and a test case

was added.

Additionally, this change ensures that adapters have absolute

partitions to ensure consistency with regards to feed log

manager.

Change-Id: I8f6e982440d3577343f2479c3779653a9c3db614

Reviewed-on: https://asterix-gerrit.ics.uci.edu/660

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>

  1. … 47 more files in changeset.
Enabled Feed Tests and Added External Library tests

Feed tests had been switched off for a while due to having too many

sporadic failures. Now, we are switching them back on.

In addition, a new set of tests have been added to test that external

library works as expected.

Change-Id: Idd1fccd136fa2645b2707bbf7c04e60991ae8d4a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/625

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +110
    ./org/apache/asterix/app/external/CentralFeedManager.java
    • -0
    • +402
    ./org/apache/asterix/app/external/ExternalLibraryUtils.java
    • -0
    • +190
    ./org/apache/asterix/app/external/FeedJoint.java
    • -0
    • +301
    ./org/apache/asterix/app/external/FeedLoadManager.java
    • -0
    • +96
    ./org/apache/asterix/app/external/FeedMessageReceiver.java
    • -0
    • +254
    ./org/apache/asterix/app/external/FeedOperations.java
    • -0
    • +187
    ./org/apache/asterix/app/external/FeedTrackingManager.java
    • -0
    • +206
    ./org/apache/asterix/app/external/FeedWorkCollection.java
  1. … 190 more files in changeset.
Remove Trailing Whitespace

This change removes all trailing whitespace in

java files in asterixdb

Change-Id: I32cda384f71dd5237a3c63e83c661f1cf64dc0eb

Reviewed-on: https://asterix-gerrit.ics.uci.edu/646

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -1
    • +1
    ./org/apache/asterix/drivers/AsterixWebServer.java
    • -1
    • +1
    ./org/apache/asterix/feed/FeedTrackingManager.java
  1. … 318 more files in changeset.
Asterix NCs Failback Support

- Allow Failed NCs to failback and takeover their partitions.

- New cluster API servlet for cluster state description.

- Remove nodeId from txn logs except remote FLUSH_LOG.

- Add partition id in UPDATE and Entity_COMMIT logs.

- Adapt remote recovery to new logs format.

- Refactor RecoveryManager and split Analysis and Redo phase.

- Spill remote recover logs to temporary file.

- Replicate files to remote replicas for partitions of interest only.

- Introduce NC active/inactive partitions concept.

- Test case for failback.

Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24

Reviewed-on: https://asterix-gerrit.ics.uci.edu/613

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +51
    ./org/apache/asterix/api/http/servlet/ClusterAPIServlet.java
  1. … 86 more files in changeset.
Move to non-copy-based evaluator interfaces for all function implementations, including: - scalar functions, - aggregate functions, - running aggregate functions, - unnesting functions

Change-Id: If132857cf8d707ce96951f12ba6dd86f54aa4069

Reviewed-on: https://asterix-gerrit.ics.uci.edu/615

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 313 more files in changeset.
Add Support for Upsert Operation

This change allows users to execute upsert commands which

couples delete if found with insert. It locks the primary

keys before doing the search ensuring consistency.

Change-Id: I8999000331795a5949d621d2dd003903e057a521

Reviewed-on: https://asterix-gerrit.ics.uci.edu/477

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 250 more files in changeset.
Add flush() to IFrameWriter

This method is expected to be used with feeds to push

frames all the way to storage when needed. As of now, it is

needed in two cases:

1. No activities in ingestion node and need to push content

so it can be stored.

2. When the ingestion node needs to move the checkpoint ahead

if the at least once semantics are used.

Two feeds make use of this function. The filesystem feed and

couchbase feed which was introduced as well in this change.

Change-Id: Id862ce9e9b1360864c6976f2aea2137092f51203

Reviewed-on: https://asterix-gerrit.ics.uci.edu/585

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -1
    • +6
    ./org/apache/asterix/feed/FeedMessageReceiver.java
  1. … 102 more files in changeset.
remove end-of-line whitespace

Change-Id: I5c0415f47d4c3a9827574fbdab949b45718d9ea4

Reviewed-on: https://asterix-gerrit.ics.uci.edu/601

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 136 more files in changeset.