Clone Tools
  • last updated 26 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE] Support multipart dataverse names

- user model changes: yes

- storage format changes: no

- interface changes: yes

Details:

- Support dataverse names that consist of multiple parts

- Introduce DataverseName class and adopt it throughout the system

- Add testcases for multipart dataverse names

Change-Id: I1313b2dc0e8df6a9b0ded48dea122afc3ba5dbe7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/4004

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 284 more files in changeset.
[NO ISSUE][RT] Fix Secondary Indexes Bulkload/Upsert

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Prevent null values from being upserted into

secondary indexes.

- Use secondary index tuple filter when bulkloading

to prevent null/missing values from being inserted

into indexes of optional fields.

Change-Id: I7b847aece06a4387cc41389b0566d0c0dc98487d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3304

Reviewed-by: Luo Chen <cluo8@uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -2
    • +6
    ./ExternalIndexBulkLoadOperatorDescriptor.java
    • -3
    • +4
    ./ExternalIndexBulkLoadOperatorNodePushable.java
    • -3
    • +7
    ./ExternalIndexBulkModifyOperatorDescriptor.java
    • -2
    • +4
    ./ExternalIndexBulkModifyOperatorNodePushable.java
  1. … 8 more files in changeset.
[NO ISSUE][ING] Add timeout to stop active message

Change-Id: Ie0416d76670e945cb958f5a1c235201a3e016009

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2762

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 9 more files in changeset.
[ASTERIXDB-2393][COMP][RT] Add source location to error messages

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Add source locations to compiler and runtime error messages

- Add source location enforcement to the test framework

and enable it for SqlppExecutionTest

Change-Id: Ie279ce345d1edcb5dea8e55cdb0233151c7bfd66

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2659

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -1
    • +1
    ./ExternalFilesIndexModificationOperatorDescriptor.java
  1. … 680 more files in changeset.
[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- This change introduces some improvements to storage

operations.

- Local RecoveryManager is now extensible.

- Bulk loaders now call the IO callback similar to

Flushes, making them less special and creating a

unified lifecycle for adding an index component.

- As a result, The IndexCheckpointManager doesn't need

to have a special treatment for components loaded

through the bulk load operation.

- Component Id have been added to the index checkpoint

files.

- Cleanup for the code of local recovery for failed flush

operations.

- Ensure that after local recovery of flushes, primary

and secondary indexes have the same index for mutable

memory component.

- The use of WAIT logs to ensure in-flight flushes

are scheduled didn't work as expected. A new log type

WAIT_FOR_FLUSHES was introduced to acheive the expected

behavior.

- The local test framework was made Extensible to support

more use cases.

- Test cases were added for component ids in checkpoint files.

The following scenarios were covered:

- Primary and secondary both have values when a flush is

shceduled.

- Primary have values but not secondary when a flush is

scheduled.

- Primary is empty and an index is created through bulk

load.

- Primary has a single component and secondary is created

through bulk load.

- Primary has multiple components and secondary is created

through bulk load.

- Each primary opTracker now keeps a list of ongoing flushes.

- FlushDataset now waits only for flushes only and

not all io operations.

- Previously, we had many flushes scheduled on open datasets.

This was not detected but after this change, a failure

is thrown in such cases.

- Flush operations dont need to extend the comparable

interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2632

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -3
    • +10
    ./ExternalFilesIndexCreateOperatorDescriptor.java
    • -1
    • +4
    ./ExternalFilesIndexModificationOperatorDescriptor.java
    • -0
    • +14
    ./ExternalIndexBulkLoadOperatorNodePushable.java
    • -2
    • +5
    ./ExternalIndexBulkModifyOperatorNodePushable.java
  1. … 86 more files in changeset.
[NO ISSUE][TRACE] Cleanups for performance tracer

- user model changes: no

- storage format changes: no

- interface changes: yes

Details:

- Move all Trace categories into a single place.

- Cleanup the interface and add some javadocs.

- Performance optimizations when logger is used.

- Add a new storage log for last record in.

Change-Id: Ib65edd6e7093dfd6f94ef750c484501b92c7a8a0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2558

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 8 more files in changeset.
[NO ISSUE] Incremental cleanup of deprecated exception ctors

Change-Id: I1e7c3655828fc6530cef83ea502a6cfbf41acddf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2533

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -1
    • +1
    ./AbstractExternalDatasetIndexesOperatorDescriptor.java
    • -1
    • +1
    ./ExternalBTreeSearchOperatorNodePushable.java
    • -1
    • +1
    ./ExternalIndexBulkModifyOperatorNodePushable.java
    • -3
    • +3
    ./ExternalLookupOperatorDescriptor.java
    • -2
    • +2
    ./ExternalRTreeSearchOperatorNodePushable.java
    • -1
    • +1
    ./FeedCollectOperatorNodePushable.java
  1. … 232 more files in changeset.
[NO ISSUE][ING] Follow the IFrameWriter contract in Feed Pipeline

- user model changes: no

- storage format changes: no

- interface changes: yes

- it removes unneeded ITupleForwarder interface.

Details:

- Previously, if a failure happens at the writer.open

call in the FeedIntake operator, the IFrameWriter

protocol is not followed since we don't fail before

close is called.

- Previously, fail calls can be done as well inside

of the feed adapter.

- This change moves failing the pipeline outside the

adapter and so the adapter's responsibility is

getting records from outside the system while

the operator takes care of opening, failing, and

closing the writer.

Change-Id: Ife679fb9643dc6b39d035e0eecdb915b227503a5

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2463

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +10
    ./FeedIntakeOperatorNodePushable.java
  1. … 22 more files in changeset.
[ASTERIXDB-2229][OTR] Restore Thread Names in Thread Pool

- user model changes: no

- storage format changes: no

- interface changes: no

- Restore thread names to their original names

before returning them to the pool of the NC

app.

- Remove explicit thread name resets.

- Delete unused ThreadExecutor class.

Change-Id: I3bda1b65e7aefd35d2b8cfa814f73369c3bf5a18

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2447

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 9 more files in changeset.
[NO ISSUE][ING] Close and return on interrupt of ingestion

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Previously, we would still attempt to write through the network

in case an ingestion task is interrupted.

- The goal was to try and get as much data in as possible but in

the case where the cluster was in a bad state, this could

lead to hanging threads.

- After this change, each record reader must implement the stop

method correctly to allow for graceful stop while interrupts

will always mean abort the task and return as soon as possible.

Change-Id: I6119617d133fb161a48b39f9812ec79e0189975b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2428

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 4 more files in changeset.
[ASTERIXDB-2182][ING] Use JSON-like syntax for feed configuration

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

To be consistent with the other JSON-like syntax in defining

configurations, CREATE FEED will change to use JSON object instead of

key-value pairs. Also, the <USING> adapter_name syntax is removed. The

adapter name becomes as part of the configuration. After this change,

the feed configuration are specified as:

<CREATE> <FEED> testFeed <WITH> {

"adapter-name" : "localfs",

"path" : "localhost://../../../../../tweets.json",

"format" : "adm",

"type-name" : "typeTweet"

};

This patch also fixes two minor mistakes when drop feed and adapter in

Metadata.

Change-Id: I2dfb0fdf74d28ae0c45dafcf6babb624bd8b2149

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2209

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 107 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +3
    ./ExternalDatasetIndexesCommitOperatorDescriptor.java
  1. … 386 more files in changeset.
[ASTERIXDB-2132] Replace ITracer.check

- user model changes: no

- storage format changes: no

- interface changes: yes

Trace categories in the ITracer interface are represented by long

values where each bit represents a category.

Introduce ITraceCategoryRegistry, an interface for a registry/mapping

from Strings to bit positions representing categories.

- add simple tests for Tracer and TraceCategoryRegistry

Change-Id: I3f26ae988d3786ef4ddb17e8fe78f911e7e628f1

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2087

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 17 more files in changeset.
[NO ISSUE][OTH] Trace ingestion performance at Feed Store

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Trace the ingestion performance at store side

Change-Id: I80f41429d23cb46385e6672bfd59715249c3b7ba

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2074

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 1 more file in changeset.
[ASTERIXDB-2064][ING] Timeout Stop Feed

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- The abort feed message stops the reader and wait for the

dataflow controller to signal end of life.

- If the reader returns true to stop but the dataflow controller

never signal ends, it can get stuck.

- This change adds a timeout after which, the task thread is

interrupted.

Change-Id: If609a8343767ee7a80689a58af35a1b3fca2964b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1964

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +16
    ./FeedIntakeOperatorNodePushable.java
  1. … 7 more files in changeset.
[ASTERIXDB-2008][CLUS] Only add pending removal if node known

[ASTERIXDB-2023][ING] Introduce Enums instead of using bytes

- user model changes: no

- storage format changes: no

- interface changes: no

details:

- Only nodes which are known to cluster manager are added

to the list of nodes pending removal. Other nodes are ignored

- Enums introduced:

- ActiveEvent.Kind

- ActivePartitionMessage.Event

- Remove AdapterRuntimeManager

- Remove AdapterExecutor

Change-Id: I7044896559798426c04a3f46861bc5335b25d140

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -1
    • +1
    ./ExternalIndexBulkModifyOperatorNodePushable.java
    • -2
    • +5
    ./ExternalLookupOperatorDescriptor.java
    • -38
    • +60
    ./FeedIntakeOperatorNodePushable.java
  1. … 99 more files in changeset.
[ASTERIXDB-1950][ING][API] REST API for ActiveEntity stats

- user model changes: yes

Added ActiveEntity stats REST API.

- storage format changes: no

- interface changes: yes

Changed ActiveEntityEventsListener & EventSubscrber interfaces.

Added getStats method to IActiveRuntime.

Details:

1. Added HttpAPI for active feed stats.

2. Replaced FeedEventSubscriber with WaitForStateSubscriber.

3. Added StatsSubscriber for monitoring stats request.

4. Moved the message related methods from FeedEventsListener to

ActiveEntityEventsListener for possible reuses in other cases.

Change-Id: I46b48b52a1c9906510c5bdce778d1672845f75ca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1748

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 41 more files in changeset.
Change DataflowHelperFactory not to require Task Context

Change-Id: I9dcd95dbefca131c4bbdb43306f00f6f8ea60800

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1758

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -2
    • +2
    ./AbstractExternalDatasetIndexesOperatorDescriptor.java
    • -1
    • +2
    ./ExternalFilesIndexCreateOperatorDescriptor.java
    • -1
    • +2
    ./ExternalFilesIndexModificationOperatorDescriptor.java
    • -1
    • +2
    ./ExternalLookupOperatorDescriptor.java
  1. … 16 more files in changeset.
Rename TaskUtil.getInSharedMap(..) -> TaskUtil.get(..).

Change-Id: Ic795c988ca63db7c760a2c8e7638451945cb72aa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1817

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 6 more files in changeset.
Fix upsert deadlock and upsert with filtered primary only

This change fixes a deadlock that happens when 3 operations

an upsert, a search and a flush happen simulteniously.

If all the memory components are full, the upsert

gets blocked, the upsert could've obtained a lock on the

search key which would block the search not allowing it

to exit the components and not allowing the components

to be cleared and reused.

In addition, the change refactors common LSM index code.

Change-Id: I93fac0f27ab0b3cc071ff38aef90d850cbbce488

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1762

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -1
    • +1
    ./ExternalFilesIndexModificationOperatorDescriptor.java
    • -4
    • +3
    ./ExternalIndexBulkModifyOperatorDescriptor.java
    • -6
    • +5
    ./ExternalIndexBulkModifyOperatorNodePushable.java
  1. … 84 more files in changeset.
Add Error code for unknown pending operation

Change-Id: I41d0bed5afdfee9b2c29d0218e794247f13e6350

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1739

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -3
    • +3
    ./ExternalFilesIndexModificationOperatorDescriptor.java
  1. … 2 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -21
    • +7
    ./AbstractExternalDatasetIndexesOperatorDescriptor.java
    • -22
    • +15
    ./ExternalBTreeSearchOperatorDescriptor.java
    • -13
    • +19
    ./ExternalBTreeSearchOperatorNodePushable.java
    • -10
    • +11
    ./ExternalDatasetIndexesAbortOperatorDescriptor.java
    • -16
    • +11
    ./ExternalDatasetIndexesCommitOperatorDescriptor.java
    • -11
    • +11
    ./ExternalDatasetIndexesRecoverOperatorDescriptor.java
    • -0
    • +111
    ./ExternalFilesIndexCreateOperatorDescriptor.java
    • -0
    • +124
    ./ExternalFilesIndexModificationOperatorDescriptor.java
    • -162
    • +0
    ./ExternalFilesIndexOperatorDescriptor.java
    • -0
    • +51
    ./ExternalIndexBulkLoadOperatorDescriptor.java
    • -0
    • +46
    ./ExternalIndexBulkLoadOperatorNodePushable.java
    • -30
    • +9
    ./ExternalIndexBulkModifyOperatorDescriptor.java
    • -9
    • +9
    ./ExternalIndexBulkModifyOperatorNodePushable.java
    • -23
    • +13
    ./ExternalLookupOperatorDescriptor.java
    • -17
    • +14
    ./ExternalRTreeSearchOperatorDescriptor.java
  1. … 557 more files in changeset.
Carry filter in 2ndary-to-primary index search

Change-Id: I287f1dbd230aa649f1350114abf0a1d47e2bb53c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1720

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Luo Chen <cluo8@uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -1
    • +1
    ./ExternalBTreeSearchOperatorNodePushable.java
    • -1
    • +1
    ./ExternalRTreeSearchOperatorNodePushable.java
  1. … 55 more files in changeset.
Remove static cc application context instance

Change-Id: Ia2e250405967ec880e7af6387aa981f39b3392c0

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1606

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

    • -2
    • +2
    ./FeedCollectOperatorNodePushable.java
  1. … 178 more files in changeset.
Cleanup storage exceptions

Change-Id: I020d2b4b1f4ae48fc2df0b720e70a1ce95867d34

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1619

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -19
    • +13
    ./ExternalDatasetIndexesCommitOperatorDescriptor.java
    • -15
    • +11
    ./ExternalFilesIndexOperatorDescriptor.java
    • -8
    • +3
    ./ExternalIndexBulkModifyOperatorNodePushable.java
  1. … 186 more files in changeset.
Feed Policy Refactoring

1. Fix the framesize inconsistency in feed dataflow.

2. Add a runtime test case for create feed with policy.

3. Fix the FeedRuntimeInputHandler flush() logic. Only flush after the

writer is opened.

4. Refactor FeedPolicyAccessor and BuiltinPolices. Now we only have

spill and discard polices.

5. Remove PolicyEnforcer. Merge the functionality into

FeedPolicyAccessor.

6. Revise SocketServerInputStream. Make the expected exception more

friendly.

7. Fixed one test case fail in change feed.

8. Refactor FeedRuntimeInputHandler consumer thread logic. Change the

poison consumer thread mechanism.

Change-Id: Ibc10139925cfedee66d1263990ba80b94675f182

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1591

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./FeedCollectOperatorNodePushable.java
  1. … 25 more files in changeset.
Fix record loss for certain certain feed type

1. Fix blindly replace connector between FeedCollector and

AssignOperator.

2. Wrap AssignOperator into the FeedMetaOperator to make sure the

operators inside (udf, accessor, etc.) can handle messages in the

feed workflow.

3. Revise feed connection job merge function.

4. Test case fix.

Change-Id: I4e1f7dd3621482a11feb675a93b826ae2cb965a7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1652

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 5 more files in changeset.
Unify structure of Cluster Controller and Node Controller

Change-Id: Ife3a002371a07ee9cdd32e1ffd50cc775bf1d453

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1514

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -2
    • +2
    ./FeedCollectOperatorNodePushable.java
  1. … 157 more files in changeset.
Cleanup and bug fixes in Feeds pipeline

The bug fix is for the MessagingFrameTupleAppender. It used

to consume one extra byte per frame.

Cleanups are for feeds. These include:

1. Remove the use of feed dataflow marker. Feeds which

need to send progress can and should do that without

a marker thread.

2. Lock the memory component for feed commit to be able

to add information to the memory component's metadata

page safely.

In addition, this change introduces a frame level callback

for index operations.

Change-Id: Ie97b2133ebecb7380cf0ba336e60ed714d06f8ee

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1523

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -2
    • +13
    ./FeedIntakeOperatorNodePushable.java
  1. … 72 more files in changeset.
ASTERIXDB-1791, ASTERIXDB-1796: fix failure handling in runtime operators.

This change includes the following parts:

- Fix the implementation of fail() and close() in several runtime operators

to avoid file handle leak and job hang;

- Add an erase method to RunFileWriter which closes files before deleting

them in order release the holding disk space;

- Call RunFileWriter.close() and RunFileReader.close() in "finally" blocks.

- Fix RunFileReader to not truncate files to be deleted - it is not the root

cause of un-released disk space - open deleted files are the root cuase;

- Check file handle leaks in LangExecutionUtil.tearDown().

Change-Id: I203168171e6dac16b57d2eda960823e3810e22a3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1513

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 53 more files in changeset.