Clone Tools
  • last updated 23 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
[NO ISSUE] Apply / enforce java import order

The process-sources target will now sort imports as well as

format source code; the source-format job will likewise verify

import order in addition to source code format

Change-Id: I55d976c4df10d9919c6a25683be2a3e3304e65d9

Reviewed-on: https://asterix-gerrit.ics.uci.edu/3288

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

Reviewed-by: Till Westmann <tillw@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./MaterializingPipelinedPartition.java
  1. … 624 more files in changeset.
[NO ISSUE][NET] Ensure CLOSE Is Not Sent After Channel ERROR

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently it is possible to send network channel

CLOSE command after a channel ERROR was sent. When this

happens and the channel was recycled to be reused

on the receiver side, the CLOSE command will result

in NPE. There is no need to send a CLOSE command

after an ERROR command because when an ERROR command

is received, it is treated as ERROR + CLOSE on the

receiver side.

- Avoid registering partition requests for failed jobs.

Change-Id: I17a769a46f4d13220adb22dd255e56dc4ccc458d

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2954

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Reviewed-by: Michael Blow <mblow@apache.org>

  1. … 3 more files in changeset.
[NO ISSUE][NET] Propagate Network Failure Error Codes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Pass the network error code to the channel

to determine whether the error should be

sent to the receiver or not.

Change-Id: I29cffa916df9fecc9942f449ba6c5ca7c84b43b3

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2912

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 4 more files in changeset.
[NO ISSUE][OTH] Remove Unused Imports

- user model changes: no

- storage format changes: no

- interface changes: no

Change-Id: Iafff39073d0fedaff74a26ef7e3260008a79ff0c

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2893

Reviewed-by: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

  1. … 67 more files in changeset.
[NO ISSUE] Don't log interrupts as WARNings

Change-Id: I0e847b8197fa12f1ce235dde404df24196939a83

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2855

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -2
    • +4
    ./MaterializingPipelinedPartition.java
  1. … 7 more files in changeset.
[NO ISSUE] Set logging level to WARN in MaterializingPipelinedPartition

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

Failures in MaterializingPipelinedPartition can be due to many reasons

such as a job cancellation. WARN log level seems to be adequate for

such failures keeping ERROR for more critical failures.

Change-Id: I45f475afae114dc71d2d89d9deaab7f3b3c5a0b8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2832

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -2
    • +2
    ./MaterializingPipelinedPartition.java
[NO ISSUE][RT] Abort tasks on local network failures

- user model changes: no

- storage format changes: no

- interface changes: yes

Add error code to IInputChannelMonitor.notifyFailure

Details:

- Previously, there was an assumption that all failures

reported to an IInputChannelMonitor come from a remote

task.

- This assumption is not always true and could lead

to jobs hanging.

- To fix this, we report an error code indicating whether

the failure is local or remote and if the failure is local

then we fail the local task and report the failure to cc.

Change-Id: I7ea5b9008383faaac7c563671242b03919090b35

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2806

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +1
    ./ReceiveSideMaterializingCollector.java
  1. … 14 more files in changeset.
[NO ISSUE][NET] Networking Fixes

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Ensure received partitions requests after job failure

are aborted to prevent leaked network channels.

- Do not send channel close after channel write error

since the contract is to close the channel when remote

errors are received.

- Only remove closed outgoing connections to establish

new connections since incoming connections need to be

reestablished by the remote destination.

- Do not perform further operations on failed multiplexed

connections to avoid CanceledKeyException.

- Add test case for received partition requests after

job failure.

Change-Id: Idc45f47fdf0419bf75d461e16f028237a5143de7

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2613

Reviewed-by: Michael Blow <mblow@apache.org>

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Michael Blow <mblow@apache.org>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 11 more files in changeset.
[NO ISSUE] Incremental cleanup of deprecated exception ctors

Change-Id: I1e7c3655828fc6530cef83ea502a6cfbf41acddf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2533

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -1
    • +1
    ./MaterializedPartitionInputChannel.java
    • -1
    • +1
    ./ReceiveSideMaterializingCollector.java
  1. … 239 more files in changeset.
[ASTERIXDB-2229][OTR] Restore Thread Names in Thread Pool

- user model changes: no

- storage format changes: no

- interface changes: no

- Restore thread names to their original names

before returning them to the pool of the NC

app.

- Remove explicit thread name resets.

- Delete unused ThreadExecutor class.

Change-Id: I3bda1b65e7aefd35d2b8cfa814f73369c3bf5a18

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2447

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -2
    • +0
    ./MaterializingPipelinedPartition.java
  1. … 9 more files in changeset.
[ASTERIXDB-2256] Reformat sources using code format template

Change-Id: I4faa141c1a8c9700d5e9ac50b839acc9d1eede73

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2310

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

    • -2
    • +2
    ./MaterializingPipelinedPartition.java
  1. … 982 more files in changeset.
[ASTERIXDB-2110] Introduce Cluster Controller Id

Change-Id: Iec1b01444bfbd923e38f5c162c5244e17c4d5f03

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2323

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -1
    • +2
    ./MaterializingPipelinedPartition.java
  1. … 77 more files in changeset.
[ASTERIXDB-1564][CONF] Consistently Use Log4j2

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Replace java.util.logging by Log4j2.

- Excluded classes due to their tests:

- IoUtil

- Tracer

Change-Id: Ic137571292f45de1f1994c61d328b97185012197

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2226

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -7
    • +8
    ./MaterializingPipelinedPartition.java
  1. … 389 more files in changeset.
[ASTERIXDB-1871][ASTERIXDB-2095] Stop Consumer Thread on Deallocate

- user model changes: no

- storage format changes: no

- interface changes: no

Details:

- Currently there is a chance that a MaterializingPipelinedPartition

is deallocated before the consuming thread starts (e.g. due to job

abort) and therefore the consuming thread will not be interrupted

leading to leaked threads and files. This change checks if partition

was deallocated before the consumer starts, then the consumer thread

cleans up any files then exits.

- Make TaskAttemptId non-final class to mock it.

- Add test case.

Change-Id: I18c9fb085c149f41a202fff83aa6ec3aaeba6a77

Reviewed-on: https://asterix-gerrit.ics.uci.edu/2143

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -16
    • +11
    ./MaterializingPipelinedPartition.java
  1. … 5 more files in changeset.
[ASTERIXDB-1992][ING] Suspend/Resume active entities

- user model changes: wait for completion only returns

when the entity becomes inactive.

- storage format changes: no

- interface changes:

01) Introduce:

IMetadataLockManager for entity locking.

IActiveNotificationHandler for handling active events.

IRetryPolicy for recovery of failed active jobs.

IActiveEntityController for controlling active entities.

02) IJobLifecycleListener.notifyJobFinish now passes the

JobStatus and Exceptions.

03) IActiveEntityEventsListener.isActive() returns true,

if entity is active, false otherwise.

04) IActiveEntityEventsListener.unregister() removes the

listener upon entity deletion.

05) IActiveEntityEventsListener.getJobFailure() returns

the job failure if the entity is in a failed state.

06) IStatementExecutor.getComponentProvider() returns

the storage component provider.

07) IStatementExecutor.getApplicationContext() returns

the application context.

08) IMetadataManager.upsertEntity to perfrom metadata

entities' upsert operations.

09) IMetadataNode.upsertEntity to perfrom metadata

entities' upsert operations

10) ICcApplicationContext.getMetadataLockManager() returns

the lock manger.

11) ICcApplicationContext.getClusterStateManager() returns

the cluster state manager.

details:

- Starting and stopping of active entities now go through

their listeners rather than having the listener indicating

active entity and not having it indicating inactive entity.

This facilitates suspend/resume operations of long running

active jobs for the sake of DDL operations or topology

changes.

- Unit tests for the vast majority of code paths and

different possible scenarios have been added.

Change-Id: Ifeac8c73e6bad39a13663b84a52121356e3c6b40

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1875

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <mhubail@apache.org>

  1. … 140 more files in changeset.
Separate index build from index access

This change separates index build from index access.

All indexes now have a single dataflow helper which

uses the index path to locate the resource on the nc

to read the resource from memory or disk.

Existing resource metadata and dataflow helpers were

combined into resource builders eliminating lots of

duplicated code.

Change-Id: Ie4ea3aaa63dff8d246fa43ca7c7359729bc8cf47

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1728

Integration-Tests: Ian Maxon <imaxon@apache.org>

Tested-by: Ian Maxon <imaxon@apache.org>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -3
    • +3
    ./MaterializingPipelinedPartition.java
  1. … 570 more files in changeset.
Add a REST endpoint for query cancellation.

This change also includes the following parts:

- Fix failure handling and potentially thread leakage in

MaterializingPipelinedPartition;

- Fix failure handling in PartitionDataWriter;

- Add a new test suite: SqlppExecutionWithCancellationTest.

Change-Id: I2936ac83f71bbef533e2695ed0a2b220c23fc483

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1564

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -55
    • +87
    ./MaterializingPipelinedPartition.java
  1. … 44 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +2
    ./MaterializingPipelinedPartition.java
  1. … 251 more files in changeset.
Remove HyracksRootContext

Change-Id: I24f9d7b1fc5bdcf9d09abdac107001ff6b70bf02

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1321

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 15 more files in changeset.
Reduced logging

Change-Id: I3a3756427999c4b79a01cd5a9cbfe79adf44b95b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1204

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -4
    • +6
    ./MaterializingPipelinedPartition.java
  1. … 3 more files in changeset.
Add Test NodeController, Test Data Generator, and Marker Logs

This test enable creating a node controller for unit test purposes.

The Node controller is identical to the regular node controller

except that it doesn't communicate with a cluster controller at all.

In this change, Test Data Generator is introduced which should

facilitate writing unit test cases which requires data generation.

The change also includes enabling feeds to send progress data. progress

information can then be sent through the pipeline and persisted in the

transaction logs and primary index component. A Unit test case has

been created to test adding progress markers to logs and index

components and then reading them.

The last part of this change is the addition of marker logs and their

callbacks. They enable components to create arbitrary logs and get a

callback when they are written to the transaction logs. Initial set of

unit tests were added for marker logs.

Change-Id: I3b9aa8de758b7d26ca34868b16e5ce693e0c0243

Reviewed-on: https://asterix-gerrit.ics.uci.edu/962

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -11
    • +7
    ./MaterializingPipelinedPartition.java
  1. … 82 more files in changeset.
Move Hyracks to subfolder

    • -0
    • +109
    ./MaterializedPartition.java
    • -0
    • +146
    ./MaterializedPartitionInputChannel.java
    • -0
    • +111
    ./MaterializedPartitionWriter.java
    • -0
    • +216
    ./MaterializingPipelinedPartition.java
    • -0
    • +155
    ./PartitionManager.java
    • -0
    • +128
    ./PipelinedPartition.java
    • -0
    • +184
    ./ReceiveSideMaterializingCollector.java
  1. … 4421 more files in changeset.