Clone Tools
  • last updated 20 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Configuration Revamp

- Ini section of node / cc details now returns ini param names instead of

managix option names

- Normalized command line -vs- ini file configuration parameter names

- Eliminated unused parameters

- Ini validation

- Migrate *DB parameters out of [app] and into nc / cc sections as

appropriate

- Eliminate [app] section. Cluster-wide configuration lives in [common]

- Sort properties alphabetically when returned by HTTP api

Change-Id: I95b7e0bd4538ef42817c8826e76412150074b754

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1487

Reviewed-by: Michael Blow <mblow@apache.org>

Integration-Tests: Michael Blow <mblow@apache.org>

Tested-by: Michael Blow <mblow@apache.org>

    • -2
    • +1
    ./support/TestNCApplicationContext.java
  1. … 173 more files in changeset.
RAT Cleanup, Resulting LICENSE fixes

Also:

- removed unused minimal javascript

- relocated storage test datagen package to hyracks-test-support

- eliminated MiniDFSCluster writes outside of target

- clean up .gitignore

- add line comments to PigletParser (taken from SQLPP.jj grammar)

Change-Id: I029dee65ac6b1d7ab7bddba66257a4857f682caa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1468

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

  1. … 71 more files in changeset.
Introduce IStorageComponentProvider

Change-Id: If86750cdb2436c713f6598e54d4aaaf23d9f7bbf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1451

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -0
    • +50
    ./support/TestStorageManager.java
    • -1
    • +1
    ./support/TestStorageManagerComponentHolder.java
    • -50
    • +0
    ./support/TestStorageManagerInterface.java
  1. … 425 more files in changeset.
Remove Append Only Flag

Change-Id: Id5d6917db8ab29aa01521596f556006e25a502fe

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1385

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -2
    • +10
    ./support/TestStorageManagerComponentHolder.java
  1. … 213 more files in changeset.
Continue Cleaning Up File References and Splits

1. Make FileSplit an abstract class with two subclasses;

Managed and Unmanaged. A Managed FileSplit can be mapped

in a new subclass MappedFileSplit that maps a relative path to an

IO device. UnmanagedFileSplit is for files outside the io devices.

2. Remove all usages of absolute paths in file split in test cases. The

only remaining place is the write statement.

3. Fix some of the hidden issues in the tests that were working because

of our use of the absolute paths.

4. Revert the decision of selecting the IO device to the CC.

Change-Id: I166af8f9b3a2257f94d7b05db94888fb7cb4c79e

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1359

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -2
    • +2
    ./support/TestNCApplicationContext.java
    • -2
    • +3
    ./support/TestStorageManagerComponentHolder.java
  1. … 250 more files in changeset.
Remove HyracksRootContext

Change-Id: I24f9d7b1fc5bdcf9d09abdac107001ff6b70bf02

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1321

Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -6
    • +7
    ./support/TestNCApplicationContext.java
  1. … 12 more files in changeset.
Small Cleanup Towards Fixing LifeCycle Issues

Before this change, dataset lifecycle manager was providing a set

of functionalities that are loosly related to management of datasets

and indexes. However, it was not clear what the possible states of

a dataset or an index and what is the responsibility of each object.

This change takes the first step towards fixing this area. Indexes

of a dataset are now grouped together under a single lifecycle class

A resource aka dataset must be created outside the lifecycle manager

and registered with it before it can be assigned resources (memory)

and before it can be used by any operation. This is still not the

case.

Change-Id: I84005a33837725f41ae63297a3711215dccce1d8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/1148

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -3
    • +4
    ./support/TestIndexLifecycleManagerProvider.java
    • -4
    • +7
    ./support/TestStorageManagerComponentHolder.java
  1. … 69 more files in changeset.
Introduce MessagingNetworkManager for NC2NC AppMessaging

This change introduces MessagingNetworkManager to NodeControllerService.

The MessagingNetworkManager is used to open channels that are used for NC2NC

application messaging. The read and write interfaces of the messaging channels

are set by a MessagingChannelInterfaceFactory which is set by the application.

Change-Id: I5c0bd7c11c1e78954ebceff49cb274d8073a64bd

Reviewed-on: https://asterix-gerrit.ics.uci.edu/897

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

    • -0
    • +11
    ./support/TestNCApplicationContext.java
  1. … 74 more files in changeset.
Add Test NodeController, Test Data Generator, and Marker Logs

This test enable creating a node controller for unit test purposes.

The Node controller is identical to the regular node controller

except that it doesn't communicate with a cluster controller at all.

In this change, Test Data Generator is introduced which should

facilitate writing unit test cases which requires data generation.

The change also includes enabling feeds to send progress data. progress

information can then be sent through the pipeline and persisted in the

transaction logs and primary index component. A Unit test case has

been created to test adding progress markers to logs and index

components and then reading them.

The last part of this change is the addition of marker logs and their

callbacks. They enable components to create arbitrary logs and get a

callback when they are written to the transaction logs. Initial set of

unit tests were added for marker logs.

Change-Id: I3b9aa8de758b7d26ca34868b16e5ce693e0c0243

Reviewed-on: https://asterix-gerrit.ics.uci.edu/962

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <mblow@apache.org>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 82 more files in changeset.
Eliminate Hard Tabs From Java Files

tabs->spaces

Change-Id: I0dfbeaa9b9b24d3da106ead85afd9df02b14ddaa

Reviewed-on: https://asterix-gerrit.ics.uci.edu/921

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -5
    • +4
    ./support/TestStorageManagerInterface.java
  1. … 66 more files in changeset.
Improve Messaging Connector and Add Unit Tests

Before this change, messaging connector always reserves 100 bytes

for messages which are mostly un-used. With this change, it only

reserves two bytes and sends null messages by default. In case a

new message doesn't fit in the leftover space of a frame, it sends

the frame with a null message, followed by a dedicated frame for

the message.

Change-Id: If4336e9c234e8d282798cfba9f48432b46cccfca

Reviewed-on: https://asterix-gerrit.ics.uci.edu/880

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

Reviewed-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

  1. … 23 more files in changeset.
Cleanup Feed CodeBase

In order to expedite cleaning feeds up and reaching a maintainable

state, we disabled the following:

1. Policies (At least once, throttling, discarding, elasticity).

2. Statistics Reporting.

3. Load management.

4. Feed re-activation upon System reboot.

Right now on master, none of these features work reliably.

We will re-introduce them one feature at a time.

The rules followed in this change:

1. Keep X if X is tested in a test case.

2. Remove X if X is not used in test cases.

After a few meetings with Mike and Till, the policies

1. Buffering

2. Spill

3. Discard

4. Throttle

have been fixed and unit tests have been added.

Change-Id: I545bc4f8560564e4c868a80d27c77a4edd97a8b8

Reviewed-on: https://asterix-gerrit.ics.uci.edu/798

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

  1. … 141 more files in changeset.
Overhaul of Hyracks configuration management.

Includes Asterix changes to make use of new conf management as a

Hyracks application.

Change-Id: Ie3027c8c839f25ea858790bd3340187f4b11f212

Reviewed-on: https://asterix-gerrit.ics.uci.edu/336

Tested-by: Chris Hillery <ceej@lambda.nu>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Michael Blow <michael.blow@couchbase.com>

Reviewed-by: Ian Maxon <imaxon@apache.org>

    • -0
    • +6
    ./support/TestNCApplicationContext.java
  1. … 36 more files in changeset.
need just 1 interface method as byte[] is Serializable

Change-Id: I471c800037781b678d2a6ba2ba0281f78623e7bf

Reviewed-on: https://asterix-gerrit.ics.uci.edu/800

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>

  1. … 3 more files in changeset.
Move Hyracks to subfolder

    • -0
    • +52
    ./support/CounterContext.java
    • -0
    • +34
    ./support/TestIndexLifecycleManagerProvider.java
    • -0
    • +129
    ./support/TestJobletContext.java
    • -0
    • +137
    ./support/TestNCApplicationContext.java
    • -0
    • +52
    ./support/TestRootContext.java
    • -0
    • +136
    ./support/TestStorageManagerComponentHolder.java
    • -0
    • +51
    ./support/TestStorageManagerInterface.java
    • -0
    • +158
    ./support/TestTaskContext.java
    • -0
    • +44
    ./support/TestUtils.java
    • -0
    • +52
    ./support/TestVirtualBufferCacheProvider.java
  1. … 4418 more files in changeset.