pregelix

Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
add support for Hadoop Counters via job.setCounterAggregatorClass

The PregelixJob.setCounterAggregatorClass sets up a (user-specified)

global aggregator and an iterationComplete hook to save Counter values.

The user-specified Counter-based aggregator (must extend

HadoopCountersAggregator) is saved to HDFS in each iteration and should

be restart/snapshot-aware.

The usage for setting up counters is to make a call to

job.setCounterAggregatorClass. After job completion, the Counters may

be retrieved from HDFS using BspUtils.getCounters(job).

Note that there is currently only one spot for iterationComplete hooks

and this behavior occupies it.

add an "iteration complete" hook for aggregation/reporting across iterations

This commit allows the user to specify a class which will be called upon

completion of each pregelix iteration. This allows us to perform a user-

specified action between iterations.

As an example, a PerIterationGlobalAggregatesHook is provided which

saves the complete set of global aggregator states from every iteration,

allowing the user to observe aggregates from all iterations.

The default hook does nothing.

The hook instance is attached directly to the PregelixJob so that it can

be retrieved by the Driver's caller.

fix NPE when no custom aggregator is set

api for specifying update state for activate() and voteToHalt()

allow global aggregators to be specified in xml

explicitly setting the aggregator in the PregelixJob constructors would

override any values read in from the conf's resources.

Instead, this commit doesn't set the conf explicitly and instead

specifies an array of aggregator class names which will always be in

place when `getGlobalAggregatorClasses` is called.

add APT for retrieving slave counters

  1. … 4 more files in changeset.
add the dyanmic optimization skeleton

fix driver logging

let pregelix driver print perf. counter

fix jobgen

1. add deployment retry 2. support plan switch

  1. … 1 more file in changeset.
added ability to trigger an application state dump through the rest api

  1. … 21 more files in changeset.
fix application lifecyle mgmt in hyracks nc

  1. … 2 more files in changeset.
Merge branch 'master' into zheilbron/hyracks_msr_demo

  1. … 2 more files in changeset.
fix the pinned page issue during a node failure

NodeControllers clean up appEntryPoints on shutdown (2nd try)

  1. … 1 more file in changeset.
fix an issue found by Sattam

support multiple user-defined global aggregators

  1. … 21 more files in changeset.
fix IIndexAccessor interface, add a boolean exclusiveMode parameter for the createSearchCursor method

  1. … 34 more files in changeset.
fix file write race condition

  1. … 9 more files in changeset.
cleanup pregelix job context

add pagerank multi-job test

fix the duplicate jobid issue

support multiple concurrent jobs

turn the default configuration to be variable-sized vertex

disable in-place update for variable-sized updates

use in-place update for smaller sized updates

fix in-place update

    • -20
    • +23
    ./pregelix-dataflow-std-base/pom.xml
  1. … 2 more files in changeset.
fix the issue found by Genomix P4 algorithm

make update tests more extensive