Clone
Jake Biesinger <jake.biesinger@gmail.com>
committed
on 28 Nov 13
add support for Hadoop Counters via job.setCounterAggregatorClass
The PregelixJob.setCounterAggregatorClass sets up a (user-specified)
globa… Show more
add support for Hadoop Counters via job.setCounterAggregatorClass

The PregelixJob.setCounterAggregatorClass sets up a (user-specified)

global aggregator and an iterationComplete hook to save Counter values.

The user-specified Counter-based aggregator (must extend

HadoopCountersAggregator) is saved to HDFS in each iteration and should

be restart/snapshot-aware.

The usage for setting up counters is to make a call to

job.setCounterAggregatorClass.  After job completion, the Counters may

be retrieved from HDFS using BspUtils.getCounters(job).

Note that there is currently only one spot for iterationComplete hooks

and this behavior occupies it.

Show less

master + 7 more