Clone Tools
  • last updated 14 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -34
    • +0
    ./IIterationCompleteReporterHook.java
  1. … 955 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 276 more files in changeset.
add support for Hadoop Counters via job.setCounterAggregatorClass

The PregelixJob.setCounterAggregatorClass sets up a (user-specified)

global aggregator and an iterationComplete hook to save Counter values.

The user-specified Counter-based aggregator (must extend

HadoopCountersAggregator) is saved to HDFS in each iteration and should

be restart/snapshot-aware.

The usage for setting up counters is to make a call to

job.setCounterAggregatorClass. After job completion, the Counters may

be retrieved from HDFS using BspUtils.getCounters(job).

Note that there is currently only one spot for iterationComplete hooks

and this behavior occupies it.

  1. … 7 more files in changeset.
add an "iteration complete" hook for aggregation/reporting across iterations

This commit allows the user to specify a class which will be called upon

completion of each pregelix iteration. This allows us to perform a user-

specified action between iterations.

As an example, a PerIterationGlobalAggregatesHook is provided which

saves the complete set of global aggregator states from every iteration,

allowing the user to observe aggregates from all iterations.

The default hook does nothing.

The hook instance is attached directly to the PregelixJob so that it can

be retrieved by the Driver's caller.

    • -0
    • +32
    ./IIterationCompleteReporterHook.java
  1. … 6 more files in changeset.
allow global aggregators to be specified in xml

explicitly setting the aggregator in the PregelixJob constructors would

override any values read in from the conf's resources.

Instead, this commit doesn't set the conf explicitly and instead

specifies an array of aggregator class names which will always be in

place when `getGlobalAggregatorClasses` is called.

  1. … 1 more file in changeset.
add the dyanmic optimization skeleton

  1. … 4 more files in changeset.
support multiple user-defined global aggregators

  1. … 35 more files in changeset.
turn the default configuration to be variable-sized vertex

  1. … 10 more files in changeset.
make failure-recovery work in multi-JVM test

  1. … 6 more files in changeset.
fix fault-tolerance and error reporting to handle disk failures

  1. … 31 more files in changeset.
add vertex checkpointing support

    • -0
    • +24
    ./ICheckpointHook.java
  1. … 9 more files in changeset.
avoid btree update code path when the target entry has sufficient space

  1. … 5 more files in changeset.
merge from zheilbron/hyracks_msr

  1. … 291 more files in changeset.
rename a method

  1. … 1 more file in changeset.
add LSM support in pregelix

  1. … 15 more files in changeset.
add the support for customized partitioner

  1. … 15 more files in changeset.
add/update license headers

  1. … 2287 more files in changeset.
add normalized key computer support in Pregelix

  1. … 37 more files in changeset.
cross merge fullstack_release_candidate into trunk

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3208 123451ca-8445-de46-9d55-352943316053

  1. … 902 more files in changeset.
Merge fullstack_asterix_stabilization into fullstack_hyracks_result_distribution branch.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2862 123451ca-8445-de46-9d55-352943316053

  1. … 390 more files in changeset.
support job specific frame size in Pregelix

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2682 123451ca-8445-de46-9d55-352943316053

  1. … 6 more files in changeset.
let Pregelix support dynamic vertex value size

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2677 123451ca-8445-de46-9d55-352943316053

  1. … 14 more files in changeset.
reduce the memory copies in the index join runtime

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2674 123451ca-8445-de46-9d55-352943316053

  1. … 6 more files in changeset.
add the reachibility query example

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2079 123451ca-8445-de46-9d55-352943316053

  1. … 12 more files in changeset.
add support for non combiner jobs

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2077 123451ca-8445-de46-9d55-352943316053

  1. … 21 more files in changeset.
rename a few classes

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2075 123451ca-8445-de46-9d55-352943316053

  1. … 19 more files in changeset.
add global aggregator support

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2066 123451ca-8445-de46-9d55-352943316053

  1. … 13 more files in changeset.
necessary modifications for adding global aggregator

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@2065 123451ca-8445-de46-9d55-352943316053

  1. … 11 more files in changeset.
add more comments for Vertex abstract class

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@1994 123451ca-8445-de46-9d55-352943316053

  1. … 2 more files in changeset.
update copyright header for pregelix source code

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_staging@1993 123451ca-8445-de46-9d55-352943316053

  1. … 54 more files in changeset.