Clone Tools
  • last updated 25 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 949 more files in changeset.
Support big vertex in Pregelix. --For those vertice beyond page size, we store them on HDFS as immutable files. --Updates on those big vertice will trigger creations of new immutable files.

Change-Id: I6b6f0528b6b5360c96dcdace1fa360d42c517f22

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/72

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

  1. … 17 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -0
    • +71
    ./VertexContext.java
  1. … 273 more files in changeset.
fix conflict in comment

add support for Hadoop Counters via job.setCounterAggregatorClass

The PregelixJob.setCounterAggregatorClass sets up a (user-specified)

global aggregator and an iterationComplete hook to save Counter values.

The user-specified Counter-based aggregator (must extend

HadoopCountersAggregator) is saved to HDFS in each iteration and should

be restart/snapshot-aware.

The usage for setting up counters is to make a call to

job.setCounterAggregatorClass. After job completion, the Counters may

be retrieved from HDFS using BspUtils.getCounters(job).

Note that there is currently only one spot for iterationComplete hooks

and this behavior occupies it.

  1. … 8 more files in changeset.
api for specifying update state for activate() and voteToHalt()

Merge branch 'master' into zheilbron/hyracks_msr_demo

  1. … 2 more files in changeset.
fix IIndexAccessor interface, add a boolean exclusiveMode parameter for the createSearchCursor method

  1. … 43 more files in changeset.
fix file write race condition

  1. … 23 more files in changeset.
Merge branch 'master' into dev/hyracks_msr

Conflicts:

hyracks/hyracks-control/hyracks-control-cc/src/main/java/edu/uci/ics/hyracks/control/cc/job/JobRun.java

hyracks/hyracks-dataflow-std/src/main/java/edu/uci/ics/hyracks/dataflow/std/sort/FrameSorter.java

hyracks/hyracks-storage-am-common/src/main/java/edu/uci/ics/hyracks/storage/am/common/impls/AbstractTreeIndex.java

pregelix/pregelix-api/src/main/java/edu/uci/ics/pregelix/api/graph/Vertex.java

pregelix/pregelix-example/src/main/java/edu/uci/ics/pregelix/example/io/VLongWritable.java

pregelix/pregelix-example/src/test/java/edu/uci/ics/pregelix/example/lib/io/SizeEstimationTest.java

pregelix/pregelix-runtime/src/main/java/edu/uci/ics/pregelix/runtime/function/ComputeUpdateFunctionFactory.java

pregelix/pregelix-runtime/src/main/java/edu/uci/ics/pregelix/runtime/simpleagg/AccumulatingAggregatorFactory.java

  1. … 4 more files in changeset.
merge from zheilbron/hyracks_msr

  1. … 288 more files in changeset.
use hadoop's vlong and fix driver's re-attempt to be 1

  1. … 2 more files in changeset.
addd partition early termination support

  1. … 11 more files in changeset.
address Vinayak's comments on open/close of vertex

  1. … 7 more files in changeset.
add message overflow support

  1. … 86 more files in changeset.
refactoring dataflow operators to be more general--suport both b-tree and lsm b-tree

  1. … 16 more files in changeset.
add the support for customized partitioner

    • -0
    • +36
    ./VertexPartitioner.java
  1. … 15 more files in changeset.
add/update license headers

  1. … 2282 more files in changeset.
add normalized key computer support in Pregelix

    • -0
    • +37
    ./NormalizedKeyComputer.java
  1. … 37 more files in changeset.
Merged fullstack_lsm_staging upto r3336

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3339 123451ca-8445-de46-9d55-352943316053

  1. … 898 more files in changeset.
fix the halt-activate issue reported by Anbang

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3269 123451ca-8445-de46-9d55-352943316053

  1. … 2 more files in changeset.
cross merge fullstack_release_candidate into trunk

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3208 123451ca-8445-de46-9d55-352943316053

  1. … 899 more files in changeset.
Merge fullstack_asterix_stabilization with fullstack_hyracks_result_distribution.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@3170 123451ca-8445-de46-9d55-352943316053

  1. … 190 more files in changeset.
Merged fullstack_asterix_stabilization -r 3157:3163

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_ioc@3165 123451ca-8445-de46-9d55-352943316053

  1. … 12 more files in changeset.
Merged fullstack_asterix_stabilization -r 2933:3157

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_ioc@3164 123451ca-8445-de46-9d55-352943316053

  1. … 1166 more files in changeset.
fix an issue reported by anbang

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3161 123451ca-8445-de46-9d55-352943316053

Merge fullstack_asterix_stabilization into fullstack_hyracks_result_distribution.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@3124 123451ca-8445-de46-9d55-352943316053

  1. … 705 more files in changeset.
fix addVertex in the inner join plan

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_release_cleanup@3090 123451ca-8445-de46-9d55-352943316053

  1. … 6 more files in changeset.
fix addVertex and add a test case

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_release_cleanup@3089 123451ca-8445-de46-9d55-352943316053

  1. … 8 more files in changeset.
Merged fullstack_asterix_stabilization -r 2813:2933

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_ioc@2995 123451ca-8445-de46-9d55-352943316053

  1. … 90 more files in changeset.