Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -72
    • +0
    ./uci/ics/pregelix/example/FailureRecoveryTest.java
    • -72
    • +0
    ./uci/ics/pregelix/example/SkewTest.java
  1. … 943 more files in changeset.
Support big vertex in Pregelix. --For those vertice beyond page size, we store them on HDFS as immutable files. --Updates on those big vertice will trigger creations of new immutable files.

Change-Id: I6b6f0528b6b5360c96dcdace1fa360d42c517f22

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/72

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

    • -0
    • +72
    ./uci/ics/pregelix/example/SkewTest.java
  1. … 17 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -18
    • +1
    ./uci/ics/pregelix/example/FailureRecoveryTest.java
    • -0
    • +93
    ./uci/ics/pregelix/example/GraphSampleVertexTest.java
  1. … 266 more files in changeset.
1. make startcc/nc scripts flexible for different physical memory size; 2. add dynamic optimization option in the Client

  1. … 6 more files in changeset.
support heterogenous cluster

    • -0
    • +104
    ./uci/ics/pregelix/example/util/Record.java
    • -60
    • +59
    ./uci/ics/pregelix/example/util/TestUtils.java
  1. … 34 more files in changeset.
add new example for Counters usage

  1. … 2 more files in changeset.
support multiple user-defined global aggregators

  1. … 34 more files in changeset.
add pagerank multi-job test

    • -0
    • +103
    ./uci/ics/pregelix/example/MultiJobConnectedComponentsTest.java
    • -0
    • +98
    ./uci/ics/pregelix/example/MultiJobPageRankTest.java
  1. … 1 more file in changeset.
support multiple concurrent jobs

    • -0
    • +103
    ./uci/ics/pregelix/example/MultiJobTest.java
  1. … 14 more files in changeset.
turn the default configuration to be variable-sized vertex

  1. … 7 more files in changeset.
fix in-place update

  1. … 16 more files in changeset.
fix the issue found by Genomix P4 algorithm

  1. … 11 more files in changeset.
make update tests more extensive

add update test

    • -0
    • +54
    ./uci/ics/pregelix/example/UpdateVertex.java
    • -0
    • +87
    ./uci/ics/pregelix/example/UpdateVertexInputFormat.java
    • -0
    • +57
    ./uci/ics/pregelix/example/UpdateVertexOutputFormat.java
    • -0
    • +63
    ./uci/ics/pregelix/example/UpdateVertexTest.java
  1. … 1 more file in changeset.
fix fault-tolerance and error reporting to handle disk failures

    • -0
    • +90
    ./uci/ics/pregelix/example/FailureRecoveryInnerJoinTest.java
    • -0
    • +36
    ./uci/ics/pregelix/example/FailureVertex.java
  1. … 27 more files in changeset.
1. fix the node failure scenario in job scheduler; 2. add fault-tolerance support and tests in pregelix

    • -19
    • +18
    ./uci/ics/pregelix/example/FailureRecoveryTest.java
  1. … 19 more files in changeset.
fix the checkpoint writing for vertex checkpointing and message checkpointing

  1. … 4 more files in changeset.
support large-size global aggreate values

    • -0
    • +86
    ./uci/ics/pregelix/example/FailureRecoveryTest.java
    • -0
    • +70
    ./uci/ics/pregelix/example/FailureVertexTest.java
    • -0
    • +85
    ./uci/ics/pregelix/example/JobConcatenationTest.java
    • -0
    • +79
    ./uci/ics/pregelix/example/OverflowAggregatorTest.java
    • -0
    • +75
    ./uci/ics/pregelix/example/aggregator/OverflowAggregator.java
    • -150
    • +0
    ./uci/ics/pregelix/example/test/TestCluster.java
    • -0
    • +150
    ./uci/ics/pregelix/example/util/TestCluster.java
  1. … 15 more files in changeset.
implemented checkpoint read

  1. … 25 more files in changeset.
Merge branch 'master' into dev/hyracks_msr

Conflicts:

hyracks/hyracks-control/hyracks-control-cc/src/main/java/edu/uci/ics/hyracks/control/cc/job/JobRun.java

hyracks/hyracks-dataflow-std/src/main/java/edu/uci/ics/hyracks/dataflow/std/sort/FrameSorter.java

hyracks/hyracks-storage-am-common/src/main/java/edu/uci/ics/hyracks/storage/am/common/impls/AbstractTreeIndex.java

pregelix/pregelix-api/src/main/java/edu/uci/ics/pregelix/api/graph/Vertex.java

pregelix/pregelix-example/src/main/java/edu/uci/ics/pregelix/example/io/VLongWritable.java

pregelix/pregelix-example/src/test/java/edu/uci/ics/pregelix/example/lib/io/SizeEstimationTest.java

pregelix/pregelix-runtime/src/main/java/edu/uci/ics/pregelix/runtime/function/ComputeUpdateFunctionFactory.java

pregelix/pregelix-runtime/src/main/java/edu/uci/ics/pregelix/runtime/simpleagg/AccumulatingAggregatorFactory.java

  1. … 4 more files in changeset.
add job concatenation support

  1. … 8 more files in changeset.
cleanup client error message propagation

    • -0
    • +67
    ./uci/ics/pregelix/example/test/FailureVertexTest.java
    • -0
    • +150
    ./uci/ics/pregelix/example/test/TestCluster.java
  1. … 7 more files in changeset.
merge from zheilbron/hyracks_msr

  1. … 290 more files in changeset.
inherit Hadoop's VLongWritable format

  1. … 3 more files in changeset.
addd partition early termination support

  1. … 11 more files in changeset.
complete sizeestimation test

    • -0
    • +151
    ./uci/ics/pregelix/example/lib/io/SizeEstimationTest.java
  1. … 1 more file in changeset.
code cleanup

  1. … 63 more files in changeset.
add message overflow support

    • -0
    • +57
    ./uci/ics/pregelix/example/lib/io/VLongWritableTest.java
  1. … 88 more files in changeset.
rename a method

  1. … 1 more file in changeset.
increase the lsm mem budget to fix the budget exceed issue

  1. … 3 more files in changeset.