Clone Tools
  • last updated 18 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -199
    • +0
    ./ConnectedComponentsVertex.java
    • -229
    • +0
    ./GraphSampleUndirectedVertex.java
    • -113
    • +0
    ./MessageOverflowFixedsizeVertex.java
    • -70
    • +0
    ./data/VLongNormalizedKeyComputer.java
    • -119
    • +0
    ./inputformat/TextConnectedComponentsInputFormat.java
    • -115
    • +0
    ./inputformat/TextGraphSampleVertexInputFormat.java
    • -114
    • +0
    ./inputformat/TextPageRankInputFormat.java
  1. … 943 more files in changeset.
Support big vertex in Pregelix. --For those vertice beyond page size, we store them on HDFS as immutable files. --Updates on those big vertice will trigger creations of new immutable files.

Change-Id: I6b6f0528b6b5360c96dcdace1fa360d42c517f22

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/72

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

  1. … 17 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -31
    • +52
    ./ConnectedComponentsVertex.java
    • -0
    • +229
    ./GraphSampleUndirectedVertex.java
    • -0
    • +252
    ./GraphSampleVertex.java
    • -7
    • +13
    ./inputformat/TextConnectedComponentsInputFormat.java
    • -0
    • +115
    ./inputformat/TextGraphSampleVertexInputFormat.java
    • -6
    • +7
    ./inputformat/TextPageRankInputFormat.java
    • -8
    • +8
    ./inputformat/TextReachibilityVertexInputFormat.java
    • -6
    • +7
    ./inputformat/TextShortestPathsInputFormat.java
  1. … 262 more files in changeset.
fix client dyn-opt setting

1. make startcc/nc scripts flexible for different physical memory size; 2. add dynamic optimization option in the Client

  1. … 6 more files in changeset.
add new example for Counters usage

    • -1
    • +1
    ./trianglecounting/TriangleCountingVertex.java
    • -0
    • +77
    ./trianglecounting/TriangleCountingWithAggregateHadoopCountersVertex.java
  1. … 1 more file in changeset.
support multiple user-defined global aggregators

    • -2
    • +2
    ./maximalclique/MaximalCliqueVertex.java
    • -3
    • +3
    ./trianglecounting/TriangleCountingVertex.java
  1. … 34 more files in changeset.
turn the default configuration to be variable-sized vertex

  1. … 10 more files in changeset.
fix the shortestpaths input format

    • -0
    • +1
    ./inputformat/TextShortestPathsInputFormat.java
improve the performance of the reachability vertex and the shortest paths vertex

address Vinayak's code review comments

    • -11
    • +2
    ./data/VLongNormalizedKeyComputer.java
  1. … 17 more files in changeset.
avoid input stream abstraction for vlong's comparator

    • -0
    • +56
    ./utils/SerDeUtils.java
make failure-recovery work in multi-JVM test

  1. … 6 more files in changeset.
fix fault-tolerance and error reporting to handle disk failures

  1. … 31 more files in changeset.
add job concatenation support

  1. … 9 more files in changeset.
cleanup client error message propagation

    • -0
    • +36
    ./FailureVertex.java
  1. … 8 more files in changeset.
merge from zheilbron/hyracks_msr

    • -0
    • +113
    ./MessageOverflowFixedsizeVertex.java
    • -1
    • +1
    ./inputformat/TextPageRankInputFormat.java
    • -1
    • +1
    ./inputformat/TextShortestPathsInputFormat.java
  1. … 277 more files in changeset.
inherit Hadoop's VLongWritable format

    • -27
    • +40
    ./data/VLongNormalizedKeyComputer.java
  1. … 2 more files in changeset.
fix the pregelix example driver to deal with multiple input paths

addd partition early termination support

    • -0
    • +106
    ./EarlyTerminationVertex.java
  1. … 8 more files in changeset.
complete sizeestimation test

  1. … 2 more files in changeset.
address Vinayak's comments on open/close of vertex

  1. … 9 more files in changeset.
code cleanup

    • -0
    • +66
    ./data/VLongNormalizedKeyComputer.java
    • -1
    • +1
    ./inputformat/TextConnectedComponentsInputFormat.java
    • -2
    • +2
    ./inputformat/TextPageRankInputFormat.java
    • -1
    • +1
    ./inputformat/TextReachibilityVertexInputFormat.java
    • -2
    • +2
    ./inputformat/TextShortestPathsInputFormat.java
    • -0
    • +36
    ./io/BooleanWritable.java
    • -0
    • +35
    ./io/ByteWritable.java
    • -0
    • +37
    ./io/BytesWritable.java
  1. … 50 more files in changeset.
making test cases smaller

  1. … 9 more files in changeset.
add message overflow support

    • -0
    • +113
    ./MessageOverflowFixedsizeVertex.java
    • -0
    • +112
    ./MessageOverflowVertex.java
    • -66
    • +0
    ./data/VLongNormalizedKeyComputer.java
    • -1
    • +1
    ./inputformat/TextConnectedComponentsInputFormat.java
    • -2
    • +2
    ./inputformat/TextPageRankInputFormat.java
    • -1
    • +1
    ./inputformat/TextReachibilityVertexInputFormat.java
    • -2
    • +2
    ./inputformat/TextShortestPathsInputFormat.java
    • -0
    • +66
    ./lib/data/VLongNormalizedKeyComputer.java
    • -0
    • +36
    ./lib/io/BooleanWritable.java
  1. … 75 more files in changeset.
add/update license headers

    • -3
    • +3
    ./inputformat/TextConnectedComponentsInputFormat.java
    • -3
    • +3
    ./inputformat/TextPageRankInputFormat.java
    • -3
    • +3
    ./inputformat/TextReachibilityVertexInputFormat.java
    • -3
    • +3
    ./inputformat/TextShortestPathsInputFormat.java
    • -1
    • +1
    ./maximalclique/AdjacencyListWritable.java
    • -0
    • +14
    ./maximalclique/MaximalCliqueAggregator.java
    • -1
    • +1
    ./maximalclique/MaximalCliqueVertex.java
  1. … 2273 more files in changeset.
add normalized key computer support in Pregelix

    • -0
    • +66
    ./data/VLongNormalizedKeyComputer.java
    • -0
    • +2
    ./maximalclique/MaximalCliqueVertex.java
    • -0
    • +2
    ./trianglecounting/TriangleCountingVertex.java
  1. … 30 more files in changeset.
polish pregelix tests

  1. … 1 more file in changeset.
reintegrate fullstack_dynamic_deployment

    • -2
    • +2
    ./inputformat/TextPageRankInputFormat.java
    • -2
    • +23
    ./maximalclique/CliquesWritable.java
    • -25
    • +30
    ./maximalclique/MaximalCliqueVertex.java
  1. … 201 more files in changeset.
cross merge fullstack_release_candidate into trunk

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3208 123451ca-8445-de46-9d55-352943316053

    • -0
    • +99
    ./maximalclique/AdjacencyListWritable.java
    • -0
    • +138
    ./maximalclique/CliquesWritable.java
    • -0
    • +65
    ./maximalclique/MaximalCliqueAggregator.java
    • -0
    • +347
    ./maximalclique/MaximalCliqueVertex.java
    • -0
    • +113
    ./maximalclique/TextMaximalCliqueInputFormat.java
    • -0
    • +111
    ./trianglecounting/TextTriangleCountingInputFormat.java
    • -0
    • +41
    ./trianglecounting/TriangleCountingAggregator.java
    • -0
    • +153
    ./trianglecounting/TriangleCountingVertex.java
  1. … 888 more files in changeset.