Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

  1. … 943 more files in changeset.
1. make limit operator work in nested plan 2. avoid send-side materialize but use receive-side materialize to reduce cc/nc communications.

Change-Id: I1a90c70be0514fcc0b286afa456664618d68910f

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/80

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

  1. … 1 more file in changeset.
Support big vertex in Pregelix. --For those vertice beyond page size, we store them on HDFS as immutable files. --Updates on those big vertice will trigger creations of new immutable files.

Change-Id: I6b6f0528b6b5360c96dcdace1fa360d42c517f22

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/72

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

  1. … 15 more files in changeset.
1. fix asterixdb issue 782 --- push nested pipeline before a nested group-by operator into the combiner group-by operator in the AbstractIntroduceGroupByCombinerRule --- add a processNullTest abstract method in the AbstractIntroduceGroupByCombinerRule -- fix the join order in a subplan 2. allow user-configurable buffer cache page size (B-tree page size) in Pregelix

commit 4d9a11d0c05281a41bbabe03066478fe851b3a2b

Author: buyingyi <buyingyi@gmail.com>

Change-Id: Ib7761370df8606c55ac34c126554319586e824f0

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/64

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 8 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 270 more files in changeset.
add support for Hadoop Counters via job.setCounterAggregatorClass

The PregelixJob.setCounterAggregatorClass sets up a (user-specified)

global aggregator and an iterationComplete hook to save Counter values.

The user-specified Counter-based aggregator (must extend

HadoopCountersAggregator) is saved to HDFS in each iteration and should

be restart/snapshot-aware.

The usage for setting up counters is to make a call to

job.setCounterAggregatorClass. After job completion, the Counters may

be retrieved from HDFS using BspUtils.getCounters(job).

Note that there is currently only one spot for iterationComplete hooks

and this behavior occupies it.

  1. … 7 more files in changeset.
add an "iteration complete" hook for aggregation/reporting across iterations

This commit allows the user to specify a class which will be called upon

completion of each pregelix iteration. This allows us to perform a user-

specified action between iterations.

As an example, a PerIterationGlobalAggregatesHook is provided which

saves the complete set of global aggregator states from every iteration,

allowing the user to observe aggregates from all iterations.

The default hook does nothing.

The hook instance is attached directly to the PregelixJob so that it can

be retrieved by the Driver's caller.

  1. … 6 more files in changeset.
added ability to trigger an application state dump through the rest api

  1. … 21 more files in changeset.
fix application lifecyle mgmt in hyracks nc

  1. … 6 more files in changeset.
fix the pinned page issue during a node failure

  1. … 3 more files in changeset.
fix an issue found by Sattam

support multiple user-defined global aggregators

  1. … 34 more files in changeset.
fix file write race condition

  1. … 22 more files in changeset.
cleanup pregelix job context

add pagerank multi-job test

  1. … 3 more files in changeset.
fix the duplicate jobid issue

  1. … 1 more file in changeset.
support multiple concurrent jobs

    • -0
    • +55
    ./main/java/edu/uci/ics/pregelix/dataflow/context/TaskID.java
  1. … 7 more files in changeset.
fix fault-tolerance and error reporting to handle disk failures

  1. … 28 more files in changeset.
1. fix the node failure scenario in job scheduler; 2. add fault-tolerance support and tests in pregelix

  1. … 20 more files in changeset.
fix the checkpoint writing for vertex checkpointing and message checkpointing

  1. … 4 more files in changeset.
Merge branch 'master' into salsubaiee/master_lsm

support large-size global aggreate values

  1. … 20 more files in changeset.
implemented checkpoint read

    • -0
    • +106
    ./main/java/edu/uci/ics/pregelix/dataflow/KeyValueWriterFactory.java
  1. … 24 more files in changeset.
add vertex checkpointing support

  1. … 10 more files in changeset.
add job concatenation support

  1. … 8 more files in changeset.
Implemented k-buffering for lsm indexes. Also add a fix for issues 589 and 594.

  1. … 93 more files in changeset.
merge from zheilbron/hyracks_msr

  1. … 290 more files in changeset.
addd partition early termination support

  1. … 11 more files in changeset.
making test cases smaller

  1. … 10 more files in changeset.
Merged master

  1. … 88 more files in changeset.