Clone Tools
  • last updated 25 mins ago
Constraints: committers
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to; 2. Move Hivesterix codebase to .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7


Tested-by: Jenkins <>

Reviewed-by: Ian Maxon <>

  1. … 957 more files in changeset.
Added replicate operator with materialization

be more aggressive to find shared plans in ExtractCommonOperatorRule

- find all the isomorphic subgraphs instead of just the ones on join build branches

- while expanding candidates handle the operators with multiple inputs

- analyze the DAG to find all the operators that can be co-scheduled, and infer the dependencies between clusters

- based on the dependencies, decide which outputs of a replicate operator needs materialization

- if the shared branch needs materialization, and it consists of only trivial operators (such as assign, unnest, datasource scan), that branch is discarded from the candidates

- modified the replicate operator descriptor to materialize the input if needed, and read from the materialized file for the outputs that requires materialization

- removed redundant decor variables in group-by

- fixed a bug on computing live variables for unnest-map operator: if the operator does not propagate inputs, those input variables should not be live anymore

- fixed a bug in ComplexUnnestToProductRule

Change-Id: If221d1507844f9409bf1163f93b0c04ef5848578


Tested-by: Jenkins <>

Reviewed-by: Yingyi Bu <>

  1. … 44 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6


Tested-by: Jenkins <>

Reviewed-by: Till Westmann <>

  1. … 276 more files in changeset.
merge from zheilbron/hyracks_msr

  1. … 291 more files in changeset.
fix the execution mode of operators and the location constraint of algebricks meta operator

  1. … 29 more files in changeset.
migrate hivesterix to depend on hive-0.11.0

  1. … 74 more files in changeset.
cross merge fullstack_release_candidate into trunk

git-svn-id: 123451ca-8445-de46-9d55-352943316053

  1. … 902 more files in changeset.
Merged fullstack_asterix_stabilization -r 2933:3157

git-svn-id: 123451ca-8445-de46-9d55-352943316053

  1. … 1167 more files in changeset.
Merge fullstack_asterix_stabilization into fullstack_hyracks_result_distribution.

git-svn-id: 123451ca-8445-de46-9d55-352943316053

  1. … 706 more files in changeset.
rename hivesterix-core to hivesterix-dist

git-svn-id: 123451ca-8445-de46-9d55-352943316053

    • -0
    • +192
  1. … 600 more files in changeset.