Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort 1,3,and 4 is to reduce the job overhead. 2 is to reduce the unecessary NC resource consumptions such as memory and files. 5 and 6 are improvements to runtime operators.
One change in algebricks: -- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator
Several important changes in pregelix: -- remove static states in vertex -- direct check halt bit without deserialization -- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array
Loading ...