Clone Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
VariableSizeFrame(VSizeFrame) support for Hyracks.

This patch replaced Frame/Accessor/Appender with the new API which

supports BigObject.

The ExternalSorter/TopKSorter/ExternalGroupSorter

have been implemented to support big object.

The Groupby && Join should work with BigObject also. But it will break the

memory budget when it encounter a big object. I will fix the memory

problem later in a separate CR.

The design about the frame allocation is

here:https://docs.google.com/presentation/d/15h9iQf5OYsgGZoQTbGHkj1yS2G9q2fd0s1lDAD1EJq0/edit?usp=sharing

Suggest review order:

Patch 12: It includes all of the sorting operators.

Patch 13: It applys the new IFrame API to all Hyracks codes.

Patch 14: Some bug fixes to pass all Asterix's tests.

Patch 15: Skip it!

Patch 16: Some bug fixes to the Asterix's tests in small frame setting.

Later Patch: address the comments

Change-Id: I2e08692078683f6f2cf17387e39037ad851fc05b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/234

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

  1. … 219 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 72 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.15

  1. … 72 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 72 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.14

  1. … 72 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 89 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.13

  1. … 89 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 89 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.12

  1. … 89 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 276 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 88 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.11

  1. … 88 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 88 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.10

  1. … 88 more files in changeset.
merge from zheilbron/hyracks_msr

  1. … 291 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 87 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.9

  1. … 87 more files in changeset.
[maven-release-plugin] rollback the release of fullstack-0.2.9

  1. … 87 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 87 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.9

  1. … 87 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 87 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.8

  1. … 87 more files in changeset.
[maven-release-plugin] rollback the release of fullstack-0.2.8

  1. … 87 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.8

  1. … 87 more files in changeset.
[maven-release-plugin] rollback the release of fullstack-0.2.8

  1. … 87 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 87 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.8

  1. … 87 more files in changeset.
[maven-release-plugin] prepare for next development iteration

  1. … 87 more files in changeset.
[maven-release-plugin] prepare release fullstack-0.2.7

  1. … 87 more files in changeset.
Merged master

  1. … 88 more files in changeset.