Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -807
    • +0
    ./optimizerts/hive/conf/hive-default.xml
    • -10
    • +0
    ./optimizerts/queries/h11_share_scan.hive
    • -40
    • +0
    ./optimizerts/queries/q12_shipping.hive
    • -27
    • +0
    ./optimizerts/queries/q13_customer_distribution.hive
    • -25
    • +0
    ./optimizerts/queries/q14_promotion_effect.hive
    • -53
    • +0
    ./optimizerts/queries/q16_parts_supplier_relationship.hive
    • -38
    • +0
    ./optimizerts/queries/q17_small_quantity_order_revenue.hive
    • -41
    • +0
    ./optimizerts/queries/q18_large_volume_customer.hive
  1. … 943 more files in changeset.
Added replicate operator with materialization

be more aggressive to find shared plans in ExtractCommonOperatorRule

- find all the isomorphic subgraphs instead of just the ones on join build branches

- while expanding candidates handle the operators with multiple inputs

- analyze the DAG to find all the operators that can be co-scheduled, and infer the dependencies between clusters

- based on the dependencies, decide which outputs of a replicate operator needs materialization

- if the shared branch needs materialization, and it consists of only trivial operators (such as assign, unnest, datasource scan), that branch is discarded from the candidates

- modified the replicate operator descriptor to materialize the input if needed, and read from the materialized file for the outputs that requires materialization

- removed redundant decor variables in group-by

- fixed a bug on computing live variables for unnest-map operator: if the operator does not propagate inputs, those input variables should not be live anymore

- fixed a bug in ComplexUnnestToProductRule

Change-Id: If221d1507844f9409bf1163f93b0c04ef5848578

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/86

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -46
    • +90
    ./optimizerts/results/q7_volume_shipping.plan
    • -10
    • +24
    ./optimizerts/results/u10_nestedloop_join.plan
    • -10
    • +22
    ./optimizerts/results/u3_union.plan
  1. … 42 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -38
    • +34
    ./optimizerts/results/q12_shipping.plan
    • -5
    • +5
    ./optimizerts/results/q13_customer_distribution.plan
    • -4
    • +4
    ./optimizerts/results/q16_parts_supplier_relationship.plan
    • -6
    • +6
    ./optimizerts/results/q17_small_quantity_order_revenue.plan
    • -7
    • +7
    ./optimizerts/results/q18_large_volume_customer.plan
    • -26
    • +22
    ./optimizerts/results/q1_pricing_summary_report.plan
    • -4
    • +4
    ./optimizerts/results/q20_potential_part_promotion.plan
    • -28
    • +20
    ./optimizerts/results/q21_suppliers_who_kept_orders_waiting.plan
    • -50
    • +46
    ./optimizerts/results/q22_global_sales_opportunity.plan
    • -10
    • +6
    ./optimizerts/results/q2_minimum_cost_supplier.plan
  1. … 262 more files in changeset.
updated hivestrix test case for running aggregation fix

    • -2
    • +2
    ./optimizerts/results/q21_suppliers_who_kept_orders_waiting.plan
merge from zheilbron/hyracks_msr

    • -2
    • +0
    ./optimizerts/queries/q12_shipping.hive
    • -2
    • +0
    ./optimizerts/queries/q18_large_volume_customer.hive
    • -2
    • +0
    ./optimizerts/queries/q1_pricing_summary_report.hive
    • -1
    • +0
    ./optimizerts/queries/q20_potential_part_promotion.hive
    • -1
    • +0
    ./optimizerts/queries/q5_local_supplier_volume.hive
    • -1
    • +4
    ./optimizerts/queries/u7_multi_join.hive
  1. … 277 more files in changeset.
fix the execution mode of operators and the location constraint of algebricks meta operator

    • -3
    • +3
    ./optimizerts/results/q12_shipping.plan
    • -2
    • +2
    ./optimizerts/results/q13_customer_distribution.plan
    • -2
    • +2
    ./optimizerts/results/q16_parts_supplier_relationship.plan
    • -4
    • +4
    ./optimizerts/results/q17_small_quantity_order_revenue.plan
    • -7
    • +7
    ./optimizerts/results/q18_large_volume_customer.plan
    • -3
    • +3
    ./optimizerts/results/q1_pricing_summary_report.plan
    • -7
    • +7
    ./optimizerts/results/q20_potential_part_promotion.plan
    • -10
    • +10
    ./optimizerts/results/q21_suppliers_who_kept_orders_waiting.plan
    • -6
    • +6
    ./optimizerts/results/q22_global_sales_opportunity.plan
    • -7
    • +7
    ./optimizerts/results/q2_minimum_cost_supplier.plan
  1. … 15 more files in changeset.
cleanup hivesterix poms and assembly

    • -69
    • +53
    ./runtimefunctionts/hive/conf/hive-default.xml
    • -0
    • +8
    ./runtimefunctionts/queries/u8_order_by.hive
    • -0
    • +25
    ./runtimefunctionts/results/u8_order_by.result
  1. … 11 more files in changeset.
migrate hivesterix to depend on hive-0.11.0

    • -2
    • +0
    ./optimizerts/queries/q12_shipping.hive
    • -2
    • +0
    ./optimizerts/queries/q18_large_volume_customer.hive
    • -2
    • +0
    ./optimizerts/queries/q1_pricing_summary_report.hive
    • -1
    • +0
    ./optimizerts/queries/q20_potential_part_promotion.hive
    • -1
    • +0
    ./optimizerts/queries/q5_local_supplier_volume.hive
    • -1
    • +4
    ./optimizerts/queries/u7_multi_join.hive
  1. … 60 more files in changeset.
add/update license headers

    • -0
    • +14
    ./optimizerts/hive/conf/hive-default.xml
    • -0
    • +14
    ./runtimefunctionts/hadoop/conf/core-site.xml
    • -0
    • +14
    ./runtimefunctionts/hadoop/conf/hdfs-site.xml
    • -0
    • +14
    ./runtimefunctionts/hadoop/conf/mapred-site.xml
    • -0
    • +14
    ./runtimefunctionts/hive/conf/hive-default.xml
    • -0
    • +14
    ./runtimefunctionts/hive/conf/topology.xml
    • -0
    • +14
    ./runtimefunctionts/logging.properties
  1. … 2279 more files in changeset.
add normalized key computer support in Pregelix

  1. … 37 more files in changeset.
fix hivesterix optimizer test

    • -1
    • +1
    ./runtimefunctionts/hive/conf/hive-default.xml
  1. … 1 more file in changeset.
fix for issue196

    • -2
    • +1
    ./runtimefunctionts/logging.properties
  1. … 9 more files in changeset.
cross merge fullstack_release_candidate into trunk

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3208 123451ca-8445-de46-9d55-352943316053

    • -0
    • +793
    ./optimizerts/hive/conf/hive-default.xml
    • -0
    • +10
    ./optimizerts/queries/h11_share_scan.hive
    • -0
    • +42
    ./optimizerts/queries/q12_shipping.hive
    • -0
    • +27
    ./optimizerts/queries/q13_customer_distribution.hive
    • -0
    • +28
    ./optimizerts/queries/q14_promotion_effect.hive
    • -0
    • +53
    ./optimizerts/queries/q16_parts_supplier_relationship.hive
    • -0
    • +38
    ./optimizerts/queries/q17_small_quantity_order_revenue.hive
    • -0
    • +43
    ./optimizerts/queries/q18_large_volume_customer.hive
  1. … 888 more files in changeset.
Merge fullstack_asterix_stabilization with fullstack_hyracks_result_distribution.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@3170 123451ca-8445-de46-9d55-352943316053

    • -8
    • +3
    ./runtimefunctionts/hive/conf/topology.xml
  1. … 190 more files in changeset.
Merged fullstack_asterix_stabilization -r 3157:3163

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_ioc@3165 123451ca-8445-de46-9d55-352943316053

    • -8
    • +3
    ./runtimefunctionts/hive/conf/topology.xml
  1. … 12 more files in changeset.
Merged fullstack_asterix_stabilization -r 2933:3157

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_ioc@3164 123451ca-8445-de46-9d55-352943316053

    • -0
    • +793
    ./optimizerts/hive/conf/hive-default.xml
    • -0
    • +10
    ./optimizerts/queries/h11_share_scan.hive
    • -0
    • +42
    ./optimizerts/queries/q12_shipping.hive
    • -0
    • +27
    ./optimizerts/queries/q13_customer_distribution.hive
    • -0
    • +28
    ./optimizerts/queries/q14_promotion_effect.hive
    • -0
    • +53
    ./optimizerts/queries/q16_parts_supplier_relationship.hive
    • -0
    • +38
    ./optimizerts/queries/q17_small_quantity_order_revenue.hive
    • -0
    • +43
    ./optimizerts/queries/q18_large_volume_customer.hive
  1. … 1153 more files in changeset.
1. update script to add result-distribution paramteres; 2. fix rack-aware scheduler for boundary cases

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3163 123451ca-8445-de46-9d55-352943316053

    • -8
    • +3
    ./runtimefunctionts/hive/conf/topology.xml
  1. … 8 more files in changeset.
Merge fullstack_asterix_stabilization into fullstack_hyracks_result_distribution.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@3124 123451ca-8445-de46-9d55-352943316053

    • -0
    • +793
    ./optimizerts/hive/conf/hive-default.xml
    • -0
    • +10
    ./optimizerts/queries/h11_share_scan.hive
    • -0
    • +42
    ./optimizerts/queries/q12_shipping.hive
    • -0
    • +27
    ./optimizerts/queries/q13_customer_distribution.hive
    • -0
    • +28
    ./optimizerts/queries/q14_promotion_effect.hive
    • -0
    • +53
    ./optimizerts/queries/q16_parts_supplier_relationship.hive
    • -0
    • +38
    ./optimizerts/queries/q17_small_quantity_order_revenue.hive
    • -0
    • +43
    ./optimizerts/queries/q18_large_volume_customer.hive
  1. … 692 more files in changeset.
rename hivesterix-core to hivesterix-dist

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_release_cleanup@3076 123451ca-8445-de46-9d55-352943316053

    • -0
    • +65
    ./logging.properties
    • -0
    • +793
    ./optimizerts/hive/conf/hive-default.xml
    • -0
    • +0
    ./optimizerts/ignore.txt
    • -0
    • +10
    ./optimizerts/queries/h11_share_scan.hive
    • -0
    • +6
    ./optimizerts/queries/h12_select_struct.hive
    • -0
    • +37
    ./optimizerts/queries/q10_returned_item.hive
    • -0
    • +47
    ./optimizerts/queries/q11_important_stock.hive
    • -0
    • +42
    ./optimizerts/queries/q12_shipping.hive
    • -0
    • +27
    ./optimizerts/queries/q13_customer_distribution.hive
    • -0
    • +28
    ./optimizerts/queries/q14_promotion_effect.hive
    • -0
    • +45
    ./optimizerts/queries/q15_top_supplier.hive
    • -0
    • +53
    ./optimizerts/queries/q16_parts_supplier_relationship.hive
    • -0
    • +38
    ./optimizerts/queries/q17_small_quantity_order_revenue.hive
    • -0
    • +43
    ./optimizerts/queries/q18_large_volume_customer.hive
  1. … 586 more files in changeset.