Clone Tools
  • last updated a few minutes ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -1557
    • +0
    ./java/org/apache/hadoop/hive/ql/Driver.java
    • -50
    • +0
    ./resources/conf/cluster.properties
    • -26
    • +0
    ./resources/conf/debugnc.properties
  1. … 943 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

  1. … 276 more files in changeset.
merge from zheilbron/hyracks_msr

    • -282
    • +514
    ./java/org/apache/hadoop/hive/ql/Driver.java
    • -773
    • +0
    ./resources/conf/hive-default.xml
    • -1
    • +1
    ./resources/conf/hive-log4j.properties
    • -0
    • +5189
    ./resources/conf/hive-site.xml
  1. … 280 more files in changeset.
fix hive script to support various hadoop versions

clean up the package and console output

add debugging utility scripts

    • -0
    • +7
    ./resources/scripts/copylog.sh
    • -0
    • +12
    ./resources/scripts/dumpAll.sh
    • -0
    • +15
    ./resources/scripts/dumptrace.sh
  1. … 3 more files in changeset.
cleanup hivesterix poms and assembly

    • -773
    • +0
    ./resources/conf/hive-default.xml
    • -1
    • +1
    ./resources/conf/hive-log4j.properties
    • -0
    • +5189
    ./resources/conf/hive-site.xml
  1. … 9 more files in changeset.
migrate hivesterix to depend on hive-0.11.0

    • -282
    • +514
    ./java/org/apache/hadoop/hive/ql/Driver.java
  1. … 71 more files in changeset.
check in lsm support and test for sattam to debug

  1. … 21 more files in changeset.
add LSM support in pregelix

  1. … 14 more files in changeset.
add/update license headers

    • -0
    • +14
    ./java/org/apache/hadoop/hive/ql/Driver.java
    • -0
    • +14
    ./resources/conf/cluster.properties
    • -0
    • +14
    ./resources/conf/debugnc.properties
  1. … 2273 more files in changeset.
reintegrate fullstack_dynamic_deployment

    • -0
    • +122
    ./resources/scripts/hivesterixcc
    • -0
    • +117
    ./resources/scripts/hivesterixnc
    • -16
    • +18
    ./resources/scripts/startCluster.sh
    • -3
    • +21
    ./resources/scripts/startDebugNc.sh
  1. … 190 more files in changeset.
Merged fullstack_lsm_staging upto r3336

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3339 123451ca-8445-de46-9d55-352943316053

  1. … 898 more files in changeset.
cross merge fullstack_release_candidate into trunk

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3208 123451ca-8445-de46-9d55-352943316053

    • -0
    • +1310
    ./java/org/apache/hadoop/hive/ql/Driver.java
    • -0
    • +37
    ./resources/conf/cluster.properties
    • -0
    • +12
    ./resources/conf/debugnc.properties
  1. … 888 more files in changeset.
fix code and scripts for rack-awareness

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_release_candidate@3181 123451ca-8445-de46-9d55-352943316053

    • -0
    • +7
    ./resources/conf/topology-template.xml
  1. … 11 more files in changeset.
merged from fullstack_asterix_stabilization to fullstack_lsm_staging -r3100:3171

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_lsm_staging@3173 123451ca-8445-de46-9d55-352943316053

  1. … 22 more files in changeset.
Merge fullstack_asterix_stabilization with fullstack_hyracks_result_distribution.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@3170 123451ca-8445-de46-9d55-352943316053

  1. … 187 more files in changeset.
Merged fullstack_asterix_stabilization -r 3157:3163

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_ioc@3165 123451ca-8445-de46-9d55-352943316053

  1. … 11 more files in changeset.
Merged fullstack_asterix_stabilization -r 2933:3157

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_ioc@3164 123451ca-8445-de46-9d55-352943316053

    • -0
    • +1310
    ./java/org/apache/hadoop/hive/ql/Driver.java
    • -0
    • +37
    ./resources/conf/cluster.properties
    • -0
    • +12
    ./resources/conf/debugnc.properties
  1. … 1153 more files in changeset.
1. update script to add result-distribution paramteres; 2. fix rack-aware scheduler for boundary cases

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3163 123451ca-8445-de46-9d55-352943316053

  1. … 7 more files in changeset.
add a path for the concurrency issue in Hive LazyObjectInspectorFactory

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3149 123451ca-8445-de46-9d55-352943316053

Merge fullstack_asterix_stabilization into fullstack_hyracks_result_distribution.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@3124 123451ca-8445-de46-9d55-352943316053

    • -0
    • +1310
    ./java/org/apache/hadoop/hive/ql/Driver.java
    • -0
    • +37
    ./resources/conf/cluster.properties
    • -0
    • +12
    ./resources/conf/debugnc.properties
    • -0
    • +758
    ./resources/conf/hive-default.xml
  1. … 692 more files in changeset.
1. add a concurrent hash map patch for hive serde2; 2. minimize the hyracks cc history size

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3118 123451ca-8445-de46-9d55-352943316053

    • -0
    • +128
    ./java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java
  1. … 5 more files in changeset.
fix getip.sh for hivesterix

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3117 123451ca-8445-de46-9d55-352943316053

clean up hivesterix client ip/port issues

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3115 123451ca-8445-de46-9d55-352943316053

  1. … 2 more files in changeset.
merge r3038:3100 fullstack_asterix_stabilization -> fullstack_lsm_staging

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_lsm_staging@3106 123451ca-8445-de46-9d55-352943316053

  1. … 5 more files in changeset.
clean up hivesterix scripts' classpath

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3100 123451ca-8445-de46-9d55-352943316053

    • -110
    • +0
    ./resources/scripts/pregelix.bat
  1. … 1 more file in changeset.
refactoring hivesterix codebase

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_release_cleanup@3080 123451ca-8445-de46-9d55-352943316053

    • -0
    • +24
    ./resources/conf/configuration.xsl
    • -0
    • +758
    ./resources/conf/hive-default.xml
    • -0
    • +58
    ./resources/conf/hive-log4j.properties
    • -0
    • +16
    ./resources/scripts/startCluster.sh
  1. … 9 more files in changeset.
update scripts

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_release_cleanup@3079 123451ca-8445-de46-9d55-352943316053

    • -0
    • +37
    ./resources/conf/cluster.properties
    • -0
    • +12
    ./resources/conf/debugnc.properties
    • -0
    • +1
    ./resources/conf/master
    • -0
    • +1
    ./resources/conf/slaves
    • -0
    • +28
    ./resources/scripts/ext/cli.sh
    • -0
    • +36
    ./resources/scripts/ext/help.sh
    • -0
    • +35
    ./resources/scripts/ext/hiveserver.sh
    • -0
    • +50
    ./resources/scripts/ext/hwi.sh
    • -0
    • +47
    ./resources/scripts/ext/jar.sh
    • -0
    • +38
    ./resources/scripts/ext/lineage.sh
    • -0
    • +35
    ./resources/scripts/ext/metastore.sh
    • -0
    • +27
    ./resources/scripts/ext/rcfilecat.sh
    • -0
    • +32
    ./resources/scripts/ext/util/execHiveCmd.sh
    • -0
    • +21
    ./resources/scripts/getip.sh
    • -0
    • +213
    ./resources/scripts/hive
  1. … 17 more files in changeset.
rename hivesterix-core to hivesterix-dist

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_release_cleanup@3076 123451ca-8445-de46-9d55-352943316053

    • -0
    • +26
    ./assembly/binary-assembly.xml
    • -0
    • +1310
    ./java/org/apache/hadoop/hive/ql/Driver.java
    • -0
    • +170
    ./java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java
    • -0
    • +272
    ./java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java
  1. … 589 more files in changeset.