Clone Tools
  • last updated 18 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -114
    • +0
    ./pregelix/dataflow/std/FunctionCallOperatorDescriptor.java
    • -177
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorDescriptor.java
    • -265
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -123
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorDescriptor.java
    • -208
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorNodePushable.java
    • -332
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -283
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinOperatorNodePushable.java
    • -279
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -270
    • +0
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionOperatorNodePushable.java
    • -63
    • +0
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorDescriptor.java
    • -110
    • +0
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
    • -92
    • +0
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorDescriptor.java
    • -261
    • +0
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
  1. … 943 more files in changeset.
Support big vertex in Pregelix. --For those vertice beyond page size, we store them on HDFS as immutable files. --Updates on those big vertice will trigger creations of new immutable files.

Change-Id: I6b6f0528b6b5360c96dcdace1fa360d42c517f22

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/72

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>

  1. … 17 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -36
    • +0
    ./pregelix/dataflow/group/IClusteredAggregatorDescriptorFactory.java
    • -15
    • +13
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -6
    • +6
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -5
    • +5
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -3
    • +13
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
    • -5
    • +5
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
    • -0
    • +78
    ./pregelix/dataflow/std/collectors/SortMergeFrameReader.java
    • -0
    • +72
    ./pregelix/dataflow/std/connectors/MToNPartitioningMergingConnectorDescriptor.java
    • -0
    • +50
    ./pregelix/dataflow/std/group/ClusteredGroupOperatorDescriptor.java
    • -0
    • +79
    ./pregelix/dataflow/std/group/ClusteredGroupOperatorNodePushable.java
    • -0
    • +144
    ./pregelix/dataflow/std/group/ClusteredGroupWriter.java
  1. … 262 more files in changeset.
fixed issue 731, 740, and more

commit 8911cc529e72e2bb544d9b472d6e10f173d173af

Author: Young-Seok <kisskys@gmail.com>

Date: Sun May 18 11:28:28 2014 -0700

another fix for picking available index for leftouterjoin plan

commit 9bce43087615fee53613467a027833dd53e190f9

Merge: c8e85ac efab69f

Author: Young-Seok <kisskys@gmail.com>

Date: Sun May 11 22:22:10 2014 -0700

merged master to kisskys/left-outer-join-issue branch

commit c8e85aca31545c13b2a02ff6dc259943e2cf66ad

Author: Young-Seok <kisskys@gmail.com>

Date: Sun May 11 20:17:17 2014 -0700

changes for left-outer-join to pick available indexes

Change-Id: Ib0fc186bc9388802f95445edee92c428b3bb69cc

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/34

Reviewed-by: Inci Cetindil <icetindil@gmail.com>

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

    • -6
    • +6
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorDescriptor.java
    • -6
    • +6
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorDescriptor.java
    • -2
    • +2
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorDescriptor.java
    • -2
    • +2
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorDescriptor.java
  1. … 53 more files in changeset.
fix application lifecyle mgmt in hyracks nc

    • -2
    • +10
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -2
    • +10
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -4
    • +13
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -5
    • +13
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
  1. … 3 more files in changeset.
fix the pinned page issue during a node failure

    • -0
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -0
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -1
    • +2
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
  1. … 1 more file in changeset.
fix IIndexAccessor interface, add a boolean exclusiveMode parameter for the createSearchCursor method

    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
  1. … 37 more files in changeset.
fix file write race condition

    • -2
    • +2
    ./pregelix/dataflow/util/FunctionProxy.java
  1. … 23 more files in changeset.
disable in-place update for variable-sized updates

    • -19
    • +0
    ./pregelix/dataflow/util/CopyUpdateUtil.java
use in-place update for smaller sized updates

    • -1
    • +1
    ./pregelix/dataflow/util/CopyUpdateUtil.java
fix in-place update

    • -2
    • +10
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -4
    • +12
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -4
    • +12
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -2
    • +10
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
    • -12
    • +21
    ./pregelix/dataflow/util/CopyUpdateUtil.java
    • -7
    • +9
    ./pregelix/dataflow/util/FunctionProxy.java
    • -0
    • +21
    ./pregelix/dataflow/util/StorageType.java
  1. … 10 more files in changeset.
fix the issue found by Genomix P4 algorithm

    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -2
    • +2
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -2
    • +2
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
    • -4
    • +11
    ./pregelix/dataflow/util/CopyUpdateUtil.java
  1. … 9 more files in changeset.
fix fault-tolerance and error reporting to handle disk failures

    • -3
    • +2
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
  1. … 31 more files in changeset.
1. fix the node failure scenario in job scheduler; 2. add fault-tolerance support and tests in pregelix

    • -1
    • +9
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -1
    • +9
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -0
    • +7
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinOperatorNodePushable.java
    • -1
    • +9
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -0
    • +7
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionOperatorNodePushable.java
    • -1
    • +7
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
    • -1
    • +9
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
  1. … 14 more files in changeset.
avoid btree update code path when the target entry has sufficient space

    • -2
    • +2
    ./pregelix/dataflow/util/CopyUpdateUtil.java
    • -10
    • +14
    ./pregelix/dataflow/util/TupleDeserializer.java
  1. … 4 more files in changeset.
add runtime checks to improve pregelix debug-ability

    • -2
    • +2
    ./pregelix/dataflow/util/CopyUpdateUtil.java
    • -5
    • +5
    ./pregelix/dataflow/util/TupleDeserializer.java
  1. … 2 more files in changeset.
merge from zheilbron/hyracks_msr

    • -0
    • +36
    ./pregelix/dataflow/group/IClusteredAggregatorDescriptorFactory.java
    • -4
    • +22
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -7
    • +21
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -0
    • +17
    ./pregelix/dataflow/util/UpdateBuffer.java
  1. … 285 more files in changeset.
add runtime checks for out-of-bound reads

    • -107
    • +87
    ./pregelix/dataflow/util/TupleDeserializer.java
insert upon non-existing key

    • -1
    • +7
    ./pregelix/dataflow/util/UpdateBuffer.java
use upsert instead of update

    • -1
    • +1
    ./pregelix/dataflow/util/UpdateBuffer.java
add message overflow support

    • -0
    • +50
    ./pregelix/dataflow/group/ClusteredGroupOperatorDescriptor.java
    • -0
    • +79
    ./pregelix/dataflow/group/ClusteredGroupOperatorNodePushable.java
    • -0
    • +165
    ./pregelix/dataflow/group/ClusteredGroupWriter.java
    • -0
    • +36
    ./pregelix/dataflow/group/IClusteredAggregatorDescriptorFactory.java
    • -4
    • +22
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -7
    • +21
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -0
    • +17
    ./pregelix/dataflow/util/UpdateBuffer.java
  1. … 83 more files in changeset.
Merged master

    • -6
    • +3
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorDescriptor.java
    • -28
    • +22
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -26
    • +21
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -25
    • +20
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -0
    • +92
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorDescriptor.java
    • -0
    • +236
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
    • -4
    • +4
    ./pregelix/dataflow/util/UpdateBuffer.java
  1. … 82 more files in changeset.
fix inl right outer join

    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinOperatorNodePushable.java
  1. … 1 more file in changeset.
checkin the fixed bulk reload for Sattam to debug

    • -0
    • +2
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
  1. … 1 more file in changeset.
fix parameters

    • -2
    • +2
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorDescriptor.java
  1. … 2 more files in changeset.
for sattam to debug

    • -3
    • +3
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -3
    • +3
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorNodePushable.java
    • -3
    • +3
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -3
    • +3
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinOperatorNodePushable.java
    • -3
    • +3
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -3
    • +3
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionOperatorNodePushable.java
    • -3
    • +3
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
    • -4
    • +4
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
  1. … 2 more files in changeset.
add LSM support in pregelix

    • -5
    • +2
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorDescriptor.java
    • -2
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorDescriptor.java
  1. … 14 more files in changeset.
refactoring dataflow operators to be more general--suport both b-tree and lsm b-tree

    • -91
    • +0
    ./pregelix/dataflow/std/BTreeSearchFunctionUpdateOperatorDescriptor.java
    • -212
    • +0
    ./pregelix/dataflow/std/BTreeSearchFunctionUpdateOperatorNodePushable.java
    • -23
    • +17
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -11
    • +11
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorNodePushable.java
    • -22
    • +17
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -19
    • +14
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinOperatorNodePushable.java
    • -20
    • +15
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -18
    • +13
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionOperatorNodePushable.java
    • -0
    • +91
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorDescriptor.java
    • -0
    • +235
    ./pregelix/dataflow/std/TreeSearchFunctionUpdateOperatorNodePushable.java
    • -5
    • +5
    ./pregelix/dataflow/util/CopyUpdateUtil.java
    • -2
    • +2
    ./pregelix/dataflow/util/UpdateBuffer.java
  1. … 5 more files in changeset.
Pass a boolean argument to the bulkload to decides if checking for an empty index is needed.

    • -1
    • +1
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
  1. … 25 more files in changeset.
add/update license headers

    • -1
    • +1
    ./pregelix/dataflow/std/BTreeSearchFunctionUpdateOperatorDescriptor.java
    • -1
    • +1
    ./pregelix/dataflow/std/BTreeSearchFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorDescriptor.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorDescriptor.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopJoinOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopRightOuterJoinOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionFunctionUpdateOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/IndexNestedLoopSetUnionOperatorNodePushable.java
    • -1
    • +1
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorDescriptor.java
    • -1
    • +1
    ./pregelix/dataflow/std/TreeIndexBulkReLoadOperatorNodePushable.java
  1. … 2273 more files in changeset.