Clone Tools
  • last updated 17 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Move Pregelix and Hivesterix codebase to new repositories: 1. Move Pregelix codebase to https://github.com/pregelix/pregelix; 2. Move Hivesterix codebase to https://code.google.com/p/hivesterix .

Change-Id: Iede698fcb92a0ad0a7a4918ea69b54886fd64fc7

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/155

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Ian Maxon <imaxon@uci.edu>

    • -239
    • +0
    ./agg/AccumulatingAggregatorFactory.java
    • -46
    • +0
    ./agg/AggregationFunctionFactory.java
    • -230
    • +0
    ./agg/SerializableAggregateFunction.java
    • -40
    • +0
    ./agg/SerializableAggregationFunctionFactory.java
    • -103
    • +0
    ./agg/SerializableAggregatorDescriptorFactory.java
    • -36
    • +0
    ./bootstrap/IndexLifeCycleManagerProvider.java
    • -43
    • +0
    ./bootstrap/NCApplicationEntryPoint.java
    • -52
    • +0
    ./bootstrap/StorageManagerInterface.java
    • -41
    • +0
    ./bootstrap/VirtualBufferCacheProvider.java
    • -319
    • +0
    ./function/ComputeUpdateFunctionFactory.java
    • -95
    • +0
    ./function/ExtractLiveVertexIdFunctionFactory.java
    • -69
    • +0
    ./function/NoOpUpdateFunctionFactory.java
    • -309
    • +0
    ./function/StartComputeUpdateFunctionFactory.java
    • -114
    • +0
    ./touchpoint/DatatypeHelper.java
  1. … 943 more files in changeset.
1. fix asterixdb issue 782 --- push nested pipeline before a nested group-by operator into the combiner group-by operator in the AbstractIntroduceGroupByCombinerRule --- add a processNullTest abstract method in the AbstractIntroduceGroupByCombinerRule -- fix the join order in a subplan 2. allow user-configurable buffer cache page size (B-tree page size) in Pregelix

commit 4d9a11d0c05281a41bbabe03066478fe851b3a2b

Author: buyingyi <buyingyi@gmail.com>

Change-Id: Ib7761370df8606c55ac34c126554319586e824f0

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/64

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -1
    • +5
    ./bootstrap/NCApplicationEntryPoint.java
  1. … 8 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -0
    • +239
    ./agg/AccumulatingAggregatorFactory.java
    • -0
    • +172
    ./agg/AggregationFunction.java
    • -0
    • +46
    ./agg/AggregationFunctionFactory.java
    • -0
    • +230
    ./agg/SerializableAggregateFunction.java
    • -0
    • +40
    ./agg/SerializableAggregationFunctionFactory.java
    • -0
    • +103
    ./agg/SerializableAggregatorDescriptorFactory.java
    • -4
    • +15
    ./function/ComputeUpdateFunctionFactory.java
    • -5
    • +16
    ./function/StartComputeUpdateFunctionFactory.java
    • -228
    • +0
    ./simpleagg/AccumulatingAggregatorFactory.java
    • -152
    • +0
    ./simpleagg/AggregationFunction.java
    • -46
    • +0
    ./simpleagg/AggregationFunctionFactory.java
    • -16
    • +11
    ./touchpoint/VertexIdPartitionComputerFactory.java
    • -1
    • +1
    ./touchpoint/WritableSerializerDeserializerFactory.java
  1. … 262 more files in changeset.
1. add deployment retry 2. support plan switch

    • -0
    • +95
    ./function/ExtractLiveVertexIdFunctionFactory.java
  1. … 4 more files in changeset.
NodeControllers clean up appEntryPoints on shutdown (2nd try)

    • -2
    • +4
    ./bootstrap/NCApplicationEntryPoint.java
  1. … 1 more file in changeset.
support multiple user-defined global aggregators

    • -12
    • +20
    ./function/ComputeUpdateFunctionFactory.java
    • -12
    • +20
    ./function/StartComputeUpdateFunctionFactory.java
  1. … 34 more files in changeset.
fix IIndexAccessor interface, add a boolean exclusiveMode parameter for the createSearchCursor method

    • -7
    • +3
    ./function/ComputeUpdateFunctionFactory.java
    • -7
    • +3
    ./function/StartComputeUpdateFunctionFactory.java
  1. … 42 more files in changeset.
fix file write race condition

    • -0
    • +4
    ./function/ComputeUpdateFunctionFactory.java
    • -40
    • +16
    ./touchpoint/DatatypeHelper.java
    • -13
    • +23
    ./touchpoint/VertexIdPartitionComputerFactory.java
    • -2
    • +3
    ./touchpoint/WritableSerializerDeserializerFactory.java
  1. … 19 more files in changeset.
fix in-place update

    • -3
    • +12
    ./function/ComputeUpdateFunctionFactory.java
    • -1
    • +3
    ./function/NoOpUpdateFunctionFactory.java
    • -2
    • +12
    ./function/StartComputeUpdateFunctionFactory.java
  1. … 14 more files in changeset.
checkpoint: added support on running aggregation using group-by runtime. Aggregator interface is also updated in order to handle both accumulating and running aggregation.

    • -2
    • +3
    ./simpleagg/AccumulatingAggregatorFactory.java
  1. … 20 more files in changeset.
fix fault-tolerance and error reporting to handle disk failures

    • -3
    • +3
    ./touchpoint/RecoveryRuntimeHookFactory.java
  1. … 31 more files in changeset.
support large-size global aggreate values

    • -2
    • +6
    ./function/ComputeUpdateFunctionFactory.java
    • -3
    • +7
    ./function/StartComputeUpdateFunctionFactory.java
    • -1
    • +1
    ./touchpoint/PreSuperStepRuntimeHookFactory.java
    • -0
    • +57
    ./touchpoint/RecoveryRuntimeHookFactory.java
  1. … 20 more files in changeset.
add job concatenation support

    • -4
    • +1
    ./function/StartComputeUpdateFunctionFactory.java
    • -4
    • +4
    ./touchpoint/PreSuperStepRuntimeHookFactory.java
  1. … 8 more files in changeset.
add runtime checks to improve pregelix debug-ability

    • -0
    • +1
    ./touchpoint/MsgListNullWriterFactory.java
  1. … 4 more files in changeset.
Implemented k-buffering for lsm indexes. Also add a fix for issues 589 and 594.

    • -2
    • +4
    ./bootstrap/VirtualBufferCacheProvider.java
  1. … 93 more files in changeset.
merge from zheilbron/hyracks_msr

    • -4
    • +14
    ./function/ComputeUpdateFunctionFactory.java
    • -2
    • +8
    ./function/StartComputeUpdateFunctionFactory.java
    • -41
    • +126
    ./simpleagg/AccumulatingAggregatorFactory.java
    • -20
    • +51
    ./simpleagg/AggregationFunction.java
    • -3
    • +4
    ./simpleagg/AggregationFunctionFactory.java
  1. … 287 more files in changeset.
inherit Hadoop's VLongWritable format

    • -1
    • +5
    ./function/ComputeUpdateFunctionFactory.java
  1. … 3 more files in changeset.
a temp fix for size estimation

    • -2
    • +2
    ./simpleagg/AccumulatingAggregatorFactory.java
refactoring

    • -31
    • +31
    ./function/ComputeUpdateFunctionFactory.java
    • -21
    • +21
    ./function/StartComputeUpdateFunctionFactory.java
addd partition early termination support

    • -27
    • +31
    ./function/ComputeUpdateFunctionFactory.java
    • -18
    • +22
    ./function/StartComputeUpdateFunctionFactory.java
  1. … 10 more files in changeset.
address Vinayak's comments on open/close of vertex

    • -2
    • +8
    ./function/ComputeUpdateFunctionFactory.java
    • -0
    • +2
    ./function/StartComputeUpdateFunctionFactory.java
    • -22
    • +32
    ./simpleagg/AccumulatingAggregatorFactory.java
    • -0
    • +20
    ./simpleagg/AggregationFunction.java
  1. … 6 more files in changeset.
add message overflow support

    • -43
    • +118
    ./simpleagg/AccumulatingAggregatorFactory.java
    • -20
    • +31
    ./simpleagg/AggregationFunction.java
    • -3
    • +4
    ./simpleagg/AggregationFunctionFactory.java
  1. … 87 more files in changeset.
add LSM support in pregelix

    • -0
    • +39
    ./bootstrap/VirtualBufferCacheProvider.java
    • -0
    • +67
    ./function/NoOpUpdateFunctionFactory.java
  1. … 14 more files in changeset.
add the support for customized partitioner

    • -0
    • +73
    ./touchpoint/VertexPartitionComputerFactory.java
  1. … 15 more files in changeset.
add/update license headers

    • -0
    • +14
    ./bootstrap/IndexLifeCycleManagerProvider.java
    • -0
    • +14
    ./bootstrap/NCApplicationEntryPoint.java
    • -1
    • +1
    ./bootstrap/StorageManagerInterface.java
    • -3
    • +3
    ./function/ComputeUpdateFunctionFactory.java
    • -3
    • +3
    ./function/StartComputeUpdateFunctionFactory.java
    • -1
    • +1
    ./simpleagg/AccumulatingAggregatorFactory.java
    • -1
    • +1
    ./simpleagg/AggregationFunctionFactory.java
    • -1
    • +1
    ./touchpoint/MergePartitionComputerFactory.java
    • -3
    • +3
    ./touchpoint/MsgListNullWriterFactory.java
    • -3
    • +3
    ./touchpoint/PostSuperStepRuntimeHookFactory.java
    • -3
    • +3
    ./touchpoint/PreSuperStepRuntimeHookFactory.java
    • -3
    • +3
    ./touchpoint/VertexIdNullWriterFactory.java
  1. … 2273 more files in changeset.
reduce unnecessary B-Tree (non-inplace) update

    • -7
    • +4
    ./function/ComputeUpdateFunctionFactory.java
    • -7
    • +4
    ./function/StartComputeUpdateFunctionFactory.java
    • -56
    • +0
    ./touchpoint/VLongAscNormalizedKeyComputerFactory.java
    • -23
    • +0
    ./touchpoint/VLongDescNormalizedKeyComputerFactory.java
  1. … 2 more files in changeset.
add normalized key computer support in Pregelix

    • -0
    • +51
    ./touchpoint/VertexIdNormalizedKeyComputerFactory.java
  1. … 37 more files in changeset.
reintegrate fullstack_dynamic_deployment

    • -1
    • +1
    ./function/ComputeUpdateFunctionFactory.java
    • -1
    • +1
    ./function/StartComputeUpdateFunctionFactory.java
    • -2
    • +2
    ./simpleagg/AccumulatingAggregatorFactory.java
    • -2
    • +4
    ./simpleagg/AggregationFunctionFactory.java
    • -0
    • +126
    ./touchpoint/DatatypeHelper.java
    • -1
    • +1
    ./touchpoint/PreSuperStepRuntimeHookFactory.java
    • -3
    • +5
    ./touchpoint/VertexIdPartitionComputerFactory.java
    • -0
    • +21
    ./touchpoint/WritableSerializerDeserializerFactory.java
  1. … 194 more files in changeset.
Merged fullstack_lsm_staging upto r3336

git-svn-id: https://hyracks.googlecode.com/svn/trunk/fullstack@3339 123451ca-8445-de46-9d55-352943316053

    • -0
    • +22
    ./bootstrap/IndexLifeCycleManagerProvider.java
    • -0
    • +23
    ./bootstrap/NCApplicationEntryPoint.java
    • -0
    • +12
    ./bootstrap/StorageManagerInterface.java
    • -35
    • +0
    ./bootstrap/TreeIndexRegistryProvider.java
    • -1
    • +5
    ./function/ComputeUpdateFunctionFactory.java
    • -1
    • +5
    ./function/StartComputeUpdateFunctionFactory.java
  1. … 892 more files in changeset.
reformat the code in in r3269

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_asterix_stabilization@3270 123451ca-8445-de46-9d55-352943316053

    • -245
    • +229
    ./function/ComputeUpdateFunctionFactory.java
    • -251
    • +234
    ./function/StartComputeUpdateFunctionFactory.java