Clone Tools
  • last updated 13 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Integrated more native MR operators and a Shuffle Connector

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1244 123451ca-8445-de46-9d55-352943316053

    • -0
    • +22
    ./collectors/IPartitionAcceptor.java
    • -0
    • +24
    ./collectors/IPartitionBatchManager.java
    • -0
    • +77
    ./collectors/InputChannelFrameReader.java
    • -0
    • +163
    ./collectors/NonDeterministicChannelReader.java
    • -0
    • +53
    ./collectors/NonDeterministicFrameReader.java
    • -0
    • +78
    ./collectors/NonDeterministicPartitionBatchManager.java
    • -228
    • +0
    ./collectors/NonDeterministicPartitionCollector.java
    • -0
    • +81
    ./collectors/PartitionCollector.java
    • -0
    • +79
    ./collectors/SortMergeFrameReader.java
    • -275
    • +0
    ./collectors/SortMergePartitionCollector.java
    • -2
    • +7
    ./connectors/LocalityAwareMToNPartitioningConnectorDescriptor.java
    • -2
    • +6
    ./connectors/MToNPartitioningConnectorDescriptor.java
    • -3
    • +13
    ./connectors/MToNPartitioningMergingConnectorDescriptor.java
    • -2
    • +6
    ./connectors/MToNReplicatingConnectorDescriptor.java
    • -2
    • +6
    ./connectors/OneToOneConnectorDescriptor.java
  1. … 15 more files in changeset.
Added the ability for connectors to indicate source partitions that make it to a given target

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1212 123451ca-8445-de46-9d55-352943316053

    • -0
    • +7
    ./base/AbstractMToNConnectorDescriptor.java
    • -0
    • +7
    ./connectors/OneToOneConnectorDescriptor.java
  1. … 1 more file in changeset.
Added map-based connector.

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1148 123451ca-8445-de46-9d55-352943316053

    • -0
    • +50
    ./connectors/GlobalHashingLocalityMap.java
    • -0
    • +72
    ./connectors/HashtableLocalityMap.java
    • -0
    • +27
    ./connectors/ILocalityMap.java
    • -0
    • +82
    ./connectors/LocalityAwareMToNPartitioningConnectorDescriptor.java
    • -0
    • +125
    ./connectors/LocalityAwarePartitionDataWriter.java
Added support code for JobDetails page in admin console

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1140 123451ca-8445-de46-9d55-352943316053

    • -1
    • +1
    ./base/AbstractConnectorDescriptor.java
    • -1
    • +1
    ./base/AbstractOperatorDescriptor.java
  1. … 5 more files in changeset.
fix a memory recycle issue in sorter's memory manager

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_sort_join_opts@1135 123451ca-8445-de46-9d55-352943316053

    • -0
    • +1
    ./sort/OptimizedExternalSortRunGenerator.java
    • -0
    • +1
    ./sort/OptimizedExternalSortRunGeneratorWithLimit.java
fix the memory recycle issue in FrameSorter

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_sort_join_opts@1131 123451ca-8445-de46-9d55-352943316053

    • -1
    • +2
    ./sort/ExternalSortOperatorDescriptor.java
    • -29
    • +32
    ./sort/ExternalSortRunMerger.java
    • -29
    • +21
    ./sort/OptimizedExternalSortRunGenerator.java
  1. … 8 more files in changeset.
Fixed a bug on the merging phase of the external grouper; added a float sum aggregator (with test cases).

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1087 123451ca-8445-de46-9d55-352943316053

    • -13
    • +13
    ./group/ExternalGroupOperatorDescriptor.java
    • -0
    • +158
    ./group/aggregators/FloatSumFieldAggregatorFactory.java
  1. … 1 more file in changeset.
Optimized thread usage in CC. Reduced sizes of critical sections in PartitionCollector

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1067 123451ca-8445-de46-9d55-352943316053

    • -18
    • +16
    ./collectors/NonDeterministicPartitionCollector.java
  1. … 5 more files in changeset.
merge hyracks_dev_next r978:1064

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@1065 123451ca-8445-de46-9d55-352943316053

    • -0
    • +4
    ./group/ExternalGroupOperatorDescriptor.java
  1. … 2 more files in changeset.
Added get/set display name to Operators and Connectors

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1053 123451ca-8445-de46-9d55-352943316053

    • -0
    • +12
    ./base/AbstractConnectorDescriptor.java
    • -0
    • +12
    ./base/AbstractOperatorDescriptor.java
  1. … 2 more files in changeset.
Added more logging

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@1029 123451ca-8445-de46-9d55-352943316053

    • -0
    • +16
    ./collectors/NonDeterministicPartitionCollector.java
  1. … 1 more file in changeset.
Fixed bugs for key maintanance codes in groupers (after being moved out of the aggregators).

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@1024 123451ca-8445-de46-9d55-352943316053

    • -0
    • +4
    ./group/HashSpillableTableFactory.java
implementing writing keys to output in group-by operator rather than aggregators

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@983 123451ca-8445-de46-9d55-352943316053

    • -0
    • +4
    ./group/ExternalGroupOperatorDescriptor.java
    • -0
    • +6
    ./group/HashSpillableTableFactory.java
  1. … 1 more file in changeset.
add explicit system.gc() call at the beginning of merge phase

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@982 123451ca-8445-de46-9d55-352943316053

    • -1
    • +6
    ./group/ExternalGroupOperatorDescriptor.java
reformat the source code

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@981 123451ca-8445-de46-9d55-352943316053

    • -2
    • +2
    ./file/AbstractFileWriteOperatorDescriptor.java
    • -5
    • +1
    ./file/DelimitedDataTupleParserFactory.java
    • -2
    • +2
    ./file/FileScanOperatorDescriptor.java
    • -2
    • +2
    ./file/FrameFileWriterOperatorDescriptor.java
    • -2
    • +9
    ./file/PlainFileWriterOperatorDescriptor.java
    • -4
    • +7
    ./file/RecordFileScanOperatorDescriptor.java
    • -1
    • +1
    ./group/DeserializedPreclusteredGroupOperator.java
    • -158
    • +84
    ./group/ExternalGroupOperatorDescriptor.java
    • -42
    • +28
    ./group/FrameToolsForGroupers.java
    • -238
    • +214
    ./group/GroupingHashTable.java
    • -34
    • +18
    ./group/HashGroupOperatorDescriptor.java
    • -395
    • +332
    ./group/HashSpillableTableFactory.java
    • -4
    • +4
    ./group/IAggregatorDescriptorFactory.java
  1. … 17 more files in changeset.
merge hyracks_dev_next r847:977

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@978 123451ca-8445-de46-9d55-352943316053

    • -1
    • +0
    ./base/AbstractConnectorDescriptor.java
    • -1
    • +0
    ./base/AbstractOperatorDescriptor.java
    • -0
    • +1
    ./collectors/NonDeterministicPartitionCollector.java
    • -3
    • +3
    ./file/AbstractDeserializedFileScanOperatorDescriptor.java
    • -3
    • +3
    ./file/PlainFileWriterOperatorDescriptor.java
    • -219
    • +238
    ./group/GroupingHashTable.java
    • -322
    • +322
    ./group/HashSpillableTableFactory.java
    • -141
    • +153
    ./group/PreclusteredGroupWriter.java
  1. … 141 more files in changeset.
Updated the aggregator interfaces to use ArrayTupleBuilder; removed the state length calculator.

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@977 123451ca-8445-de46-9d55-352943316053

    • -63
    • +55
    ./group/ExternalGroupOperatorDescriptor.java
    • -103
    • +91
    ./group/HashSpillableTableFactory.java
    • -9
    • +0
    ./group/IFieldAggregateDescriptor.java
    • -1
    • +1
    ./group/PreclusteredGroupOperatorDescriptor.java
    • -19
    • +22
    ./group/PreclusteredGroupWriter.java
    • -74
    • +0
    ./group/aggregators/MinMaxStringFieldAggregatorFactory.java
    • -130
    • +16
    ./group/aggregators/MultiFieldsAggregatorFactory.java
  1. … 2 more files in changeset.
Removed getter and setter from AggregateState

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@972 123451ca-8445-de46-9d55-352943316053

    • -18
    • +18
    ./group/aggregators/MinMaxStringFieldAggregatorFactory.java
Major changes: - Based on Yingyi's review comments, the new interface is written to have the frame output logic out of the aggregators. - Added methods for better management of the size of aggregation states. Now aggregator developer can decide the memory size to be used, and the groupers will assign the memory/frame spaces based on these information. - Fixed some bugs in multiple-field aggregator.

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@971 123451ca-8445-de46-9d55-352943316053

    • -34
    • +75
    ./group/ExternalGroupOperatorDescriptor.java
    • -0
    • +107
    ./group/FrameToolsForGroupers.java
    • -63
    • +135
    ./group/HashSpillableTableFactory.java
    • -37
    • +0
    ./group/IAggregateStateFactory.java
    • -12
    • +29
    ./group/IFieldAggregateDescriptor.java
    • -31
    • +65
    ./group/PreclusteredGroupWriter.java
    • -21
    • +78
    ./group/aggregators/MinMaxStringFieldAggregatorFactory.java
    • -113
    • +169
    ./group/aggregators/MultiFieldsAggregatorFactory.java
  1. … 1 more file in changeset.
Removed the initFromPartial() method from the new aggregator interface; added AvgFieldMergeAggregatorFactory for the merge phase of AVG().

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@967 123451ca-8445-de46-9d55-352943316053

    • -2
    • +2
    ./group/ExternalGroupOperatorDescriptor.java
    • -19
    • +0
    ./group/IFieldAggregateDescriptor.java
    • -0
    • +190
    ./group/aggregators/AvgFieldGroupAggregatorFactory.java
    • -0
    • +195
    ./group/aggregators/AvgFieldMergeAggregatorFactory.java
  1. … 2 more files in changeset.
Adding Optimization for the merge step in external sort

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_sort_join_opts@965 123451ca-8445-de46-9d55-352943316053

    • -2
    • +3
    ./sort/OptimizedExternalSortOperatorDescriptor.java
Pre-merge cleanup: removed old aggregator interfaces, and updated codes to use the new interface.

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@963 123451ca-8445-de46-9d55-352943316053

    • -702
    • +0
    ./aggregations/ExternalGroupOperatorDescriptor.java
    • -231
    • +0
    ./aggregations/GroupingHashTable.java
    • -209
    • +0
    ./aggregations/HashGroupOperatorDescriptor.java
    • -502
    • +0
    ./aggregations/HashSpillableTableFactory.java
    • -37
    • +0
    ./aggregations/IAggregateStateFactory.java
    • -134
    • +0
    ./aggregations/IAggregatorDescriptor.java
    • -32
    • +0
    ./aggregations/IAggregatorDescriptorFactory.java
    • -133
    • +0
    ./aggregations/IFieldAggregateDescriptor.java
    • -32
    • +0
    ./aggregations/IFieldAggregateDescriptorFactory.java
    • -39
    • +0
    ./aggregations/ISpillableTable.java
    • -31
    • +0
    ./aggregations/ISpillableTableFactory.java
    • -100
    • +0
    ./aggregations/PreclusteredGroupOperatorDescriptor.java
    • -134
    • +0
    ./aggregations/PreclusteredGroupWriter.java
    • -219
    • +0
    ./aggregations/aggregators/AvgFieldAggregatorFactory.java
  1. … 48 more files in changeset.
Fixed bugs on key fields arrangement when partial grouping values are stored; added test cases to benchmark the new interface.

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@962 123451ca-8445-de46-9d55-352943316053

    • -1
    • +9
    ./aggregations/ExternalGroupOperatorDescriptor.java
    • -1
    • +12
    ./aggregations/GroupingHashTable.java
    • -1
    • +7
    ./aggregations/HashSpillableTableFactory.java
    • -1
    • +1
    ./aggregations/IAggregatorDescriptorFactory.java
    • -1
    • +1
    ./aggregations/PreclusteredGroupOperatorDescriptor.java
    • -0
    • +1
    ./aggregations/PreclusteredGroupWriter.java
    • -8
    • +10
    ./aggregations/aggregators/MultiFieldsAggregatorFactory.java
  1. … 3 more files in changeset.
Added new IPC mechanism to Hyracks. Migrated all remote communications to use new IPC layer

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@935 123451ca-8445-de46-9d55-352943316053

    • -0
    • +1
    ./collectors/NonDeterministicPartitionCollector.java
  1. … 54 more files in changeset.
reverting changes

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_sort_join_opts@930 123451ca-8445-de46-9d55-352943316053

    • -28
    • +1
    ./misc/PrinterOperatorDescriptor.java
Removing unneccessary class

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_sort_join_opts@928 123451ca-8445-de46-9d55-352943316053

    • -1
    • +28
    ./misc/PrinterOperatorDescriptor.java
    • -228
    • +0
    ./sort/OptimizedExternalSortRunMerger.java
  1. … 7 more files in changeset.
Addressing Reviewer's comments

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_sort_join_opts@927 123451ca-8445-de46-9d55-352943316053

    • -1
    • +1
    ./join/OptimizedHybridHashJoinOperatorDescriptor.java
    • -16
    • +188
    ./sort/ExternalSortRunMerger.java
    • -6
    • +3
    ./sort/OptimizedExternalSortOperatorDescriptor.java
    • -14
    • +2
    ./sort/OptimizedExternalSortRunGenerator.java
    • -18
    • +2
    ./sort/OptimizedExternalSortRunGeneratorWithLimit.java
  1. … 1 more file in changeset.
Added Job timeline view

git-svn-id: https://hyracks.googlecode.com/svn/branches/hyracks_dev_next@897 123451ca-8445-de46-9d55-352943316053

    • -1
    • +0
    ./base/AbstractConnectorDescriptor.java
    • -1
    • +0
    ./base/AbstractOperatorDescriptor.java
  1. … 17 more files in changeset.
Update issue #52: rewrote GroupingHashTable to use aggregator for key fields output; fixed a bug of incorrect initialization in pre-cluster grouper; tested all integration tests for new aggregation interfaces with larger dataset (so external grouper spills).

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@893 123451ca-8445-de46-9d55-352943316053

    • -11
    • +10
    ./aggregations/GroupingHashTable.java
    • -2
    • +3
    ./aggregations/PreclusteredGroupWriter.java
Update #52: fixed bugs on external grouper on merging phase; fixed bugs on aggregation integration tests.

git-svn-id: https://hyracks.googlecode.com/svn/branches/aggregators_dev_next@883 123451ca-8445-de46-9d55-352943316053

    • -13
    • +2
    ./aggregations/ExternalGroupOperatorDescriptor.java
    • -5
    • +13
    ./aggregations/GroupingHashTable.java
    • -2
    • +0
    ./aggregations/HashSpillableTableFactory.java
    • -1
    • +6
    ./aggregations/IAggregateStateFactory.java
    • -0
    • +21
    ./aggregations/IAggregatorDescriptor.java
    • -4
    • +16
    ./aggregations/IFieldAggregateDescriptor.java
    • -15
    • +18
    ./aggregations/aggregators/MinMaxStringFieldAggregatorFactory.java
    • -8
    • +48
    ./aggregations/aggregators/MultiFieldsAggregatorFactory.java
  1. … 1 more file in changeset.