Clone Tools
  • last updated 15 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Change folder structure for Java repackage

Change only the folders, not the files, for our package name change.

This will break the build, and needs to be followed by a change to

the package name in all of the source files. However performing

the folder move and file change in two steps lets Git understand

that the files are the same, and lets us track revisions across

those files.

Change-Id: I08aff75e25ac7c6298c32cf3402febbc4a318c2a

Reviewed-on: https://asterix-gerrit.ics.uci.edu/307

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

    • -47
    • +0
    ./api/application/IApplicationContext.java
    • -22
    • +0
    ./api/application/IStateDumpHandler.java
    • -23
    • +0
    ./api/channels/IInputChannelMonitor.java
    • -45
    • +0
    ./api/client/ClusterControllerInfo.java
    • -293
    • +0
    ./api/client/HyracksClientInterfaceFunctions.java
    • -130
    • +0
    ./api/client/HyracksClientInterfaceRemoteProxy.java
    • -189
    • +0
    ./api/client/HyracksConnection.java
    • -187
    • +0
    ./api/client/IHyracksClientConnection.java
    • -55
    • +0
    ./api/client/IHyracksClientInterface.java
  1. … 3879 more files in changeset.
Introducing data replication API to LSM indexes

Change-Id: I80565fc9d74e30440d2df5917911904ba8f33c25

Reviewed-on: https://asterix-gerrit.ics.uci.edu/322

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>

    • -0
    • +27
    ./api/replication/IIOReplicationManager.java
    • -0
    • +45
    ./api/replication/IReplicationJob.java
    • -0
    • +54
    ./api/replication/impl/AbstractReplicationJob.java
  1. … 37 more files in changeset.
Add Apache RAT License Auditor plugin, and fix missing licenses

Change-Id: I39d92ec6654c73b4e6b8ba76dd66770bb60c7b79

Reviewed-on: https://asterix-gerrit.ics.uci.edu/260

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Chris Hillery <ceej@lambda.nu>

Reviewed-by: Till Westmann <tillw@apache.org>

    • -0
    • +14
    ./api/application/IStateDumpHandler.java
    • -0
    • +14
    ./api/dataflow/connectors/SendSidePipeliningReceiveSideMaterializedBlockingConnectorPolicy.java
    • -0
    • +14
    ./api/resources/memory/IMemoryManager.java
  1. … 30 more files in changeset.
VariableSizeFrame(VSizeFrame) support for Hyracks.

This patch replaced Frame/Accessor/Appender with the new API which

supports BigObject.

The ExternalSorter/TopKSorter/ExternalGroupSorter

have been implemented to support big object.

The Groupby && Join should work with BigObject also. But it will break the

memory budget when it encounter a big object. I will fix the memory

problem later in a separate CR.

The design about the frame allocation is

here:https://docs.google.com/presentation/d/15h9iQf5OYsgGZoQTbGHkj1yS2G9q2fd0s1lDAD1EJq0/edit?usp=sharing

Suggest review order:

Patch 12: It includes all of the sorting operators.

Patch 13: It applys the new IFrame API to all Hyracks codes.

Patch 14: Some bug fixes to pass all Asterix's tests.

Patch 15: Skip it!

Patch 16: Some bug fixes to the Asterix's tests in small frame setting.

Later Patch: address the comments

Change-Id: I2e08692078683f6f2cf17387e39037ad851fc05b

Reviewed-on: https://asterix-gerrit.ics.uci.edu/234

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -0
    • +59
    ./api/comm/FixedSizeFrame.java
    • -0
    • +60
    ./api/comm/IFrame.java
    • -0
    • +55
    ./api/comm/IFrameAppender.java
    • -0
    • +47
    ./api/comm/IFrameFieldAppender.java
    • -10
    • +15
    ./api/comm/IFrameTupleAccessor.java
    • -0
    • +39
    ./api/comm/IFrameTupleAppender.java
    • -0
    • +31
    ./api/comm/NoShrinkVSizeFrame.java
    • -0
    • +76
    ./api/comm/VSizeFrame.java
    • -9
    • +1
    ./api/context/IHyracksCommonContext.java
    • -0
    • +39
    ./api/context/IHyracksFrameMgrContext.java
    • -3
    • +2
    ./api/dataset/IHyracksDatasetReader.java
  1. … 205 more files in changeset.
- Fixed Type Casting issue - Reorganized duplicated internal class in the DelimitedDataParser and DelimitedDataParserFactory - Prevented a user from creating an inverted index on a dataset with a variable-length PK

Change-Id: Ic5606501223b8d860b49a258ff49afacd7d76b9a

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/191

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -0
    • +30
    ./api/dataflow/value/BinaryComparatorConstant.java
    • -3
    • +5
    ./api/dataflow/value/IBinaryHashFunction.java
  1. … 64 more files in changeset.
This change allows setting a local ordering property for assign operators. it is needed when variables that are created in the assign operator are sorted

The following commits from your working branch will be included:

commit 9e4abb36e6f50e0f73406f4603cdc79590ca7b06

Author: Abdullah Alamoudi <bamousaa@gmail.com>

Date: Mon Feb 2 16:53:45 2015 +0300

added a TODO for deserialization

commit 78f7ee6fb04358d36156658202fb4478e47059e2

Author: Abdullah Alamoudi <bamousaa@gmail.com>

Date: Mon Feb 2 16:30:56 2015 +0300

fixed NPE in substitue variable visitor

commit 6bb101e072e03aae2732613385589e1ae52b510c

Author: Abdullah Alamoudi <bamousaa@gmail.com>

Date: Mon Feb 2 15:07:14 2015 +0300

Allow assign operator to have explicitly set ordering properties

Change-Id: If995b47aa3c97ce60be952141699268341d31eea

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/214

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>

  1. … 3 more files in changeset.
Add ability to configure network ports for NCs and CC.

This includes the ability to specify different "public" ports, for situations

when a node is behind a NAT firewall.

Also eliminate IP check for data and dataset network addresses, which

should allow DNS names to be used as well.

Some internal cleanup regarding handling network addresses.

Change-Id: I9947fe3cec59daef3458cdc14d33c9353449da27

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/204

Reviewed-by: Ian Maxon <imaxon@uci.edu>

Tested-by: Ian Maxon <imaxon@uci.edu>

  1. … 25 more files in changeset.
Add Shutdown via API to Hyracks

This change adds a method to HyracksConnection called stopCluster().

When the CC recieves a message from this, it asks all NC tasks to close

and acknowledge that they have recieved the message and are closing.

If all NCs have closed, or a 10 second timeout elapses, the CC then

exits with a 0 return code if all NCs closed, or a 1 if some did

not acknowledge the shutdown request.

Change-Id: Iaf3d395dc7964e114d4929830f40063f58e0d5da

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/76

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Vinayak Borkar <vinayakb@gmail.com>

    • -1
    • +12
    ./api/client/HyracksClientInterfaceFunctions.java
    • -0
    • +16
    ./api/client/HyracksClientInterfaceRemoteProxy.java
    • -0
    • +5
    ./api/client/IHyracksClientConnection.java
    • -0
    • +2
    ./api/client/IHyracksClientInterface.java
  1. … 19 more files in changeset.
Several major changes in hyracks: -- reduced CC/NC communications for reporting partition request and availability; partition request/availability are only reported for the case of send-side materialized (without pipelining) policies in case of task re-attempt. -- changed buffer cache to dynamically allocate memory based on needs instead of pre-allocating -- changed each network channel to lazily allocate memory based on needs, and changed materialized connectors to lazily allocate files based on needs -- changed several major CCNCCFunctions to use non-java serde -- added a sort-based group-by operator which pushes group-by aggregations into an external sort -- make external sort a stable sort

1,3,and 4 is to reduce the job overhead.

2 is to reduce the unecessary NC resource consumptions such as memory and files.

5 and 6 are improvements to runtime operators.

One change in algebricks:

-- implemented a rule to push group-by aggregation into sort, i.e., using the sort-based gby operator

Several important changes in pregelix:

-- remove static states in vertex

-- direct check halt bit without deserialization

-- optimize the sort algorithm by packing yet-another 2-byte normalized key into the tPointers array

Change-Id: Id696f9a9f1647b4a025b8b33d20b3a89127c60d6

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/35

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Till Westmann <westmann@gmail.com>

    • -0
    • +1
    ./api/client/impl/JobSpecificationActivityClusterGraphGeneratorFactory.java
    • -1
    • +26
    ./api/dataflow/ConnectorDescriptorId.java
    • -2
    • +27
    ./api/dataflow/OperatorDescriptorId.java
    • -0
    • +66
    ./api/dataflow/connectors/ConnectorPolicyFactory.java
    • -0
    • +26
    ./api/dataflow/connectors/SendSidePipeliningReceiveSideMaterializedBlockingConnectorPolicy.java
    • -0
    • +30
    ./api/io/IWritable.java
    • -0
    • +23
    ./api/job/ActivityClusterGraph.java
  1. … 262 more files in changeset.
reduced communication during result distribution - when reporting the location of results, the NCs also report if the result partition is empty - the client does not try to read empty partitions better toString() for subclasses of AbstractWork

Change-Id: Ia39f657e689ea305d49d55bd27c9a512e1ff970f

Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/39

Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Reviewed-by: Yingyi Bu <buyingyi@gmail.com>

    • -2
    • +17
    ./api/dataset/DatasetDirectoryRecord.java
  1. … 22 more files in changeset.
Code performance readability update for JSON partition constraints.

Made the JSON output more readable.

Added constraints to the operator JSON.

Added constraints to the operator JSON.

adressed code review comments

fix for issue #732

  1. … 15 more files in changeset.
Merge branch 'master' into pouria/fix-memory

Conflicts:

hyracks/hyracks-api/src/main/java/edu/uci/ics/hyracks/api/context/IHyracksCommonContext.java

hyracks/hyracks-client/src/main/java/edu/uci/ics/hyracks/client/dataset/DatasetClientContext.java

hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/edu/uci/ics/hyracks/storage/am/lsm/invertedindex/ondisk/OnDiskInvertedIndex.java

hyracks/hyracks-test-support/src/main/java/edu/uci/ics/hyracks/test/support/TestTaskContext.java

    • -1
    • +1
    ./api/context/IHyracksCommonContext.java
  1. … 3 more files in changeset.
Fixing Methods signature

    • -1
    • +1
    ./api/context/IHyracksCommonContext.java
  1. … 3 more files in changeset.
formatting

  1. … 4 more files in changeset.
reformatting

    • -1
    • +1
    ./api/context/IHyracksCommonContext.java
    • -1
    • +1
    ./api/dataflow/value/IPredicateEvaluator.java
    • -1
    • +0
    ./api/job/IJobSerializerDeserializer.java
    • -1
    • +1
    ./api/partitions/ResultSetPartitionId.java
  1. … 6 more files in changeset.
reformatting

Merge branch 'master' into zheilbron/hyracks_msr_demo

Conflicts:

hyracks/hyracks-api/src/main/java/edu/uci/ics/hyracks/api/lifecycle/LifeCycleComponentManager.java

  1. … 1 more file in changeset.
removed unnecessary fields in JobInfo

  1. … 1 more file in changeset.
added ability to trigger an application state dump through the rest api

    • -0
    • +8
    ./api/application/IStateDumpHandler.java
    • -0
    • +2
    ./api/lifecycle/ILifeCycleComponent.java
  1. … 17 more files in changeset.
fix application lifecyle mgmt in hyracks nc

  1. … 6 more files in changeset.
changes to tracked locations assigned to an operator for each partition

  1. … 2 more files in changeset.
fix file write race condition

    • -1
    • +1
    ./api/job/IJobSerializerDeserializerContainer.java
    • -3
    • +8
    ./api/job/JobSerializerDeserializerContainer.java
  1. … 22 more files in changeset.
fix fault-tolerance and error reporting to handle disk failures

    • -2
    • +12
    ./api/exceptions/HyracksDataException.java
  1. … 31 more files in changeset.
1. simply activity cluster planner for all-producers-to-all-consumers kind of connectors; 2. lower the NC task threads' priority

    • -0
    • +5
    ./api/dataflow/IConnectorDescriptor.java
  1. … 7 more files in changeset.
changes for memory management

    • -0
    • +2
    ./api/context/IHyracksCommonContext.java
  1. … 5 more files in changeset.