Clone
 

madhusudancs <madhusudancs@123451ca-8445-de46-9d55-352943316053> in asterixdb

Merge fullstack_rsd_fixes from r3309:r3314.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_lsm_staging@3315 123451ca-8445-de46-9d55-352943316053

Merge fullstack_lsm_staging r3309:r3313.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_rsd_fixes@3314 123451ca-8445-de46-9d55-352943316053

Enforce ordering in exposing the directory records for partitions to the clients if ordering constraint is imposed by the ResultWriterOperatoryDescriptor.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2550 123451ca-8445-de46-9d55-352943316053

Provide the ordering specification to result writer operator descriptor in SortMergeTest.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2549 123451ca-8445-de46-9d55-352943316053

Use a slightly enhanced datastructure to store whether the results should be ordered or not.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2548 123451ca-8445-de46-9d55-352943316053

Add a parameter for the ResultWriterOperatorDescriptor to support ordered result distribution.

Let this parameter percolate throughout the layers of indirection and the

network until it hits the dataset directory service.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2547 123451ca-8445-de46-9d55-352943316053

Remove the unused IDatasetInputChannel interface definition file.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2546 123451ca-8445-de46-9d55-352943316053

We do not have a need to define a frame size statically within the dataset now because we obtain that from the job specification.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2545 123451ca-8445-de46-9d55-352943316053

Use IInputChannel instead of IDatasetInputChannel in the monitor definitions.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2544 123451ca-8445-de46-9d55-352943316053

Construct the DatasetClientContext object in the HyracksDataset constructor and pass it to the result channel's open method.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2543 123451ca-8445-de46-9d55-352943316053

Implement a client side context from IHyracksCommonContext.

Notice that we return null from the getIOManger method because

clients don't really need to have any IOManager. So the method

is just a stub to fulfill the interface requirements.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2542 123451ca-8445-de46-9d55-352943316053

Extend IDatasetInputChannelMonitor from IInputChannelMonitor rather than creating a new one altogether.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2541 123451ca-8445-de46-9d55-352943316053

Reimplement the DatasetNetworkInputChannel using the IInputChannel interface instead of creating a new interface defintion.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2540 123451ca-8445-de46-9d55-352943316053

Refactor IInputChannel.open method to take IHyracksCommonContext as the parameter instead of IHyracksTaskContext.

We do this because, all that the open method needs from the context

is to allocate a new ByteBuffer and that interface is exposed by

common context.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2539 123451ca-8445-de46-9d55-352943316053

Just use null as values to localAddress and listener for non listening TCP endpoints instead of copy pasting the constructor method.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2537 123451ca-8445-de46-9d55-352943316053

Fix a formatting issue with the comment.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2536 123451ca-8445-de46-9d55-352943316053

Update the sort merge test to use the new API.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2535 123451ca-8445-de46-9d55-352943316053

Use DatasetDirectoryRecords instead of NetworkAddresses everywhere we access that information in the dataset stack.

Also pass the result id through every method that works either on fetching

the metadata for results or the results themselves.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2534 123451ca-8445-de46-9d55-352943316053

Add support for registering the result set id with the ResultWriterOperatorDescriptor.

Along with that pass the result set id information all through the stack to

register the partition address along with this result set id in the dataset

directory service.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2533 123451ca-8445-de46-9d55-352943316053

Support adding result set ids to the job specification and retrieving it.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2532 123451ca-8445-de46-9d55-352943316053

Implement a class that holds a directory record.

A record contains the address location and some metadata.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2531 123451ca-8445-de46-9d55-352943316053

Implement ResultSetId class to represent result sets.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2530 123451ca-8445-de46-9d55-352943316053

We should print the buffer before recycling it

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2526 123451ca-8445-de46-9d55-352943316053

Synchronize reads and writes to not let the reader attempt to read the file before it is written by the writer.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2525 123451ca-8445-de46-9d55-352943316053

Formatting fixes.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2521 123451ca-8445-de46-9d55-352943316053

Make all the necessary changes or additions to report the data transfer info across the new result distribution socket for the node controller.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2520 123451ca-8445-de46-9d55-352943316053

Update the tests to set the IPAddress for result distribution socket.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2519 123451ca-8445-de46-9d55-352943316053

Update all the imports with the new locations and fix the method calls with argument changes.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2518 123451ca-8445-de46-9d55-352943316053

Complete the implementation of basic DatasetPartitionWriter class to work with the network layer.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2517 123451ca-8445-de46-9d55-352943316053

Update the ResultWriterOperatorDescriptor to work with the new network layer implementation.

git-svn-id: https://hyracks.googlecode.com/svn/branches/fullstack_hyracks_result_distribution@2516 123451ca-8445-de46-9d55-352943316053