ProActive File Transfer

package org.objectweb.proactive.api.ProFileTransfer;

static       RemoteFile pull(Node srcNode, File srcFile, File dstFile) throws IOException;
static List<RemoteFile> pull(Node srcNode, File[] srcFile, File[] dstFile) throws IOException;

static       RemoteFile push(File srcFile, Node dstNode, File dstFile) throws IOException;
static List<RemoteFile> push(File[] srcFile, Node dstNode, File[] dstFile) throws IOException;

static       RemoteFile transfer(Node srcNode, File srcFile, Node dstNode, File dstFile) throws
 IOException;
static List<RemoteFile> transfer(Node srcNode, File[] srcFile, Node dstNode, File[] dstFile)
 throws IOException;
static List<RemoteFile> transfer(Node srcNode, File[] srcFile, Node dstNode, File[] dstFile, int
 bsize, int numFlyingBlocks) throws IOException;

static       RemoteFile mkdirs(Node node, File file);

These methods are static, and handle the transfer of file between ProActive Nodes. The pull methods retrieve a file/directory located on a remote machine to the local machine. The push methods transfer a file/directory available on the local node to the specified remote node. The transfer methods allow transfer between third party nodes. The mkdirs method creates a directory on the remote machine.

The file transfers are performed in an asynchronous fashion. Each of these methods returns at RemoteFile object which represents the file transfer operation and the remote file's node/location. When these methods are invoked a RemoteFile instance is immediately returned, before the file transfer operation has been completed. The RemoteFile object provides a way to monitor the status of the file transfer:

public interface RemoteFile extends Serializable {

  public boolean isFinished();

  public void waitFor() throws IOException;

  public RemoteFile pull(File localDst) throws IOException;

  public RemoteFile push(Node dstNode, File dstRemote) throws IOException;

  public Node getRemoteNode();

  public File getRemoteFilePath();

  public boolean delete() throws IOException;
    
  public boolean exists() throws IOException;
    
  public boolean isDirectory() throws IOException;
    
  public boolean isFile() throws IOException;
}

The isFinished and waitFor can be invoked to query and wait on the file transfer status. The pull method can be used to fetch the RemoteFile from the remote node, into the local Node, and the push methods can be used to send the RemoteFile to another Node.

In the following example, a Node is deployed using a descriptor file. A File is then pushed from the local Node to a remote Node. Then the file is pulled from the Remote Node, and the computation is blocked waiting for the file transfer operation to finish.

ProActiveDescriptor pad = PADeployment.getProactiveDescriptor(XML_LOCATION);

VirtualNode testVNode = pad.getVirtualNode("test");
testVNode.activate();
Node testnode = testVNode.getNode();

RemoteFile remoteFileA = ProFileTransfer.push(new File("/tmp/test.dat"), testnode, new File(
"/tmp/test.dat"));
RemoteFile remoteFileB = remoteFileA.pull(new File("/tmp/test2.dat"));


remoteFileB.waitFor(); //blocking method

Object o  =  PAActiveObject.newActive(...);
...
Node node = PAActiveObject.getActiveObjectNode(o);

The File Transfer related tags, are placed inside the descriptor at three different parts (or levels).

The first one corresponds to the fileTransferDefinitions tag, which contains a list of FileTransfer definitions. A FileTransfer definition is a high level representation of the File Transfer, containing mainly the file names. It is created in such a way, that no low level information such as: hosts, protocols, prefix is present (this is the role of the low level representation). The following example shows a FileTranfer definition named example:

....
</deployment>
<fileTransferDefinitions>
   <fileTransfer id="example">
      <file src="hello.dat" dest="world.dat"/>
      <file src="hello.jar" dest="world.jar"/>
      <file src="hello.class" dest="world.class"/>
      <dir src="exampledir" dest="exampledir"/>
  </fileTransfer>
  <fileTransfer id="anotherExample">
      ...
  </fileTransfer>
  ...
</fileTransferDefinitions>
<infrastructure>
....         

The FileTransfer definitions can be referenced through their names, from the VirtualNode tags using two attributes:fileTransferDeploy and fileTransferRetrieve. The first one, corresponds to the file transfer that will take place at deployment time, and the second one corresponds to the file transfer that the user will trigger once the user application is done.

<virtualNode name="exampleVNode" fileTransferDeploy="example" fileTransferRetrieve="example"/>

All the low level information such as: hosts, username, protocols, prefix, etc... is declared inside each process. Both fileTransferDeploy and fileTransferRetrieve are specified separetly using a refid attribute. The refid can be a direct reference to a FileTransfer definition, or set using the keyword implicit. If implicit is used, then the reference will be inherited from the corresponding VirtualNode. In the following example both mechanisms (Deploy and Retrieve) reference indirectly and directly the example definition:

<processDefinition id="xyz">
  <sshProcess>
  ...  
<!-- Inside the process, the FileTransfer tag becomes an element instead of
an attribute.  This happens because FileTransfer information is process specific.
Note that the destination hostname and username can be omitted,
and implicitly inferred from the process information. -->

    <fileTransferDeploy refid="implicit"> <!-- referenceID or keyword "implicit" (inherit)-->
      <copyProtocol>processDefault, rcp, scp, pft</copyProtocol>
      <sourceInfo prefix="/home/user"/>
      <destinationInfo prefix="/tmp" hostname="foo.org" username="smith" />
    </fileTransferDeploy>

    <fileTransferRetrieve refid="example">
      <sourceInfo prefix="/tmp"/>
      <destinationInfo prefix="/home/user"/>
    </fileTransferRetrieve>
  </sshProcess>
</processDefinition>

In the example above, fileTransferDeploy has an implicit refid. This means that the File Transfer definitions used will be inherited from the VirtualNode. The first element shown inside this tag corresponds to copyProtocol. The copyProtocol tag specified the sequence of protocols that will be executed to achieve the FileTransfer at deployment time. Notice the processDefault keyword, which specifies the usage of the default copy protocol associated with this process. In the case of the example, this corresponds to an sshProcess and therefore the Secure Copy Protocol (scp) will be tried first. To complement the higher level File Transfer definition, other information can be specified as attributes in the sourceInfo and destinationInfo elements. For the case of FileTransferDeploy, these tags currently correspond to: prefix, hostname and username.

For fileTransferRetrieve, no copyProtocol needs to be specified. ProActive will use it's internal mechanism to transfer the files. This implies that no hostname or username are required.

import org.objectweb.proactive.core.descriptor.data;

public FileWrapper VirtualNode.fileTransferRetrieve();

import org.objectweb.proactive.core.descriptor.data;

pad = PADeployment.getProactiveDescriptor(XML_LOCATION);

VirtualNode testVNode = pad.getVirtualNode("example");
testVNode.activate();
Node[] examplenode = testVNode.getNodes();

...

FileWrapper fw = testVNode.fileTransferRetrieve();
...
File f[]=fw.getFiles() //wait-for-files to arrive

This definitions can be referenced from a VirtualNode. They contain the most basic information of a FileTransfer:

References from the VirtualNode are made using the unique definition name.

These definitions contain more architecture specific information, and are therefore contained within the Process:

If some of this information (like username or hostname) can be inferred from the process, it is not necessary to declare it in the definition. Optionally, the information contained in the protocol can be overridden if specified.

Figure 21.1. File Transfer Design

When a FileTransfer starts, both abstract and concrete information are merged using the FileTransfer Workshop. The result of this process correspons to a sequence of CopyProtocols, as specified in the Concrete Definition.

Each CopyProtocol will be tried before the deployment takes place, until one succeeds. After one succeed are all fail, the process deployment will take place.

The File Transfer API is built on top of ProActive's active object and future file asynchronism model. When pulling or pushing a file from a Node, two service Active Objects (AO) are created. One is placed on the local machine and the otherone on the remote site. The file is then split into blocks, and transfered over the network using remote invocations between these two AO.

For a given virtualnode, a File Transfer pull will take place with all the nodes deployed from this virtualnode. The detailes of the specified file transfer will correspond to the ones present in the descriptor file.