org.apache.nutch.fs
Class NDFSFileSystem

java.lang.Object
  org.apache.nutch.fs.NutchFileSystem
      org.apache.nutch.fs.NDFSFileSystem

public class NDFSFileSystem

extends NutchFileSystem

Implement the NutchFileSystem interface for the NDFS system.

Author:

Mike Cafarella

Field Summary

Fields inherited from class org.apache.nutch.fs.NutchFileSystem

LOG

Constructor Summary

NDFSFileSystem(InetSocketAddress namenode)
Create the ShareSet automatically, and then go on to the regular constructor.

Method Summary

void	close() Shut down the FS.
void	completeLocalInput(File localFile) We're done with the local stuff, so delete it
void	completeLocalOutput(File nfsOutputFile, File tmpLocalFile) Move completed local data to NDFS destination
void	copyFromLocalFile(File src, File dst) keep the src when finished.
void	copyToLocalFile(File src, File dst) Takes a hierarchy of files from the NFS system and writes to the given local target.
NFSOutputStream	create(File f) Create the file at f.
NFSOutputStream	create(File f, boolean overwrite)
boolean	delete(File f) Get rid of File f, whether a true file or dir.
boolean	exists(File f) Check if exists
NDFSClient	getClient()
long	getLength(File f)
String	getName() Returns a name for this filesystem, suitable to pass to NutchFileSystem.getNamed(String).
boolean	isDirectory(File f)
File[]	listFiles(File f)
void	lock(File f, boolean shared) Obtain a filesystem lock at File f.
void	mkdirs(File f) Make the given file and all non-existent parents into directories.
void	moveFromLocalFile(File src, File dst) Remove the src when finished.
NFSInputStream	open(File f) Open the file at f
void	release(File f) Release a held lock
boolean	rename(File src, File dst) Rename files/dirs
File	startLocalInput(File nfsInputFile, File tmpLocalFile) Fetch remote NDFS file, place at tmpLocalFile
File	startLocalOutput(File nfsOutputFile, File tmpLocalFile) Output will go to the tmp working area.
String	toString()

Methods inherited from class org.apache.nutch.fs.NutchFileSystem

createNewFile, get, getNamed, isFile, listFiles, parseArgs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

NDFSFileSystem

public NDFSFileSystem(InetSocketAddress namenode)
               throws IOException

Create the ShareSet automatically, and then go on to the regular constructor.

Method Detail

getName

public String getName()

Description copied from class: NutchFileSystem

Returns a name for this filesystem, suitable to pass to NutchFileSystem.getNamed(String).

Specified by:

getName in class NutchFileSystem

open

public NFSInputStream open(File f)
                    throws IOException

Open the file at f

Specified by:

open in class NutchFileSystem

Throws:

IOException

create

public NFSOutputStream create(File f)
                       throws IOException

Create the file at f.

Specified by:

create in class NutchFileSystem

Throws:

IOException

create

public NFSOutputStream create(File f,
                              boolean overwrite)
                       throws IOException

Specified by:

create in class NutchFileSystem

Throws:

IOException

rename

public boolean rename(File src,
                      File dst)
               throws IOException

Rename files/dirs

Specified by:

rename in class NutchFileSystem

Throws:

IOException

delete

public boolean delete(File f)
               throws IOException

Get rid of File f, whether a true file or dir.

Specified by:

delete in class NutchFileSystem

Throws:

IOException

exists

public boolean exists(File f)
               throws IOException

Description copied from class: NutchFileSystem

Check if exists

Specified by:

exists in class NutchFileSystem

Throws:

IOException

isDirectory

public boolean isDirectory(File f)
                    throws IOException

Specified by:

isDirectory in class NutchFileSystem

Throws:

IOException

getLength

public long getLength(File f)
               throws IOException

Specified by:

getLength in class NutchFileSystem

Throws:

IOException

listFiles

public File[] listFiles(File f)
                 throws IOException

Specified by:

listFiles in class NutchFileSystem

Throws:

IOException

mkdirs

public void mkdirs(File f)
            throws IOException

Description copied from class: NutchFileSystem

Make the given file and all non-existent parents into directories.

Specified by:

mkdirs in class NutchFileSystem

Throws:

IOException

lock

public void lock(File f,
                 boolean shared)
          throws IOException

Obtain a filesystem lock at File f.

Specified by:

lock in class NutchFileSystem

Throws:

IOException

release

public void release(File f)
             throws IOException

Release a held lock

Specified by:

release in class NutchFileSystem

Throws:

IOException

moveFromLocalFile

public void moveFromLocalFile(File src,
                              File dst)
                       throws IOException

Remove the src when finished.

Specified by:

moveFromLocalFile in class NutchFileSystem

Throws:

IOException

copyFromLocalFile

public void copyFromLocalFile(File src,
                              File dst)
                       throws IOException

keep the src when finished.

Specified by:

copyFromLocalFile in class NutchFileSystem

Throws:

IOException

copyToLocalFile

public void copyToLocalFile(File src,
                            File dst)
                     throws IOException

Takes a hierarchy of files from the NFS system and writes to the given local target.

Specified by:

copyToLocalFile in class NutchFileSystem

Throws:

IOException

startLocalOutput

public File startLocalOutput(File nfsOutputFile,
                             File tmpLocalFile)
                      throws IOException

Output will go to the tmp working area. There may be some source material that we obtain first.

Specified by:

startLocalOutput in class NutchFileSystem

Throws:

IOException

completeLocalOutput

public void completeLocalOutput(File nfsOutputFile,
                                File tmpLocalFile)
                         throws IOException

Move completed local data to NDFS destination

Specified by:

completeLocalOutput in class NutchFileSystem

Throws:

IOException

startLocalInput

public File startLocalInput(File nfsInputFile,
                            File tmpLocalFile)
                     throws IOException

Fetch remote NDFS file, place at tmpLocalFile

Specified by:

startLocalInput in class NutchFileSystem

Throws:

IOException

completeLocalInput

public void completeLocalInput(File localFile)
                        throws IOException

We're done with the local stuff, so delete it

Specified by:

completeLocalInput in class NutchFileSystem

Throws:

IOException

close

public void close()
           throws IOException

Shut down the FS. Not necessary for regular filesystem.

Specified by:

close in class NutchFileSystem

Throws:

IOException

toString

public String toString()

getClient

public NDFSClient getClient()