org.apache.nutch.db
Class EditSectionGroupWriter

java.lang.Object
  org.apache.nutch.db.EditSectionGroupWriter

public class EditSectionGroupWriter

extends Object

The EditSectionGroupWriter maintains a set of EditSectionWriter objects. It chooses the appropriate EditSectionWriter to carry out each operation.

Author:

Mike Cafarella

Nested Class Summary

static class	EditSectionGroupWriter.KeyExtractor Edit instructions are Comparable, but they also have an "inner" key like MD5Hash or URL that is also Comparable.
static class	EditSectionGroupWriter.LinkMD5Extractor Get the MD5 from a LinkInstruction
static class	EditSectionGroupWriter.LinkURLExtractor Get the URL from a LinkInstruction
static class	EditSectionGroupWriter.PageMD5Extractor Get the MD5 from a PageInstruction
static class	EditSectionGroupWriter.PageURLExtractor Get the URL from a PageInstruction

Field Summary

static String	GROUP_METAINFO
static int	MD5_KEYSPACE
static int	URL_KEYSPACE

Constructor Summary

EditSectionGroupWriter(NutchFileSystem nfs, int machineNum, int totalMachines, String label, Class keyClass, Class valClass, EditSectionGroupWriter.KeyExtractor extractor)
Start a EditSectionGroupWriter at the indicated location, for a single emitter.

Method Summary

void	append(WritableComparable key, Writable val) Add an instruction and append it.
void	close() Close down the writers
static void	createEditGroup(NutchFileSystem nfs, File dbDir, String label, int numSections, int keySpaceType) Initialize an EditSectionGroup.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

GROUP_METAINFO

public static final String GROUP_METAINFO

See Also:

Constant Field Values

URL_KEYSPACE

public static int URL_KEYSPACE

MD5_KEYSPACE

public static int MD5_KEYSPACE

Constructor Detail

EditSectionGroupWriter

public EditSectionGroupWriter(NutchFileSystem nfs,
                              int machineNum,
                              int totalMachines,
                              String label,
                              Class keyClass,
                              Class valClass,
                              EditSectionGroupWriter.KeyExtractor extractor)
                       throws IOException

Start a EditSectionGroupWriter at the indicated location, for a single emitter. There will be as many of these as there are processor-machines. (The emitter value will be different for each.) The Group must already have been created via a call to EditSectionGroupWriter.createEditSectionGroupWriter(). The EditSectionGroupWriter consists of a bunch of EditSectionWriters, each of which hold a file we append to.

Method Detail

createEditGroup

public static void createEditGroup(NutchFileSystem nfs,
                                   File dbDir,
                                   String label,
                                   int numSections,
                                   int keySpaceType)
                            throws IOException

Initialize an EditSectionGroup. Tell it the label, the keytype, and the division between keys.

Throws:

IOException

append

public void append(WritableComparable key,
                   Writable val)
            throws IOException

Add an instruction and append it. We need to find an appropriate EditSectionWriter.

Throws:

IOException

close

public void close()
           throws IOException

Close down the writers

Throws:

IOException