org.apache.nutch.fetcher
Class Fetcher

java.lang.Object
  org.apache.nutch.fetcher.Fetcher

public class Fetcher

extends Object

The fetcher. Most of the work is done by plugins.

Note by John Xing: As of 20041022, option -noParsing is introduced. Without this option, fetcher behaves the old way, i.e., it not only crawls but also parses content. With option -noParsing, fetcher does crawl only. Use ParseSegment.java to parse fetched contents. Check FetcherOutput.java and ParseSegment.java for further description.

Nested Class Summary

static class

Fetcher.FetcherStatus

Field Summary

static Logger

LOG

Constructor Summary

Fetcher(NutchFileSystem nfs, String directory, boolean parsing)

Method Summary

Fetcher.FetcherStatus	getStatus()
static void	main(String[] args) Run the fetcher.
void	run() Runs the fetcher.
static void	setLogLevel(Level level) Set the logging level.
void	setThreadCount(int threadCount) Set thread count
void	status() Display the status of the fetcher run.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final Logger LOG

Constructor Detail

Fetcher

public Fetcher(NutchFileSystem nfs,
               String directory,
               boolean parsing)
        throws IOException

Method Detail

setThreadCount

public void setThreadCount(int threadCount)

Set thread count

setLogLevel

public static void setLogLevel(Level level)

Set the logging level.

run

public void run()
         throws IOException,
                InterruptedException

Runs the fetcher.

Throws:

IOException

InterruptedException

getStatus

public Fetcher.FetcherStatus getStatus()

status

public void status()

Display the status of the fetcher run.

main

public static void main(String[] args)
                 throws Exception

Run the fetcher.

Throws:

Exception