org.apache.nutch.mapReduce
Class JobTracker

java.lang.Object
  org.apache.nutch.mapReduce.JobTracker

All Implemented Interfaces:

InterTrackerProtocol, JobSubmissionProtocol, MRConstants

public class JobTracker

extends Object

implements MRConstants, InterTrackerProtocol, JobSubmissionProtocol

JobTracker is the central location for submitting and tracking MR jobs in a network environment.

Author:

Mike Cafarella

Nested Class Summary

class

JobTracker.JobInProgress

Field Summary

static Logger	LOG
static JobTracker	tracker

Fields inherited from interface org.apache.nutch.mapReduce.MRConstants

FILE_NOT_FOUND, HEARTBEAT_INTERVAL, SUCCESS, TASKTRACKER_EXPIRY_INTERVAL

Fields inherited from interface org.apache.nutch.mapReduce.InterTrackerProtocol

TRACKERS_OK, UNKNOWN_TASKTRACKER

Method Summary

Vector	completedJobs()
static void	createTracker()
static void	createTracker(InetSocketAddress addr)
IntWritable	emitHeartbeat(TaskTrackerStatus trackerStatus, BooleanWritable initialContact) Process incoming heartbeat messages from the task trackers.
Vector	failedJobs()
static InetSocketAddress	getDefaultAddress()
String	getFilesystemName() The task tracker calls this once, to discern where it can find files referred to by the JobTracker
JobTracker.JobInProgress	getJob(String jobid)
JobProfile	getJobProfile(String jobid) Grab a handle to a job that is already known to the JobTracker
JobStatus	getJobStatus(String jobid) Grab a handle to a job that is already known to the JobTracker
String	getJobTrackerMachine()
long	getStartTime()
TaskTrackerStatus	getTaskTracker(String trackerID)
int	getTotalSubmissions()
static JobTracker	getTracker()
int	getTrackerPort()
void	initialize(String taskTrackerName)
void	killJob(String jobid) Kill the indicated job
MapOutputLocation[]	locateMapOutputs(String taskId, String[] mapTasksNeeded) A tracker wants to know the physical locations of completed, but not yet closed, tasks.
static void	main(String[] argv) Start the JobTracker process.
void	offerService() Run forever
String	pollForClosedTask(String trackerName) A tracker wants to know if any of its Tasks have been closed (because the job completed, whether successfully or not)
Task	pollForNewTask(String trackerName) A tracker wants to know if there's a Task to run
Vector	runningJobs()
JobStatus	submitJob(String jobFile) Submit a Job for execution.
Collection	taskTrackers()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final Logger LOG

tracker

public static JobTracker tracker

Method Detail

createTracker

public static void createTracker()
                          throws IOException

Throws:

IOException

createTracker

public static void createTracker(InetSocketAddress addr)
                          throws IOException

Throws:

IOException

getTracker

public static JobTracker getTracker()

getDefaultAddress

public static InetSocketAddress getDefaultAddress()

offerService

public void offerService()

Run forever

getTotalSubmissions

public int getTotalSubmissions()

getJobTrackerMachine

public String getJobTrackerMachine()

getTrackerPort

public int getTrackerPort()

getStartTime

public long getStartTime()

runningJobs

public Vector runningJobs()

failedJobs

public Vector failedJobs()

completedJobs

public Vector completedJobs()

taskTrackers

public Collection taskTrackers()

getTaskTracker

public TaskTrackerStatus getTaskTracker(String trackerID)

initialize

public void initialize(String taskTrackerName)

emitHeartbeat

public IntWritable emitHeartbeat(TaskTrackerStatus trackerStatus,
                                 BooleanWritable initialContact)

Process incoming heartbeat messages from the task trackers.

Specified by:

emitHeartbeat in interface InterTrackerProtocol

pollForNewTask

public Task pollForNewTask(String trackerName)

A tracker wants to know if there's a Task to run

Specified by:

pollForNewTask in interface InterTrackerProtocol

pollForClosedTask

public String pollForClosedTask(String trackerName)

A tracker wants to know if any of its Tasks have been closed (because the job completed, whether successfully or not)

Specified by:

pollForClosedTask in interface InterTrackerProtocol

locateMapOutputs

public MapOutputLocation[] locateMapOutputs(String taskId,
                                            String[] mapTasksNeeded)

A tracker wants to know the physical locations of completed, but not yet closed, tasks. This exists so the reduce task thread can locate map task outputs.

Specified by:

locateMapOutputs in interface InterTrackerProtocol

Parameters:

taskId - the reduce task id

mapTasksNeeded - an array of UTF8 naming map task ids whose output is needed.

Returns:

an array of MapOutputLocation

getJob

public JobTracker.JobInProgress getJob(String jobid)

submitJob

public JobStatus submitJob(String jobFile)
                    throws IOException

Description copied from interface: JobSubmissionProtocol

Submit a Job for execution. Returns the latest profile for that job.

Specified by:

submitJob in interface JobSubmissionProtocol

Throws:

IOException

killJob

public void killJob(String jobid)

Description copied from interface: JobSubmissionProtocol

Kill the indicated job

Specified by:

killJob in interface JobSubmissionProtocol

getJobProfile

public JobProfile getJobProfile(String jobid)

Description copied from interface: JobSubmissionProtocol

Grab a handle to a job that is already known to the JobTracker

Specified by:

getJobProfile in interface JobSubmissionProtocol

getJobStatus

public JobStatus getJobStatus(String jobid)

Description copied from interface: JobSubmissionProtocol

Grab a handle to a job that is already known to the JobTracker

Specified by:

getJobStatus in interface JobSubmissionProtocol

getFilesystemName

public String getFilesystemName()
                         throws IOException

Description copied from interface: InterTrackerProtocol

The task tracker calls this once, to discern where it can find files referred to by the JobTracker

Specified by:

getFilesystemName in interface InterTrackerProtocol

Throws:

IOException

main

public static void main(String[] argv)
                 throws IOException,
                        InterruptedException

Start the JobTracker process. This is used only for debugging. As a rule, JobTracker should be run as part of the NDFS Namenode process.

Throws:

IOException

InterruptedException