org.apache.hadoop.mapred
Class TaskTracker

java.lang.Object
  org.apache.hadoop.mapred.TaskTracker

All Implemented Interfaces:

Runnable, VersionedProtocol, TaskTrackerMXBean, TaskUmbilicalProtocol

public class TaskTracker
extends Object
implements TaskUmbilicalProtocol, Runnable, TaskTrackerMXBean

TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.

Nested Class Summary

static class

TaskTracker.MapOutputServlet This class is used in TaskTracker's Jetty to serve the map outputs to other nodes.

Field Summary

static org.apache.commons.logging.Log	ClientTraceLog
static long	COUNTER_UPDATE_INTERVAL
static long	DEFAULT_DISK_HEALTH_CHECK_INTERVAL How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
static int	FILE_NOT_FOUND
static String	FOR_REDUCE_TASK The reduce task number for which this map output is being transferred
static String	FROM_MAP_TASK The map task from which the map output data is being transferred
static int	HEARTBEAT_INTERVAL_MIN
static org.apache.commons.logging.Log	LOG
static String	MAP_OUTPUT_LENGTH The custom http header used for the map output length.
static String	MR_CLIENTTRACE_FORMAT
static String	RAW_MAP_OUTPUT_LENGTH The custom http header used for the "raw" map output length.
static String	SUBDIR
static int	SUCCESS
static String	TT_KEYTAB_FILE
static String	TT_LOG_TMP_DIR
static String	TT_RESOURCE_CALCULATOR_PLUGIN
static String	TT_USER_NAME
static String	WORKDIR

Fields inherited from interface org.apache.hadoop.mapred.TaskUmbilicalProtocol

versionID

Constructor Summary

TaskTracker(JobConf conf)
Start with the local machine name, and the default JobTracker

Method Summary

boolean	canCommit(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) Child checking whether it can commit
void	cleanupStorage() Removes all contents of temporary storage.
void	close() Close down the TaskTracker and all its components.
void	commitPending(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext) Task is reporting that it is in commit_pending and it is waiting for the commit Response
void	done(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) The task is done.
void	fatalError(TaskAttemptID taskId, String msg, org.apache.hadoop.mapred.JvmContext jvmContext) A child task had a fatal error.
void	fsError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext) A child task had a local filesystem error.
String	getConfigVersion()
String	getHostname()
int	getHttpPort()
static String	getJobCacheSubdir(String user)
org.apache.hadoop.mapred.InterTrackerProtocol	getJobClient() The connection to the JobTracker, used by the TaskRunner for locating remote files.
static String	getJobJarFile(String user, String jobid)
String	getJobTrackerUrl()
org.apache.hadoop.mapred.JvmManager	getJvmManagerInstance()
static String	getLocalJobDir(String user, String jobid)
static String	getLocalTaskDir(String user, String jobid, String taskid)
static String	getLocalTaskDir(String user, String jobid, String taskid, boolean isCleanupAttempt)
MapTaskCompletionEventsUpdate	getMapCompletionEvents(JobID jobId, int fromEventId, int maxLocs, TaskAttemptID id, org.apache.hadoop.mapred.JvmContext jvmContext) Called by a reduce task to get the map output locations for finished maps.
static String	getPrivateDistributedCacheDir(String user)
long	getProtocolVersion(String protocol, long clientVersion) Return protocol version corresponding to protocol interface.
static String	getPublicDistributedCacheDir()
int	getRpcPort()
JvmTask	getTask(org.apache.hadoop.mapred.JvmContext context) Called upon startup by the child process, to fetch Task data.
TaskController	getTaskController()
org.apache.hadoop.mapred.TaskMemoryManagerThread	getTaskMemoryManager()
String	getTasksInfoJson()
org.apache.hadoop.mapred.TaskTrackerInstrumentation	getTaskTrackerInstrumentation()
InetSocketAddress	getTaskTrackerReportAddress() Return the port at which the tasktracker bound to
static String	getUserDir(String user)
String	getVersion()
boolean	isHealthy()
boolean	isIdle() Is this task tracker idle?
boolean	isIdleAndClean() Is this task tracker idle and clean?
boolean	isTaskMemoryManagerEnabled() Is the TaskMemoryManager Enabled on this system?
protected void	launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip, JobConf jobConf, org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
static void	main(String[] argv) Start the TaskTracker, point toward the indicated JobTracker
void	mapOutputLost(TaskAttemptID taskid, String errorMsg) A completed map task's output has been lost.
boolean	ping(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) Child checking to see if we're alive.
void	reportDiagnosticInfo(TaskAttemptID taskid, String info, org.apache.hadoop.mapred.JvmContext jvmContext) Called when the task dies before completion, and we want to report back diagnostic info
void	reportNextRecordRange(TaskAttemptID taskid, org.apache.hadoop.mapred.SortedRanges.Range range, org.apache.hadoop.mapred.JvmContext jvmContext) Report the record range which is going to process next by the Task.
void	run() The server retry loop.
void	shuffleError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext) A reduce-task failed to shuffle the map-outputs.
void	shutdown()
boolean	statusUpdate(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext) Called periodically to report Task progress, from 0.0 to 1.0.
void	updatePrivateDistributedCacheSizes(JobID jobId, long[] sizes) The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

MR_CLIENTTRACE_FORMAT

public static final String MR_CLIENTTRACE_FORMAT

See Also:

Constant Field Values

ClientTraceLog

public static final org.apache.commons.logging.Log ClientTraceLog

SUBDIR

public static final String SUBDIR

See Also:

Constant Field Values

TT_LOG_TMP_DIR

public static final String TT_LOG_TMP_DIR

See Also:

Constant Field Values

TT_RESOURCE_CALCULATOR_PLUGIN

public static final String TT_RESOURCE_CALCULATOR_PLUGIN

See Also:

Constant Field Values

TT_USER_NAME

public static final String TT_USER_NAME

See Also:

Constant Field Values

TT_KEYTAB_FILE

public static final String TT_KEYTAB_FILE

See Also:

Constant Field Values

HEARTBEAT_INTERVAL_MIN

public static final int HEARTBEAT_INTERVAL_MIN

See Also:

Constant Field Values

COUNTER_UPDATE_INTERVAL

public static final long COUNTER_UPDATE_INTERVAL

See Also:

Constant Field Values

DEFAULT_DISK_HEALTH_CHECK_INTERVAL

public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL

How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval

See Also:

Constant Field Values

SUCCESS

public static final int SUCCESS

See Also:

Constant Field Values

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND

See Also:

Constant Field Values

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH

The custom http header used for the map output length.

See Also:

Constant Field Values

RAW_MAP_OUTPUT_LENGTH

public static final String RAW_MAP_OUTPUT_LENGTH

The custom http header used for the "raw" map output length.

See Also:

Constant Field Values

FROM_MAP_TASK

public static final String FROM_MAP_TASK

The map task from which the map output data is being transferred

See Also:

Constant Field Values

FOR_REDUCE_TASK

public static final String FOR_REDUCE_TASK

The reduce task number for which this map output is being transferred

See Also:

Constant Field Values

WORKDIR

public static final String WORKDIR

See Also:

Constant Field Values

Constructor Detail

TaskTracker

public TaskTracker(JobConf conf)
            throws IOException,
                   InterruptedException

Start with the local machine name, and the default JobTracker

Throws:

IOException

InterruptedException

Method Detail

getTaskTrackerInstrumentation

public org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()

getTaskController

public TaskController getTaskController()

getUserDir

public static String getUserDir(String user)

getPrivateDistributedCacheDir

public static String getPrivateDistributedCacheDir(String user)

getPublicDistributedCacheDir

public static String getPublicDistributedCacheDir()

getJobCacheSubdir

public static String getJobCacheSubdir(String user)

getLocalJobDir

public static String getLocalJobDir(String user,
                                    String jobid)

getJobJarFile

public static String getJobJarFile(String user,
                                   String jobid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                                     String jobid,
                                     String taskid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                                     String jobid,
                                     String taskid,
                                     boolean isCleanupAttempt)

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolVersion in interface VersionedProtocol

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

Returns:

the version that the server will speak

Throws:

IOException

cleanupStorage

public void cleanupStorage()
                    throws IOException

Removes all contents of temporary storage. Called upon startup, to remove any leftovers from previous run.

Throws:

IOException

launchTaskForJob

protected void launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip,
                                JobConf jobConf,
                                org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
                         throws IOException

Throws:

IOException

shutdown

public void shutdown()
              throws IOException,
                     InterruptedException

Throws:

IOException

InterruptedException

close

public void close()
           throws IOException,
                  InterruptedException

Close down the TaskTracker and all its components. We must also shutdown any running tasks or threads, and cleanup disk space. A new TaskTracker within the same process space might be restarted, so everything must be clean.

Throws:

InterruptedException

IOException

getJobClient

public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()

The connection to the JobTracker, used by the TaskRunner for locating remote files.

getTaskTrackerReportAddress

public InetSocketAddress getTaskTrackerReportAddress()

Return the port at which the tasktracker bound to

getJvmManagerInstance

public org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()

run

public void run()

The server retry loop. This while-loop attempts to connect to the JobTracker. It only loops when the old TaskTracker has gone bad (its state is stale somehow) and we need to reinitialize everything.

Specified by:

run in interface Runnable

getTask

public JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
                throws IOException

Called upon startup by the child process, to fetch Task data.

Specified by:

getTask in interface TaskUmbilicalProtocol

Parameters:

context - the JvmContext of the JVM w.r.t the TaskTracker that launched it

Returns:

Task object

Throws:

IOException

statusUpdate

public boolean statusUpdate(TaskAttemptID taskid,
                            TaskStatus taskStatus,
                            org.apache.hadoop.mapred.JvmContext jvmContext)
                     throws IOException

Called periodically to report Task progress, from 0.0 to 1.0.

Specified by:

statusUpdate in interface TaskUmbilicalProtocol

Parameters:

taskid - task-id of the child

taskStatus - status of the child

jvmContext - context the jvmContext running the task.

Returns:

True if the task is known

Throws:

IOException

reportDiagnosticInfo

public void reportDiagnosticInfo(TaskAttemptID taskid,
                                 String info,
                                 org.apache.hadoop.mapred.JvmContext jvmContext)
                          throws IOException

Called when the task dies before completion, and we want to report back diagnostic info

Specified by:

reportDiagnosticInfo in interface TaskUmbilicalProtocol

Parameters:

taskid - the id of the task involved

info - the text to report

jvmContext - context the jvmContext running the task.

Throws:

IOException

reportNextRecordRange

public void reportNextRecordRange(TaskAttemptID taskid,
                                  org.apache.hadoop.mapred.SortedRanges.Range range,
                                  org.apache.hadoop.mapred.JvmContext jvmContext)
                           throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report the record range which is going to process next by the Task.

Specified by:

reportNextRecordRange in interface TaskUmbilicalProtocol

Parameters:

taskid - the id of the task involved

range - the range of record sequence nos

jvmContext - context the jvmContext running the task.

Throws:

IOException

ping

public boolean ping(TaskAttemptID taskid,
                    org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException

Child checking to see if we're alive. Normally does nothing.

Specified by:

ping in interface TaskUmbilicalProtocol

Parameters:

taskid - the id of the task involved

jvmContext - context the jvmContext running the task.

Returns:

True if the task is known

Throws:

IOException

commitPending

public void commitPending(TaskAttemptID taskid,
                          TaskStatus taskStatus,
                          org.apache.hadoop.mapred.JvmContext jvmContext)
                   throws IOException

Task is reporting that it is in commit_pending and it is waiting for the commit Response

Specified by:

commitPending in interface TaskUmbilicalProtocol

Parameters:

taskid - task's id

taskStatus - status of the child

jvmContext - context the jvmContext running the task.

Throws:

IOException

canCommit

public boolean canCommit(TaskAttemptID taskid,
                         org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException

Child checking whether it can commit

Specified by:

canCommit in interface TaskUmbilicalProtocol

jvmContext - context the jvmContext running the task.

Returns:

true/false

Throws:

IOException

done

public void done(TaskAttemptID taskid,
                 org.apache.hadoop.mapred.JvmContext jvmContext)
          throws IOException

The task is done.

Specified by:

done in interface TaskUmbilicalProtocol

Parameters:

taskid - task's id

jvmContext - context the jvmContext running the task.

Throws:

IOException

shuffleError

public void shuffleError(TaskAttemptID taskId,
                         String message,
                         org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException

A reduce-task failed to shuffle the map-outputs. Kill the task.

Specified by:

shuffleError in interface TaskUmbilicalProtocol

Throws:

IOException

fsError

public void fsError(TaskAttemptID taskId,
                    String message,
                    org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException

A child task had a local filesystem error. Kill the task.

Specified by:

fsError in interface TaskUmbilicalProtocol

Throws:

IOException

fatalError

public void fatalError(TaskAttemptID taskId,
                       String msg,
                       org.apache.hadoop.mapred.JvmContext jvmContext)
                throws IOException

A child task had a fatal error. Kill the task.

Specified by:

fatalError in interface TaskUmbilicalProtocol

Throws:

IOException

getMapCompletionEvents

public MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId,
                                                            int fromEventId,
                                                            int maxLocs,
                                                            TaskAttemptID id,
                                                            org.apache.hadoop.mapred.JvmContext jvmContext)
                                                     throws IOException

Description copied from interface: TaskUmbilicalProtocol

Called by a reduce task to get the map output locations for finished maps. Returns an update centered around the map-task-completion-events. The update also piggybacks the information whether the events copy at the task-tracker has changed or not. This will trigger some action at the child-process.

Specified by:

getMapCompletionEvents in interface TaskUmbilicalProtocol

Parameters:

jobId - the reducer job id

fromEventId - the index starting from which the locations should be fetched

maxLocs - the max number of locations to fetch

id - The attempt id of the task that is trying to communicate

Returns:

A MapTaskCompletionEventsUpdate

Throws:

IOException

mapOutputLost

public void mapOutputLost(TaskAttemptID taskid,
                          String errorMsg)
                   throws IOException

A completed map task's output has been lost.

Throws:

IOException

isIdle

public boolean isIdle()

Is this task tracker idle?

Returns:

has this task tracker finished all its assigned tasks?

isIdleAndClean

public boolean isIdleAndClean()

Is this task tracker idle and clean?

Returns:

has this task tracker finished and cleaned up all of its tasks?

main

public static void main(String[] argv)
                 throws Exception

Start the TaskTracker, point toward the indicated JobTracker

Throws:

Exception

isTaskMemoryManagerEnabled

public boolean isTaskMemoryManagerEnabled()

Is the TaskMemoryManager Enabled on this system?

Returns:

true if enabled, false otherwise.

getTaskMemoryManager

public org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()

getHostname

public String getHostname()

Specified by:

getHostname in interface TaskTrackerMXBean

Returns:

the hostname of the tasktracker

getVersion

public String getVersion()

Specified by:

getVersion in interface TaskTrackerMXBean

Returns:

the version of the code base

getConfigVersion

public String getConfigVersion()

Specified by:

getConfigVersion in interface TaskTrackerMXBean

Returns:

the config version (from a config properties)

getJobTrackerUrl

public String getJobTrackerUrl()

Specified by:

getJobTrackerUrl in interface TaskTrackerMXBean

Returns:

the URL of the jobtracker

getRpcPort

public int getRpcPort()

Specified by:

getRpcPort in interface TaskTrackerMXBean

Returns:

the RPC port of the tasktracker

getHttpPort

public int getHttpPort()

Specified by:

getHttpPort in interface TaskTrackerMXBean

Returns:

the HTTP port of the tasktracker

isHealthy

public boolean isHealthy()

Specified by:

isHealthy in interface TaskTrackerMXBean

Returns:

the health status of the tasktracker

getTasksInfoJson

public String getTasksInfoJson()

Specified by:

getTasksInfoJson in interface TaskTrackerMXBean

Returns:

a json formatted info about tasks of the tasktracker

updatePrivateDistributedCacheSizes

public void updatePrivateDistributedCacheSizes(JobID jobId,
                                               long[] sizes)
                                        throws IOException

Description copied from interface: TaskUmbilicalProtocol

The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

Specified by:

updatePrivateDistributedCacheSizes in interface TaskUmbilicalProtocol

Parameters:

jobId - the job to update

sizes - the array of sizes that were computed

Throws:

IOException