org.apache.hadoop.mapred
Class TaskTracker

java.lang.Object
  extended by org.apache.hadoop.mapred.TaskTracker
All Implemented Interfaces:
Runnable, VersionedProtocol, TaskTrackerMXBean, TaskUmbilicalProtocol

public class TaskTracker
extends Object
implements TaskUmbilicalProtocol, Runnable, TaskTrackerMXBean

TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.


Nested Class Summary
static class TaskTracker.MapOutputServlet
          This class is used in TaskTracker's Jetty to serve the map outputs to other nodes.
 
Field Summary
static org.apache.commons.logging.Log ClientTraceLog
           
static long COUNTER_UPDATE_INTERVAL
           
static long DEFAULT_DISK_HEALTH_CHECK_INTERVAL
          How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
static int FILE_NOT_FOUND
           
static String FOR_REDUCE_TASK
          The reduce task number for which this map output is being transferred
static String FROM_MAP_TASK
          The map task from which the map output data is being transferred
static int HEARTBEAT_INTERVAL_MIN
           
static org.apache.commons.logging.Log LOG
           
static String MAP_OUTPUT_LENGTH
          The custom http header used for the map output length.
static String MR_CLIENTTRACE_FORMAT
           
static String RAW_MAP_OUTPUT_LENGTH
          The custom http header used for the "raw" map output length.
static String SUBDIR
           
static int SUCCESS
           
static String TT_KEYTAB_FILE
           
static String TT_LOG_TMP_DIR
           
static String TT_RESOURCE_CALCULATOR_PLUGIN
           
static String TT_USER_NAME
           
static String WORKDIR
           
 
Fields inherited from interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
versionID
 
Constructor Summary
TaskTracker(JobConf conf)
          Start with the local machine name, and the default JobTracker
 
Method Summary
 boolean canCommit(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext)
          Child checking whether it can commit
 void cleanupStorage()
          Removes all contents of temporary storage.
 void close()
          Close down the TaskTracker and all its components.
 void commitPending(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext)
          Task is reporting that it is in commit_pending and it is waiting for the commit Response
 void done(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext)
          The task is done.
 void fatalError(TaskAttemptID taskId, String msg, org.apache.hadoop.mapred.JvmContext jvmContext)
          A child task had a fatal error.
 void fsError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext)
          A child task had a local filesystem error.
 String getConfigVersion()
           
 String getHostname()
           
 int getHttpPort()
           
static String getJobCacheSubdir(String user)
           
 org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()
          The connection to the JobTracker, used by the TaskRunner for locating remote files.
static String getJobJarFile(String user, String jobid)
           
 String getJobTrackerUrl()
           
 org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()
           
static String getLocalJobDir(String user, String jobid)
           
static String getLocalTaskDir(String user, String jobid, String taskid)
           
static String getLocalTaskDir(String user, String jobid, String taskid, boolean isCleanupAttempt)
           
 MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId, int fromEventId, int maxLocs, TaskAttemptID id, org.apache.hadoop.mapred.JvmContext jvmContext)
          Called by a reduce task to get the map output locations for finished maps.
static String getPrivateDistributedCacheDir(String user)
           
 long getProtocolVersion(String protocol, long clientVersion)
          Return protocol version corresponding to protocol interface.
static String getPublicDistributedCacheDir()
           
 int getRpcPort()
           
 JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
          Called upon startup by the child process, to fetch Task data.
 TaskController getTaskController()
           
 org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()
           
 String getTasksInfoJson()
           
 org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()
           
 InetSocketAddress getTaskTrackerReportAddress()
          Return the port at which the tasktracker bound to
static String getUserDir(String user)
           
 String getVersion()
           
 boolean isHealthy()
           
 boolean isIdle()
          Is this task tracker idle?
 boolean isTaskMemoryManagerEnabled()
          Is the TaskMemoryManager Enabled on this system?
protected  void launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip, JobConf jobConf, org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
           
static void main(String[] argv)
          Start the TaskTracker, point toward the indicated JobTracker
 void mapOutputLost(TaskAttemptID taskid, String errorMsg)
          A completed map task's output has been lost.
 boolean ping(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext)
          Child checking to see if we're alive.
 void reportDiagnosticInfo(TaskAttemptID taskid, String info, org.apache.hadoop.mapred.JvmContext jvmContext)
          Called when the task dies before completion, and we want to report back diagnostic info
 void reportNextRecordRange(TaskAttemptID taskid, org.apache.hadoop.mapred.SortedRanges.Range range, org.apache.hadoop.mapred.JvmContext jvmContext)
          Report the record range which is going to process next by the Task.
 void run()
          The server retry loop.
 void shuffleError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext)
          A reduce-task failed to shuffle the map-outputs.
 void shutdown()
           
 boolean statusUpdate(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext)
          Called periodically to report Task progress, from 0.0 to 1.0.
 void updatePrivateDistributedCacheSizes(JobID jobId, long[] sizes)
          The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

MR_CLIENTTRACE_FORMAT

public static final String MR_CLIENTTRACE_FORMAT
See Also:
Constant Field Values

ClientTraceLog

public static final org.apache.commons.logging.Log ClientTraceLog

SUBDIR

public static final String SUBDIR
See Also:
Constant Field Values

TT_LOG_TMP_DIR

public static final String TT_LOG_TMP_DIR
See Also:
Constant Field Values

TT_RESOURCE_CALCULATOR_PLUGIN

public static final String TT_RESOURCE_CALCULATOR_PLUGIN
See Also:
Constant Field Values

TT_USER_NAME

public static final String TT_USER_NAME
See Also:
Constant Field Values

TT_KEYTAB_FILE

public static final String TT_KEYTAB_FILE
See Also:
Constant Field Values

HEARTBEAT_INTERVAL_MIN

public static final int HEARTBEAT_INTERVAL_MIN
See Also:
Constant Field Values

COUNTER_UPDATE_INTERVAL

public static final long COUNTER_UPDATE_INTERVAL
See Also:
Constant Field Values

DEFAULT_DISK_HEALTH_CHECK_INTERVAL

public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL
How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval

See Also:
Constant Field Values

SUCCESS

public static final int SUCCESS
See Also:
Constant Field Values

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND
See Also:
Constant Field Values

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH
The custom http header used for the map output length.

See Also:
Constant Field Values

RAW_MAP_OUTPUT_LENGTH

public static final String RAW_MAP_OUTPUT_LENGTH
The custom http header used for the "raw" map output length.

See Also:
Constant Field Values

FROM_MAP_TASK

public static final String FROM_MAP_TASK
The map task from which the map output data is being transferred

See Also:
Constant Field Values

FOR_REDUCE_TASK

public static final String FOR_REDUCE_TASK
The reduce task number for which this map output is being transferred

See Also:
Constant Field Values

WORKDIR

public static final String WORKDIR
See Also:
Constant Field Values
Constructor Detail

TaskTracker

public TaskTracker(JobConf conf)
            throws IOException,
                   InterruptedException
Start with the local machine name, and the default JobTracker

Throws:
IOException
InterruptedException
Method Detail

getTaskTrackerInstrumentation

public org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()

getTaskController

public TaskController getTaskController()

getUserDir

public static String getUserDir(String user)

getPrivateDistributedCacheDir

public static String getPrivateDistributedCacheDir(String user)

getPublicDistributedCacheDir

public static String getPublicDistributedCacheDir()

getJobCacheSubdir

public static String getJobCacheSubdir(String user)

getLocalJobDir

public static String getLocalJobDir(String user,
                                    String jobid)

getJobJarFile

public static String getJobJarFile(String user,
                                   String jobid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                                     String jobid,
                                     String taskid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                                     String jobid,
                                     String taskid,
                                     boolean isCleanupAttempt)

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException
Description copied from interface: VersionedProtocol
Return protocol version corresponding to protocol interface.

Specified by:
getProtocolVersion in interface VersionedProtocol
Parameters:
protocol - The classname of the protocol interface
clientVersion - The version of the protocol that the client speaks
Returns:
the version that the server will speak
Throws:
IOException

cleanupStorage

public void cleanupStorage()
                    throws IOException
Removes all contents of temporary storage. Called upon startup, to remove any leftovers from previous run.

Throws:
IOException

launchTaskForJob

protected void launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip,
                                JobConf jobConf,
                                org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
                         throws IOException
Throws:
IOException

shutdown

public void shutdown()
              throws IOException,
                     InterruptedException
Throws:
IOException
InterruptedException

close

public void close()
           throws IOException,
                  InterruptedException
Close down the TaskTracker and all its components. We must also shutdown any running tasks or threads, and cleanup disk space. A new TaskTracker within the same process space might be restarted, so everything must be clean.

Throws:
InterruptedException
IOException

getJobClient

public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()
The connection to the JobTracker, used by the TaskRunner for locating remote files.


getTaskTrackerReportAddress

public InetSocketAddress getTaskTrackerReportAddress()
Return the port at which the tasktracker bound to


getJvmManagerInstance

public org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()

run

public void run()
The server retry loop. This while-loop attempts to connect to the JobTracker. It only loops when the old TaskTracker has gone bad (its state is stale somehow) and we need to reinitialize everything.

Specified by:
run in interface Runnable

getTask

public JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
                throws IOException
Called upon startup by the child process, to fetch Task data.

Specified by:
getTask in interface TaskUmbilicalProtocol
Parameters:
context - the JvmContext of the JVM w.r.t the TaskTracker that launched it
Returns:
Task object
Throws:
IOException

statusUpdate

public boolean statusUpdate(TaskAttemptID taskid,
                            TaskStatus taskStatus,
                            org.apache.hadoop.mapred.JvmContext jvmContext)
                     throws IOException
Called periodically to report Task progress, from 0.0 to 1.0.

Specified by:
statusUpdate in interface TaskUmbilicalProtocol
Parameters:
taskid - task-id of the child
taskStatus - status of the child
jvmContext - context the jvmContext running the task.
Returns:
True if the task is known
Throws:
IOException

reportDiagnosticInfo

public void reportDiagnosticInfo(TaskAttemptID taskid,
                                 String info,
                                 org.apache.hadoop.mapred.JvmContext jvmContext)
                          throws IOException
Called when the task dies before completion, and we want to report back diagnostic info

Specified by:
reportDiagnosticInfo in interface TaskUmbilicalProtocol
Parameters:
taskid - the id of the task involved
info - the text to report
jvmContext - context the jvmContext running the task.
Throws:
IOException

reportNextRecordRange

public void reportNextRecordRange(TaskAttemptID taskid,
                                  org.apache.hadoop.mapred.SortedRanges.Range range,
                                  org.apache.hadoop.mapred.JvmContext jvmContext)
                           throws IOException
Description copied from interface: TaskUmbilicalProtocol
Report the record range which is going to process next by the Task.

Specified by:
reportNextRecordRange in interface TaskUmbilicalProtocol
Parameters:
taskid - the id of the task involved
range - the range of record sequence nos
jvmContext - context the jvmContext running the task.
Throws:
IOException

ping

public boolean ping(TaskAttemptID taskid,
                    org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException
Child checking to see if we're alive. Normally does nothing.

Specified by:
ping in interface TaskUmbilicalProtocol
Parameters:
taskid - the id of the task involved
jvmContext - context the jvmContext running the task.
Returns:
True if the task is known
Throws:
IOException

commitPending

public void commitPending(TaskAttemptID taskid,
                          TaskStatus taskStatus,
                          org.apache.hadoop.mapred.JvmContext jvmContext)
                   throws IOException
Task is reporting that it is in commit_pending and it is waiting for the commit Response

Specified by:
commitPending in interface TaskUmbilicalProtocol
Parameters:
taskid - task's id
taskStatus - status of the child
jvmContext - context the jvmContext running the task.
Throws:
IOException

canCommit

public boolean canCommit(TaskAttemptID taskid,
                         org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException
Child checking whether it can commit

Specified by:
canCommit in interface TaskUmbilicalProtocol
jvmContext - context the jvmContext running the task.
Returns:
true/false
Throws:
IOException

done

public void done(TaskAttemptID taskid,
                 org.apache.hadoop.mapred.JvmContext jvmContext)
          throws IOException
The task is done.

Specified by:
done in interface TaskUmbilicalProtocol
Parameters:
taskid - task's id
jvmContext - context the jvmContext running the task.
Throws:
IOException

shuffleError

public void shuffleError(TaskAttemptID taskId,
                         String message,
                         org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException
A reduce-task failed to shuffle the map-outputs. Kill the task.

Specified by:
shuffleError in interface TaskUmbilicalProtocol
Throws:
IOException

fsError

public void fsError(TaskAttemptID taskId,
                    String message,
                    org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException
A child task had a local filesystem error. Kill the task.

Specified by:
fsError in interface TaskUmbilicalProtocol
Throws:
IOException

fatalError

public void fatalError(TaskAttemptID taskId,
                       String msg,
                       org.apache.hadoop.mapred.JvmContext jvmContext)
                throws IOException
A child task had a fatal error. Kill the task.

Specified by:
fatalError in interface TaskUmbilicalProtocol
Throws:
IOException

getMapCompletionEvents

public MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId,
                                                            int fromEventId,
                                                            int maxLocs,
                                                            TaskAttemptID id,
                                                            org.apache.hadoop.mapred.JvmContext jvmContext)
                                                     throws IOException
Description copied from interface: TaskUmbilicalProtocol
Called by a reduce task to get the map output locations for finished maps. Returns an update centered around the map-task-completion-events. The update also piggybacks the information whether the events copy at the task-tracker has changed or not. This will trigger some action at the child-process.

Specified by:
getMapCompletionEvents in interface TaskUmbilicalProtocol
Parameters:
jobId - the reducer job id
fromEventId - the index starting from which the locations should be fetched
maxLocs - the max number of locations to fetch
id - The attempt id of the task that is trying to communicate
Returns:
A MapTaskCompletionEventsUpdate
Throws:
IOException

mapOutputLost

public void mapOutputLost(TaskAttemptID taskid,
                          String errorMsg)
                   throws IOException
A completed map task's output has been lost.

Throws:
IOException

isIdle

public boolean isIdle()
Is this task tracker idle?

Returns:
has this task tracker finished and cleaned up all of its tasks?

main

public static void main(String[] argv)
                 throws Exception
Start the TaskTracker, point toward the indicated JobTracker

Throws:
Exception

isTaskMemoryManagerEnabled

public boolean isTaskMemoryManagerEnabled()
Is the TaskMemoryManager Enabled on this system?

Returns:
true if enabled, false otherwise.

getTaskMemoryManager

public org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()

getHostname

public String getHostname()
Specified by:
getHostname in interface TaskTrackerMXBean
Returns:
the hostname of the tasktracker

getVersion

public String getVersion()
Specified by:
getVersion in interface TaskTrackerMXBean
Returns:
the version of the code base

getConfigVersion

public String getConfigVersion()
Specified by:
getConfigVersion in interface TaskTrackerMXBean
Returns:
the config version (from a config properties)

getJobTrackerUrl

public String getJobTrackerUrl()
Specified by:
getJobTrackerUrl in interface TaskTrackerMXBean
Returns:
the URL of the jobtracker

getRpcPort

public int getRpcPort()
Specified by:
getRpcPort in interface TaskTrackerMXBean
Returns:
the RPC port of the tasktracker

getHttpPort

public int getHttpPort()
Specified by:
getHttpPort in interface TaskTrackerMXBean
Returns:
the HTTP port of the tasktracker

isHealthy

public boolean isHealthy()
Specified by:
isHealthy in interface TaskTrackerMXBean
Returns:
the health status of the tasktracker

getTasksInfoJson

public String getTasksInfoJson()
Specified by:
getTasksInfoJson in interface TaskTrackerMXBean
Returns:
a json formatted info about tasks of the tasktracker

updatePrivateDistributedCacheSizes

public void updatePrivateDistributedCacheSizes(JobID jobId,
                                               long[] sizes)
                                        throws IOException
Description copied from interface: TaskUmbilicalProtocol
The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

Specified by:
updatePrivateDistributedCacheSizes in interface TaskUmbilicalProtocol
Parameters:
jobId - the job to update
sizes - the array of sizes that were computed
Throws:
IOException


Copyright © 2009 The Apache Software Foundation