org.apache.hadoop.mapreduce.v2.app

Class MRAppMaster

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.CompositeService

org.apache.hadoop.mapreduce.v2.app.MRAppMaster

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

public class MRAppMaster
extends org.apache.hadoop.service.CompositeService

The Map-Reduce Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. MR AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MRAppMaster.RunningAppContext

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.security.UserGroupInformation	currentUser
static String	INTERMEDIATE_DATA_ENCRYPTION_ALGO
protected boolean	isLastAMRetry
protected org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager	jobTokenSecretManager
protected MRAppMetrics	metrics
static int	SHUTDOWN_HOOK_PRIORITY Priority of the MRAppMaster shutdown hook.
protected AtomicBoolean	successfullyUnregistered
protected TaskAttemptListener	taskAttemptListener

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Constructor Summary

Constructors

Constructor and Description
MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime)
MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, long appSubmitTime)

Method Summary

Modifier and Type	Method and Description
void	cleanupStagingDir() clean up staging directories for the job.
protected ClientService	createClientService(AppContext context)
protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent>	createCommitterEventHandler(AppContext context, org.apache.hadoop.mapreduce.OutputCommitter committer)
protected ContainerAllocator	createContainerAllocator(ClientService clientService, AppContext context)
protected ContainerLauncher	createContainerLauncher(AppContext context)
protected org.apache.hadoop.yarn.event.Dispatcher	createDispatcher()
protected Job	createJob(org.apache.hadoop.conf.Configuration conf, JobStateInternal forcedState, String diagnostic) Create and initialize (but don't start) a single job.
protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent>	createJobFinishEventHandler() create an event handler that handles the job finish event.
protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>	createJobHistoryHandler(AppContext context)
protected AMPreemptionPolicy	createPreemptionPolicy(org.apache.hadoop.conf.Configuration conf)
protected Speculator	createSpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context)
protected org.apache.hadoop.service.AbstractService	createStagingDirCleaningService()
protected TaskAttemptFinishingMonitor	createTaskAttemptFinishingMonitor(org.apache.hadoop.yarn.event.EventHandler eventHandler)
protected TaskAttemptListener	createTaskAttemptListener(AppContext context, AMPreemptionPolicy preemptionPolicy)
List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo>	getAllAMInfos()
org.apache.hadoop.yarn.api.records.ApplicationId	getAppID()
org.apache.hadoop.yarn.api.records.ApplicationAttemptId	getAttemptID()
ClientService	getClientService()
org.apache.hadoop.mapreduce.OutputCommitter	getCommitter()
Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo>	getCompletedTaskFromPreviousRun()
ContainerAllocator	getContainerAllocator()
ContainerLauncher	getContainerLauncher()
AppContext	getContext()
protected org.apache.hadoop.security.Credentials	getCredentials()
org.apache.hadoop.yarn.event.Dispatcher	getDispatcher()
protected org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.conf.Configuration conf) Create the default file System for this job.
org.apache.hadoop.mapreduce.v2.api.records.JobId	getJobId()
protected RMHeartbeatHandler	getRMHeartbeatHandler()
int	getStartCount()
TaskAttemptListener	getTaskAttemptListener()
protected static void	initAndStartAppMaster(MRAppMaster appMaster, org.apache.hadoop.mapred.JobConf conf, String jobUserName)
protected void	initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf) Obtain the tokens needed by the job and put them in the UGI
Boolean	isLastAMRetry()
boolean	isNewApiCommitter()
protected boolean	keepJobFiles(org.apache.hadoop.mapred.JobConf conf, String jobTempDir)
static void	main(String[] args)
void	notifyIsLastAMRetry(boolean isLastAMRetry)
boolean	recovered()
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
protected void	serviceStop()
void	shutDownJob()
protected void	startJobs() This can be overridden to instantiate multiple jobs and create a workflow.
void	stop()
protected void	sysexit() Exit call.

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, addService, getServices, removeService

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SHUTDOWN_HOOK_PRIORITY

public static final int SHUTDOWN_HOOK_PRIORITY

Priority of the MRAppMaster shutdown hook.

See Also:

Constant Field Values

INTERMEDIATE_DATA_ENCRYPTION_ALGO

public static final String INTERMEDIATE_DATA_ENCRYPTION_ALGO

See Also:

Constant Field Values

metrics

protected final MRAppMetrics metrics

taskAttemptListener

protected TaskAttemptListener taskAttemptListener

jobTokenSecretManager

protected org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager jobTokenSecretManager

currentUser

protected org.apache.hadoop.security.UserGroupInformation currentUser

isLastAMRetry

@VisibleForTesting
protected volatile boolean isLastAMRetry

successfullyUnregistered

@VisibleForTesting
protected AtomicBoolean successfullyUnregistered

Constructor Detail

MRAppMaster

public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                   org.apache.hadoop.yarn.api.records.ContainerId containerId,
                   String nmHost,
                   int nmPort,
                   int nmHttpPort,
                   long appSubmitTime)

MRAppMaster

public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                   org.apache.hadoop.yarn.api.records.ContainerId containerId,
                   String nmHost,
                   int nmPort,
                   int nmHttpPort,
                   org.apache.hadoop.yarn.util.Clock clock,
                   long appSubmitTime)

Method Detail

createTaskAttemptFinishingMonitor

protected TaskAttemptFinishingMonitor createTaskAttemptFinishingMonitor(org.apache.hadoop.yarn.event.EventHandler eventHandler)

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.CompositeService

Throws:

Exception

createDispatcher

protected org.apache.hadoop.yarn.event.Dispatcher createDispatcher()

createPreemptionPolicy

protected AMPreemptionPolicy createPreemptionPolicy(org.apache.hadoop.conf.Configuration conf)

keepJobFiles

protected boolean keepJobFiles(org.apache.hadoop.mapred.JobConf conf,
                               String jobTempDir)

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException

Create the default file System for this job.

Parameters:

conf - the conf object

Returns:

the default filesystem for this job

Throws:

IOException

getCredentials

protected org.apache.hadoop.security.Credentials getCredentials()

cleanupStagingDir

public void cleanupStagingDir()
                       throws IOException

clean up staging directories for the job.

Throws:

IOException

sysexit

protected void sysexit()

Exit call. Just in a function call to enable testing.

shutDownJob

@VisibleForTesting
public void shutDownJob()

createJobFinishEventHandler

protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent> createJobFinishEventHandler()

create an event handler that handles the job finish event.

Returns:

the job finish event handler.

createJob

protected Job createJob(org.apache.hadoop.conf.Configuration conf,
                        JobStateInternal forcedState,
                        String diagnostic)

Create and initialize (but don't start) a single job.

Parameters:

forcedState - a state to force the job into or null for normal operation.

diagnostic - a diagnostic message to include with the job.

initJobCredentialsAndUGI

protected void initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf)

Obtain the tokens needed by the job and put them in the UGI

Parameters:

conf -

createJobHistoryHandler

protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent> createJobHistoryHandler(AppContext context)

createStagingDirCleaningService

protected org.apache.hadoop.service.AbstractService createStagingDirCleaningService()

createSpeculator

protected Speculator createSpeculator(org.apache.hadoop.conf.Configuration conf,
                                      AppContext context)

createTaskAttemptListener

protected TaskAttemptListener createTaskAttemptListener(AppContext context,
                                                        AMPreemptionPolicy preemptionPolicy)

createCommitterEventHandler

protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent> createCommitterEventHandler(AppContext context,
                                                                                                org.apache.hadoop.mapreduce.OutputCommitter committer)

createContainerAllocator

protected ContainerAllocator createContainerAllocator(ClientService clientService,
                                                      AppContext context)

getRMHeartbeatHandler

protected RMHeartbeatHandler getRMHeartbeatHandler()

createContainerLauncher

protected ContainerLauncher createContainerLauncher(AppContext context)

createClientService

protected ClientService createClientService(AppContext context)

getAppID

public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

getAttemptID

public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

getJobId

public org.apache.hadoop.mapreduce.v2.api.records.JobId getJobId()

getCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getCommitter()

isNewApiCommitter

public boolean isNewApiCommitter()

getStartCount

public int getStartCount()

getContext

public AppContext getContext()

getDispatcher

public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

getCompletedTaskFromPreviousRun

public Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo> getCompletedTaskFromPreviousRun()

getAllAMInfos

public List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo> getAllAMInfos()

getContainerAllocator

public ContainerAllocator getContainerAllocator()

getContainerLauncher

public ContainerLauncher getContainerLauncher()

getTaskAttemptListener

public TaskAttemptListener getTaskAttemptListener()

isLastAMRetry

public Boolean isLastAMRetry()

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.CompositeService

Throws:

Exception

stop

public void stop()

Specified by:

stop in interface org.apache.hadoop.service.Service

Overrides:

stop in class org.apache.hadoop.service.AbstractService

recovered

public boolean recovered()

startJobs

protected void startJobs()

This can be overridden to instantiate multiple jobs and create a workflow. TODO: Rework the design to actually support this. Currently much of the job stuff has been moved to init() above to support uberization (MR-1220). In a typical workflow, one presumably would want to uberize only a subset of the jobs (the "small" ones), which is awkward with the current design.

main

public static void main(String[] args)

notifyIsLastAMRetry

public void notifyIsLastAMRetry(boolean isLastAMRetry)

initAndStartAppMaster

protected static void initAndStartAppMaster(MRAppMaster appMaster,
                                            org.apache.hadoop.mapred.JobConf conf,
                                            String jobUserName)
                                     throws IOException,
                                            InterruptedException

Throws:

IOException

InterruptedException

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.CompositeService

Throws:

Exception

getClientService

public ClientService getClientService()