Package org.apache.hadoop.mapreduce.v2.app

Class MRAppMaster

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.service.CompositeService
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
public class MRAppMaster extends org.apache.hadoop.service.CompositeService
The Map-Reduce Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. MR AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

  • Field Details

    • SHUTDOWN_HOOK_PRIORITY

      public static final int SHUTDOWN_HOOK_PRIORITY
      Priority of the MRAppMaster shutdown hook.
      See Also:

    • INTERMEDIATE_DATA_ENCRYPTION_ALGO

      public static final String INTERMEDIATE_DATA_ENCRYPTION_ALGO
      See Also:

    • metrics

      protected final MRAppMetrics metrics

    • taskAttemptListener

      protected TaskAttemptListener taskAttemptListener

    • jobTokenSecretManager

      protected org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager jobTokenSecretManager

    • currentUser

      protected org.apache.hadoop.security.UserGroupInformation currentUser

    • isLastAMRetry

      @VisibleForTesting protected volatile boolean isLastAMRetry

    • successfullyUnregistered

      @VisibleForTesting protected AtomicBoolean successfullyUnregistered

  • Constructor Details

    • MRAppMaster

      public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, long appSubmitTime)

    • MRAppMaster

      public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime)

  • Method Details

    • createTaskAttemptFinishingMonitor

      protected TaskAttemptFinishingMonitor createTaskAttemptFinishingMonitor(org.apache.hadoop.yarn.event.EventHandler eventHandler)

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      Overrides:
      serviceInit in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • createDispatcher

      protected org.apache.hadoop.yarn.event.Dispatcher createDispatcher()

    • createPreemptionPolicy

      protected AMPreemptionPolicy createPreemptionPolicy(org.apache.hadoop.conf.Configuration conf)

    • keepJobFiles

      protected boolean keepJobFiles(org.apache.hadoop.mapred.JobConf conf, String jobTempDir)

    • getFileSystem

      protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf) throws IOException
      Create the default file System for this job.
      Parameters:
      conf - the conf object
      Returns:
      the default filesystem for this job
      Throws:
      IOException

    • getCredentials

      protected org.apache.hadoop.security.Credentials getCredentials()

    • cleanupStagingDir

      public void cleanupStagingDir() throws IOException
      clean up staging directories for the job.
      Throws:
      IOException

    • sysexit

      protected void sysexit()
      Exit call. Just in a function call to enable testing.

    • shutDownJob

      @VisibleForTesting public void shutDownJob()

    • createJobFinishEventHandler

      protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent> createJobFinishEventHandler()
      create an event handler that handles the job finish event.
      Returns:
      the job finish event handler.

    • createJob

      protected Job createJob(org.apache.hadoop.conf.Configuration conf, JobStateInternal forcedState, String diagnostic)
      Create and initialize (but don't start) a single job.
      Parameters:
      forcedState - a state to force the job into or null for normal operation.
      diagnostic - a diagnostic message to include with the job.

    • initJobCredentialsAndUGI

      protected void initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf)
      Obtain the tokens needed by the job and put them in the UGI
      Parameters:
      conf -

    • createJobHistoryHandler

      protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent> createJobHistoryHandler(AppContext context)

    • createStagingDirCleaningService

      protected org.apache.hadoop.service.AbstractService createStagingDirCleaningService()

    • createSpeculator

      protected Speculator createSpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context)

    • createTaskAttemptListener

      protected TaskAttemptListener createTaskAttemptListener(AppContext context, AMPreemptionPolicy preemptionPolicy)

    • createCommitterEventHandler

      protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent> createCommitterEventHandler(AppContext context, org.apache.hadoop.mapreduce.OutputCommitter committer)

    • createContainerAllocator

      protected ContainerAllocator createContainerAllocator(ClientService clientService, AppContext context)

    • getRMHeartbeatHandler

      protected RMHeartbeatHandler getRMHeartbeatHandler()

    • createContainerLauncher

      protected ContainerLauncher createContainerLauncher(AppContext context)

    • createClientService

      protected ClientService createClientService(AppContext context)

    • getAppID

      public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

    • getAttemptID

      public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

    • getJobId

      public org.apache.hadoop.mapreduce.v2.api.records.JobId getJobId()

    • getCommitter

      public org.apache.hadoop.mapreduce.OutputCommitter getCommitter()

    • isNewApiCommitter

      public boolean isNewApiCommitter()

    • getStartCount

      public int getStartCount()

    • getContext

      public AppContext getContext()

    • getDispatcher

      public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

    • getCompletedTaskFromPreviousRun

      public Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo> getCompletedTaskFromPreviousRun()

    • getAllAMInfos

      public List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo> getAllAMInfos()

    • getContainerAllocator

      public ContainerAllocator getContainerAllocator()

    • getContainerLauncher

      public ContainerLauncher getContainerLauncher()

    • getTaskAttemptListener

      public TaskAttemptListener getTaskAttemptListener()

    • isLastAMRetry

      public Boolean isLastAMRetry()

    • serviceStart

      protected void serviceStart() throws Exception
      Overrides:
      serviceStart in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • stop

      public void stop()
      Specified by:
      stop in interface org.apache.hadoop.service.Service
      Overrides:
      stop in class org.apache.hadoop.service.AbstractService

    • recovered

      public boolean recovered()

    • startJobs

      protected void startJobs()
      This can be overridden to instantiate multiple jobs and create a workflow. TODO: Rework the design to actually support this. Currently much of the job stuff has been moved to init() above to support uberization (MR-1220). In a typical workflow, one presumably would want to uberize only a subset of the jobs (the "small" ones), which is awkward with the current design.

    • main

      public static void main(String[] args)

    • notifyIsLastAMRetry

      public void notifyIsLastAMRetry(boolean isLastAMRetry)

    • initAndStartAppMaster

      protected static void initAndStartAppMaster(MRAppMaster appMaster, org.apache.hadoop.mapred.JobConf conf, String jobUserName) throws IOException, InterruptedException
      Throws:
      IOException
      InterruptedException

    • serviceStop

      protected void serviceStop() throws Exception
      Overrides:
      serviceStop in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • getClientService

      public ClientService getClientService()