Package org.apache.hadoop.yarn.server.uam

Class UnmanagedAMPoolManager

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.server.uam.UnmanagedAMPoolManager
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
@Public @Unstable public class UnmanagedAMPoolManager extends org.apache.hadoop.service.AbstractService
A service that manages a pool of UAM managers in UnmanagedApplicationManager.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

    org.apache.hadoop.service.Service.STATE

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final org.slf4j.Logger
    LOG
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    UnmanagedAMPoolManager(ExecutorService threadpool)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    allocateAsync(String uamId, org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request, org.apache.hadoop.yarn.util.AsyncCallback<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse> callback)
    AllocateAsync to an UAM.
    Map<String,org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse>
    batchFinishApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request, String appId)
    Complete FinishApplicationMaster interface calls in batches.
    String
    createAndRegisterNewUAM(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest, org.apache.hadoop.conf.Configuration conf, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
    Create a new UAM and register the application, without specifying uamId and appId.
    protected UnmanagedApplicationManager
    createUAM(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
    Creates the UAM instance.
    void
    drainUAMHeartbeats()
     
    org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse
    finishApplicationMaster(String uamId, org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request)
    Finish an UAM/application.
    Set<String>
    getAllUAMIds()
    Get the id of all running UAMs.
    AMRMClientRelayer
    getAMRMClientRelayer(String uamId)
    Return the rmProxy relayer of an UAM.
    protected Thread
    getFinishApplicationThread()
     
    int
    getRequestQueueSize(String uamId)
     
    protected Map<String,UnmanagedApplicationManager>
    getUnmanagedAppMasterMap()
     
    boolean
    hasUAMId(String uamId)
    Return whether an UAM exists.
    org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier>
    launchUAM(String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
    Launch a new UAM, using the provided uamId and appId.
    void
    reAttachUAM(String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> uamToken, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
    Re-attach to an existing UAM, using the provided uamIdentifier.
    org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse
    registerApplicationMaster(String uamId, org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest)
    Register application master for the UAM.
    protected void
    serviceStart()
     
    protected void
    serviceStop()
    Normally we should finish all applications before stop.
    void
    shutDownConnections()
    Shutdown all UAM clients without killing them in YarnRM.
    void
    shutDownConnections(String uamId)
    Shutdown an UAM client without killing it in YarnRM.
    void
    unAttachUAM(String uamId)
     

    Methods inherited from class org.apache.hadoop.service.AbstractService

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

  • Constructor Details

    • UnmanagedAMPoolManager

      public UnmanagedAMPoolManager(ExecutorService threadpool)

  • Method Details

    • serviceStart

      protected void serviceStart() throws Exception
      Overrides:
      serviceStart in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStop

      protected void serviceStop() throws Exception
      Normally we should finish all applications before stop. If there are still UAMs running, force kill all of them. Do parallel kill because of performance reasons.
      Overrides:
      serviceStop in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • createAndRegisterNewUAM

      public String createAndRegisterNewUAM(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest, org.apache.hadoop.conf.Configuration conf, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Create a new UAM and register the application, without specifying uamId and appId. We will ask for an appId from RM and use it as the uamId.
      Parameters:
      registerRequest - RegisterApplicationMasterRequest
      conf - configuration for this UAM
      queueName - queue of the application
      submitter - submitter name of the UAM
      appNameSuffix - application name suffix for the UAM
      keepContainersAcrossApplicationAttempts - keep container flag for UAM recovery.
      rmName - name of the YarnRM
      originalAppSubmissionContext - ApplicationSubmissionContext
      Returns:
      uamId for the UAM
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if registerApplicationMaster fails
      IOException - if registerApplicationMaster fails
      See Also:
      • #setKeepContainersAcrossApplicationAttempts(boolean)

    • launchUAM

      public org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> launchUAM(String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Launch a new UAM, using the provided uamId and appId.
      Parameters:
      uamId - uam Id
      conf - configuration for this UAM
      appId - application id for the UAM
      queueName - queue of the application
      submitter - submitter name of the UAM
      appNameSuffix - application name suffix for the UAM
      keepContainersAcrossApplicationAttempts - keep container flag for UAM recovery.
      rmName - name of the YarnRM
      originalAppSubmissionContext - AppSubmissionContext
      Returns:
      UAM token
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if fails
      IOException - if fails
      See Also:
      • #setKeepContainersAcrossApplicationAttempts(boolean)

    • reAttachUAM

      public void reAttachUAM(String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> uamToken, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Re-attach to an existing UAM, using the provided uamIdentifier.
      Parameters:
      uamId - uam Id
      conf - configuration for this UAM
      appId - application id for the UAM
      queueName - queue of the application
      submitter - submitter name of the UAM
      appNameSuffix - application name suffix for the UAM
      uamToken - UAM token
      rmName - name of the YarnRM
      originalAppSubmissionContext - AppSubmissionContext
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if fails
      IOException - if fails

    • createUAM

      @VisibleForTesting protected UnmanagedApplicationManager createUAM(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
      Creates the UAM instance. Pull out to make unit test easy.
      Parameters:
      conf - Configuration
      appId - application id
      queueName - queue of the application
      submitter - submitter name of the application
      appNameSuffix - application name suffix
      keepContainersAcrossApplicationAttempts - keep container flag for UAM
      rmName - name of the YarnRM
      originalAppSubmissionContext - ApplicationSubmissionContext
      Returns:
      the UAM instance

    • registerApplicationMaster

      public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String uamId, org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Register application master for the UAM.
      Parameters:
      uamId - uam Id
      registerRequest - RegisterApplicationMasterRequest
      Returns:
      register response
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if register fails
      IOException - if register fails

    • allocateAsync

      public void allocateAsync(String uamId, org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request, org.apache.hadoop.yarn.util.AsyncCallback<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse> callback) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      AllocateAsync to an UAM.
      Parameters:
      uamId - uam Id
      request - AllocateRequest
      callback - callback for response
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if allocate fails
      IOException - if allocate fails

    • finishApplicationMaster

      public org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse finishApplicationMaster(String uamId, org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Finish an UAM/application.
      Parameters:
      uamId - uam Id
      request - FinishApplicationMasterRequest
      Returns:
      FinishApplicationMasterResponse
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if finishApplicationMaster call fails
      IOException - if finishApplicationMaster call fails

    • shutDownConnections

      public void shutDownConnections(String uamId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Shutdown an UAM client without killing it in YarnRM.
      Parameters:
      uamId - uam Id
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if fails

    • shutDownConnections

      public void shutDownConnections() throws org.apache.hadoop.yarn.exceptions.YarnException
      Shutdown all UAM clients without killing them in YarnRM.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if fails

    • getAllUAMIds

      public Set<String> getAllUAMIds()
      Get the id of all running UAMs.
      Returns:
      uamId set

    • hasUAMId

      public boolean hasUAMId(String uamId)
      Return whether an UAM exists.
      Parameters:
      uamId - uam Id
      Returns:
      UAM exists or not

    • getAMRMClientRelayer

      public AMRMClientRelayer getAMRMClientRelayer(String uamId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Return the rmProxy relayer of an UAM.
      Parameters:
      uamId - uam Id
      Returns:
      the rmProxy relayer
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if fails

    • getRequestQueueSize

      @VisibleForTesting public int getRequestQueueSize(String uamId) throws org.apache.hadoop.yarn.exceptions.YarnException
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException

    • drainUAMHeartbeats

      @VisibleForTesting public void drainUAMHeartbeats()

    • batchFinishApplicationMaster

      public Map<String,org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse> batchFinishApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request, String appId)
      Complete FinishApplicationMaster interface calls in batches.
      Parameters:
      request - FinishApplicationMasterRequest
      appId - application Id
      Returns:
      Returns the Map, the key is subClusterId, the value is FinishApplicationMasterResponse

    • unAttachUAM

      public void unAttachUAM(String uamId)

    • getUnmanagedAppMasterMap

      @VisibleForTesting protected Map<String,UnmanagedApplicationManager> getUnmanagedAppMasterMap()

    • getFinishApplicationThread

      @VisibleForTesting protected Thread getFinishApplicationThread()