org.apache.hadoop.yarn.server.uam

Class UnmanagedAMPoolManager

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.uam.UnmanagedAMPoolManager

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public class UnmanagedAMPoolManager
extends org.apache.hadoop.service.AbstractService

A service that manages a pool of UAM managers in UnmanagedApplicationManager.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
UnmanagedAMPoolManager(ExecutorService threadpool)

Method Summary

Modifier and Type	Method and Description
void	allocateAsync(String uamId, org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request, org.apache.hadoop.yarn.util.AsyncCallback<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse> callback) AllocateAsync to an UAM.
String	createAndRegisterNewUAM(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest, org.apache.hadoop.conf.Configuration conf, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName) Create a new UAM and register the application, without specifying uamId and appId.
protected UnmanagedApplicationManager	createUAM(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName) Creates the UAM instance.
void	drainUAMHeartbeats()
org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse	finishApplicationMaster(String uamId, org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request) Finish an UAM/application.
Set<String>	getAllUAMIds() Get the id of all running UAMs.
AMRMClientRelayer	getAMRMClientRelayer(String uamId) Return the rmProxy relayer of an UAM.
int	getRequestQueueSize(String uamId)
boolean	hasUAMId(String uamId) Return whether an UAM exists.
org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier>	launchUAM(String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, String rmName) Launch a new UAM, using the provided uamId and appId.
void	reAttachUAM(String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, String queueName, String submitter, String appNameSuffix, org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> uamToken, String rmName) Re-attach to an existing UAM, using the provided uamIdentifier.
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(String uamId, org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest) Register application master for the UAM.
protected void	serviceStart()
protected void	serviceStop() Normally we should finish all applications before stop.
void	shutDownConnections() Shutdown all UAM clients without killing them in YarnRM.
void	shutDownConnections(String uamId) Shutdown an UAM client without killing it in YarnRM.

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

UnmanagedAMPoolManager

public UnmanagedAMPoolManager(ExecutorService threadpool)

Method Detail

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Normally we should finish all applications before stop. If there are still UAMs running, force kill all of them. Do parallel kill because of performance reasons. TODO: move waiting for the kill to finish into a separate thread, without blocking the serviceStop.

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

createAndRegisterNewUAM

public String createAndRegisterNewUAM(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest,
                                      org.apache.hadoop.conf.Configuration conf,
                                      String queueName,
                                      String submitter,
                                      String appNameSuffix,
                                      boolean keepContainersAcrossApplicationAttempts,
                                      String rmName)
                               throws org.apache.hadoop.yarn.exceptions.YarnException,
                                      IOException

Create a new UAM and register the application, without specifying uamId and appId. We will ask for an appId from RM and use it as the uamId.

Parameters:

registerRequest - RegisterApplicationMasterRequest

conf - configuration for this UAM

queueName - queue of the application

submitter - submitter name of the UAM

appNameSuffix - application name suffix for the UAM

keepContainersAcrossApplicationAttempts - keep container flag for UAM recovery.

rmName - name of the YarnRM

Returns:

uamId for the UAM

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if registerApplicationMaster fails

IOException - if registerApplicationMaster fails

See Also:

#setKeepContainersAcrossApplicationAttempts(boolean)

launchUAM

public org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> launchUAM(String uamId,
                                                                                                             org.apache.hadoop.conf.Configuration conf,
                                                                                                             org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                                                                             String queueName,
                                                                                                             String submitter,
                                                                                                             String appNameSuffix,
                                                                                                             boolean keepContainersAcrossApplicationAttempts,
                                                                                                             String rmName)
                                                                                                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                             IOException

Launch a new UAM, using the provided uamId and appId.

Parameters:

uamId - uam Id

conf - configuration for this UAM

appId - application id for the UAM

queueName - queue of the application

submitter - submitter name of the UAM

appNameSuffix - application name suffix for the UAM

keepContainersAcrossApplicationAttempts - keep container flag for UAM recovery.

rmName - name of the YarnRM

Returns:

UAM token

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

IOException - if fails

See Also:

#setKeepContainersAcrossApplicationAttempts(boolean)

reAttachUAM

public void reAttachUAM(String uamId,
                        org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.yarn.api.records.ApplicationId appId,
                        String queueName,
                        String submitter,
                        String appNameSuffix,
                        org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> uamToken,
                        String rmName)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Re-attach to an existing UAM, using the provided uamIdentifier.

Parameters:

uamId - uam Id

conf - configuration for this UAM

appId - application id for the UAM

queueName - queue of the application

submitter - submitter name of the UAM

appNameSuffix - application name suffix for the UAM

uamToken - UAM token

rmName - name of the YarnRM

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

IOException - if fails

createUAM

protected UnmanagedApplicationManager createUAM(org.apache.hadoop.conf.Configuration conf,
                                                org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                String queueName,
                                                String submitter,
                                                String appNameSuffix,
                                                boolean keepContainersAcrossApplicationAttempts,
                                                String rmName)

Creates the UAM instance. Pull out to make unit test easy.

Parameters:

conf - Configuration

appId - application id

queueName - queue of the application

submitter - submitter name of the application

appNameSuffix - application name suffix

keepContainersAcrossApplicationAttempts - keep container flag for UAM

rmName - name of the YarnRM

Returns:

the UAM instance

registerApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String uamId,
                                                                                                              org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest)
                                                                                                       throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                              IOException

Register application master for the UAM.

Parameters:

uamId - uam Id

registerRequest - RegisterApplicationMasterRequest

Returns:

register response

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if register fails

IOException - if register fails

allocateAsync

public void allocateAsync(String uamId,
                          org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request,
                          org.apache.hadoop.yarn.util.AsyncCallback<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse> callback)
                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                          IOException

AllocateAsync to an UAM.

Parameters:

uamId - uam Id

request - AllocateRequest

callback - callback for response

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if allocate fails

IOException - if allocate fails

finishApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse finishApplicationMaster(String uamId,
                                                                                                          org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request)
                                                                                                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                          IOException

Finish an UAM/application.

Parameters:

uamId - uam Id

request - FinishApplicationMasterRequest

Returns:

FinishApplicationMasterResponse

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if finishApplicationMaster call fails

IOException - if finishApplicationMaster call fails

shutDownConnections

public void shutDownConnections(String uamId)
                         throws org.apache.hadoop.yarn.exceptions.YarnException

Shutdown an UAM client without killing it in YarnRM.

Parameters:

uamId - uam Id

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

shutDownConnections

public void shutDownConnections()
                         throws org.apache.hadoop.yarn.exceptions.YarnException

Shutdown all UAM clients without killing them in YarnRM.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

getAllUAMIds

public Set<String> getAllUAMIds()

Get the id of all running UAMs.

Returns:

uamId set

hasUAMId

public boolean hasUAMId(String uamId)

Return whether an UAM exists.

Parameters:

uamId - uam Id

Returns:

UAM exists or not

getAMRMClientRelayer

public AMRMClientRelayer getAMRMClientRelayer(String uamId)
                                       throws org.apache.hadoop.yarn.exceptions.YarnException

Return the rmProxy relayer of an UAM.

Parameters:

uamId - uam Id

Returns:

the rmProxy relayer

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

getRequestQueueSize

public int getRequestQueueSize(String uamId)
                        throws org.apache.hadoop.yarn.exceptions.YarnException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

drainUAMHeartbeats

public void drainUAMHeartbeats()