org.apache.hadoop.yarn.client.api.async

Class AMRMClientAsync<T extends AMRMClient.ContainerRequest>

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.client.api.async.AMRMClientAsync<T>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

Direct Known Subclasses:

AMRMClientAsyncImpl

@InterfaceAudience.Public
 @InterfaceStability.Stable
public abstract class AMRMClientAsync<T extends AMRMClient.ContainerRequest>
extends org.apache.hadoop.service.AbstractService

AMRMClientAsync handles communication with the ResourceManager and provides asynchronous updates on events such as container allocations and completions. It contains a thread that sends periodic heartbeats to the ResourceManager. It should be used by implementing a CallbackHandler:

 
 class MyCallbackHandler extends AMRMClientAsync.AbstractCallbackHandler {
   public void onContainersAllocated(List<Container> containers) {
     [run tasks on the containers]
   }

   public void onContainersUpdated(List<Container> containers) {
     [determine if resource allocation of containers have been increased in
      the ResourceManager, and if so, inform the NodeManagers to increase the
      resource monitor/enforcement on the containers]
   }

   public void onContainersCompleted(List<ContainerStatus> statuses) {
     [update progress, check whether app is done]
   }
   
   public void onNodesUpdated(List<NodeReport> updated) {}
   
   public void onReboot() {}
 }
 
 

The client's lifecycle should be managed similarly to the following:

 
 AMRMClientAsync asyncClient = 
     createAMRMClientAsync(appAttId, 1000, new MyCallbackhandler());
 asyncClient.init(conf);
 asyncClient.start();
 RegisterApplicationMasterResponse response = asyncClient
    .registerApplicationMaster(appMasterHostname, appMasterRpcPort,
       appMasterTrackingUrl);
 asyncClient.addContainerRequest(containerRequest);
 [... wait for application to complete]
 asyncClient.unregisterApplicationMaster(status, appMsg, trackingUrl);
 asyncClient.stop();
 
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AMRMClientAsync.AbstractCallbackHandler The callback abstract class.
static interface	AMRMClientAsync.CallbackHandler Deprecated. Use AMRMClientAsync.AbstractCallbackHandler instead.

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected AMRMClient<T>	client
protected AMRMClientAsync.CallbackHandler	handler
protected AtomicInteger	heartbeatIntervalMs

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AMRMClientAsync(AMRMClient<T> client, int intervalMs, AMRMClientAsync.AbstractCallbackHandler callbackHandler)
protected	AMRMClientAsync(AMRMClient<T> client, int intervalMs, AMRMClientAsync.CallbackHandler callbackHandler) Deprecated.
protected	AMRMClientAsync(int intervalMs, AMRMClientAsync.AbstractCallbackHandler callbackHandler)
protected	AMRMClientAsync(int intervalMs, AMRMClientAsync.CallbackHandler callbackHandler) Deprecated.

Method Summary

Modifier and Type	Method and Description
abstract void	addContainerRequest(T req) Request containers for resources before calling allocate
void	addSchedulingRequests(Collection<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests) Add a Collection of SchedulingRequests.
static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T>	createAMRMClientAsync(AMRMClient<T> client, int intervalMs, AMRMClientAsync.AbstractCallbackHandler callbackHandler) Create a new instance of AMRMClientAsync.
static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T>	createAMRMClientAsync(AMRMClient<T> client, int intervalMs, AMRMClientAsync.CallbackHandler callbackHandler) Deprecated. Use createAMRMClientAsync(AMRMClient, int, AMRMClientAsync.AbstractCallbackHandler) instead.
static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T>	createAMRMClientAsync(int intervalMs, AMRMClientAsync.AbstractCallbackHandler callbackHandler) Create a new instance of AMRMClientAsync.
static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T>	createAMRMClientAsync(int intervalMs, AMRMClientAsync.CallbackHandler callbackHandler) Deprecated. Use createAMRMClientAsync(int, AMRMClientAsync.AbstractCallbackHandler) instead.
abstract org.apache.hadoop.yarn.api.records.Resource	getAvailableResources() Get the currently available resources in the cluster.
abstract int	getClusterNodeCount() Get the current number of nodes in the cluster.
Collection<T>	getMatchingRequests(long allocationRequestId) Returns all matching ContainerRequests that match the given AllocationRequestId.
List<? extends Collection<T>>	getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.ExecutionType executionType, org.apache.hadoop.yarn.api.records.Resource capability) Returns all matching ContainerRequests that match the given Priority, ResourceName, ExecutionType and Capability.
abstract List<? extends Collection<T>>	getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.Resource capability)
org.apache.hadoop.yarn.client.api.TimelineV2Client	getRegisteredTimelineV2Client() Get registered timeline client.
abstract org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl) Registers this application master with the resource manager.
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl, Map<Set<String>,org.apache.hadoop.yarn.api.resource.PlacementConstraint> placementConstraints) Register the application master.
void	registerTimelineV2Client(org.apache.hadoop.yarn.client.api.TimelineV2Client timelineClient) Register TimelineClient to AMRMClient.
abstract void	releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Release containers assigned by the Resource Manager.
abstract void	removeContainerRequest(T req) Remove previous container request.
void	requestContainerResourceChange(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.Resource capability) Deprecated. use requestContainerUpdate(Container, UpdateContainerRequest)
abstract void	requestContainerUpdate(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.UpdateContainerRequest updateContainerRequest) Request a container update before calling allocate.
void	setHeartbeatInterval(int interval)
abstract void	unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl) Unregister the application master.
abstract void	updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals) Update application's blacklist with addition or removal resources.
void	waitFor(Supplier<Boolean> check) Wait for check to return true for each 1000 ms.
void	waitFor(Supplier<Boolean> check, int checkEveryMillis) Wait for check to return true for each checkEveryMillis ms.
void	waitFor(Supplier<Boolean> check, int checkEveryMillis, int logInterval) Wait for check to return true for each checkEveryMillis ms.

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, serviceStart, serviceStop, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

client

protected final AMRMClient<T extends AMRMClient.ContainerRequest> client

handler

protected final AMRMClientAsync.CallbackHandler handler

heartbeatIntervalMs

protected final AtomicInteger heartbeatIntervalMs

Constructor Detail

AMRMClientAsync

protected AMRMClientAsync(int intervalMs,
                          AMRMClientAsync.AbstractCallbackHandler callbackHandler)

AMRMClientAsync

@InterfaceAudience.Private
protected AMRMClientAsync(AMRMClient<T> client,
                                                     int intervalMs,
                                                     AMRMClientAsync.AbstractCallbackHandler callbackHandler)

AMRMClientAsync

@Deprecated
protected AMRMClientAsync(int intervalMs,
                                      AMRMClientAsync.CallbackHandler callbackHandler)

Deprecated.

AMRMClientAsync

@InterfaceAudience.Private
 @Deprecated
protected AMRMClientAsync(AMRMClient<T> client,
                                                                  int intervalMs,
                                                                  AMRMClientAsync.CallbackHandler callbackHandler)

Deprecated.

Method Detail

createAMRMClientAsync

public static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs,
                                                                                               AMRMClientAsync.AbstractCallbackHandler callbackHandler)

Create a new instance of AMRMClientAsync.

Parameters:

intervalMs - heartbeat interval in milliseconds between AM and RM

callbackHandler - callback handler that processes responses from the ResourceManager

createAMRMClientAsync

public static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client,
                                                                                               int intervalMs,
                                                                                               AMRMClientAsync.AbstractCallbackHandler callbackHandler)

Create a new instance of AMRMClientAsync.

Parameters:

client - the AMRMClient instance

intervalMs - heartbeat interval in milliseconds between AM and RM

callbackHandler - callback handler that processes responses from the ResourceManager

createAMRMClientAsync

@Deprecated
public static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs,
                                                                                                           AMRMClientAsync.CallbackHandler callbackHandler)

Deprecated. Use createAMRMClientAsync(int, AMRMClientAsync.AbstractCallbackHandler) instead.

createAMRMClientAsync

@Deprecated
public static <T extends AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client,
                                                                                                           int intervalMs,
                                                                                                           AMRMClientAsync.CallbackHandler callbackHandler)

Deprecated. Use createAMRMClientAsync(AMRMClient, int, AMRMClientAsync.AbstractCallbackHandler) instead.

setHeartbeatInterval

public void setHeartbeatInterval(int interval)

getMatchingRequests

public abstract List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
                                                                  String resourceName,
                                                                  org.apache.hadoop.yarn.api.records.Resource capability)

addSchedulingRequests

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public void addSchedulingRequests(Collection<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests)

Add a Collection of SchedulingRequests. The AMRMClient will ensure that all requests in the same batch are sent in the same allocate call.

Parameters:

schedulingRequests - Collection of Scheduling Requests.

getMatchingRequests

public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
                                                         String resourceName,
                                                         org.apache.hadoop.yarn.api.records.ExecutionType executionType,
                                                         org.apache.hadoop.yarn.api.records.Resource capability)

Returns all matching ContainerRequests that match the given Priority, ResourceName, ExecutionType and Capability. NOTE: This matches only requests that were made by the client WITHOUT the allocationRequestId specified.

Parameters:

priority - Priority.

resourceName - Location.

executionType - ExecutionType.

capability - Capability.

Returns:

All matching ContainerRequests

getMatchingRequests

public Collection<T> getMatchingRequests(long allocationRequestId)

Returns all matching ContainerRequests that match the given AllocationRequestId. NOTE: This matches only requests that were made by the client WITH the allocationRequestId specified.

Parameters:

allocationRequestId - AllocationRequestId.

Returns:

All matching ContainerRequests

registerApplicationMaster

public abstract org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName,
                                                                                                                       int appHostPort,
                                                                                                                       String appTrackingUrl)
                                                                                                                throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                                       IOException

Registers this application master with the resource manager. On successful registration, starts the heartbeating thread.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

registerApplicationMaster

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName,
                                                                                                                                                                      int appHostPort,
                                                                                                                                                                      String appTrackingUrl,
                                                                                                                                                                      Map<Set<String>,org.apache.hadoop.yarn.api.resource.PlacementConstraint> placementConstraints)
                                                                                                                                                               throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                                                                                      IOException

Register the application master. This must be called before any other interaction

Parameters:

appHostName - Name of the host on which master is running

appHostPort - Port master is listening on

appTrackingUrl - URL at which the master info can be seen

placementConstraints - Placement Constraints mappings.

Returns:

RegisterApplicationMasterResponse

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

unregisterApplicationMaster

public abstract void unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus,
                                                 String appMessage,
                                                 String appTrackingUrl)
                                          throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                 IOException

Unregister the application master. This must be called in the end.

Parameters:

appStatus - Success/Failure status of the master

appMessage - Diagnostics message on failure

appTrackingUrl - New URL to get master info

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

addContainerRequest

public abstract void addContainerRequest(T req)

Request containers for resources before calling allocate

Parameters:

req - Resource request

removeContainerRequest

public abstract void removeContainerRequest(T req)

Remove previous container request. The previous container request may have already been sent to the ResourceManager. So even after the remove request the app must be prepared to receive an allocation for the previous request even after the remove request

Parameters:

req - Resource request

requestContainerResourceChange

@Deprecated
public void requestContainerResourceChange(org.apache.hadoop.yarn.api.records.Container container,
                                                       org.apache.hadoop.yarn.api.records.Resource capability)

Deprecated. use requestContainerUpdate(Container, UpdateContainerRequest)

Request container resource change before calling allocate. Any previous pending resource change request of the same container will be removed. Application that calls this method is expected to maintain the Containers that are returned from previous successful allocations or resource changes. By passing in the existing container and a target resource capability to this method, the application requests the ResourceManager to change the existing resource allocation to the target resource allocation.

Parameters:

container - The container returned from the last successful resource allocation or resource change

capability - The target resource capability of the container

requestContainerUpdate

public abstract void requestContainerUpdate(org.apache.hadoop.yarn.api.records.Container container,
                                            org.apache.hadoop.yarn.api.records.UpdateContainerRequest updateContainerRequest)

Request a container update before calling allocate. Any previous pending update request of the same container will be removed.

Parameters:

container - The container returned from the last successful resource allocation or update

updateContainerRequest - The UpdateContainerRequest.

releaseAssignedContainer

public abstract void releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Release containers assigned by the Resource Manager. If the app cannot use the container or wants to give up the container then it can release them. The app needs to make new requests for the released resource capability if it still needs it. eg. it released non-local resources

Parameters:

containerId -

getAvailableResources

public abstract org.apache.hadoop.yarn.api.records.Resource getAvailableResources()

Get the currently available resources in the cluster. A valid value is available after a call to allocate has been made

Returns:

Currently available resources

getClusterNodeCount

public abstract int getClusterNodeCount()

Get the current number of nodes in the cluster. A valid values is available after a call to allocate has been made

Returns:

Current number of nodes in the cluster

registerTimelineV2Client

public void registerTimelineV2Client(org.apache.hadoop.yarn.client.api.TimelineV2Client timelineClient)
                              throws org.apache.hadoop.yarn.exceptions.YarnException

Register TimelineClient to AMRMClient.

Parameters:

timelineClient -

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when this method is invoked even when ATS V2 is not configured.

getRegisteredTimelineV2Client

public org.apache.hadoop.yarn.client.api.TimelineV2Client getRegisteredTimelineV2Client()

Get registered timeline client.

Returns:

the registered timeline client

updateBlacklist

public abstract void updateBlacklist(List<String> blacklistAdditions,
                                     List<String> blacklistRemovals)

Update application's blacklist with addition or removal resources.

Parameters:

blacklistAdditions - list of resources which should be added to the application blacklist

blacklistRemovals - list of resources which should be removed from the application blacklist

waitFor

public void waitFor(Supplier<Boolean> check)
             throws InterruptedException

Wait for check to return true for each 1000 ms. See also waitFor(java.util.function.Supplier, int) and waitFor(java.util.function.Supplier, int, int)

Parameters:

check - the condition for which it should wait

Throws:

InterruptedException

waitFor

public void waitFor(Supplier<Boolean> check,
                    int checkEveryMillis)
             throws InterruptedException

Wait for check to return true for each checkEveryMillis ms. See also waitFor(java.util.function.Supplier, int, int)

Parameters:

check - user defined checker

checkEveryMillis - interval to call check

Throws:

InterruptedException

waitFor

public void waitFor(Supplier<Boolean> check,
                    int checkEveryMillis,
                    int logInterval)
             throws InterruptedException

Wait for check to return true for each checkEveryMillis ms. In the main loop, this method will log the message "waiting in main loop" for each logInterval times iteration to confirm the thread is alive.

Parameters:

check - user defined checker

checkEveryMillis - interval to call check

logInterval - interval to log for each

Throws:

InterruptedException