Package org.apache.hadoop.yarn.client.api.async

Class AMRMClientAsync<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.client.api.async.AMRMClientAsync<T>
All Implemented Interfaces:
Closeable, AutoCloseable, Service
@Public @Stable public abstract class AMRMClientAsync<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> extends AbstractService
AMRMClientAsync handles communication with the ResourceManager and provides asynchronous updates on events such as container allocations and completions. It contains a thread that sends periodic heartbeats to the ResourceManager. It should be used by implementing a CallbackHandler: 
 
 class MyCallbackHandler extends AMRMClientAsync.AbstractCallbackHandler {
   public void onContainersAllocated(List<Container> containers) {
     [run tasks on the containers]
   }

   public void onContainersUpdated(List<Container> containers) {
     [determine if resource allocation of containers have been increased in
      the ResourceManager, and if so, inform the NodeManagers to increase the
      resource monitor/enforcement on the containers]
   }

   public void onContainersCompleted(List<ContainerStatus> statuses) {
     [update progress, check whether app is done]
   }
   
   public void onNodesUpdated(List<NodeReport> updated) {}
   
   public void onReboot() {}
 }
The client's lifecycle should be managed similarly to the following: 
 
 AMRMClientAsync asyncClient = 
     createAMRMClientAsync(appAttId, 1000, new MyCallbackhandler());
 asyncClient.init(conf);
 asyncClient.start();
 RegisterApplicationMasterResponse response = asyncClient
    .registerApplicationMaster(appMasterHostname, appMasterRpcPort,
       appMasterTrackingUrl);
 asyncClient.addContainerRequest(containerRequest);
 [... wait for application to complete]
 asyncClient.unregisterApplicationMaster(status, appMsg, trackingUrl);
 asyncClient.stop();

  • Field Details

    • client

      protected final AMRMClient<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> client

    • handler

      protected final org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler handler

    • heartbeatIntervalMs

      protected final AtomicInteger heartbeatIntervalMs

  • Constructor Details

    • AMRMClientAsync

      protected AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

    • AMRMClientAsync

      @Private @VisibleForTesting protected AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

    • AMRMClientAsync

      @Deprecated protected AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
      Deprecated.

    • AMRMClientAsync

      @Private @VisibleForTesting @Deprecated protected AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
      Deprecated.

  • Method Details

    • createAMRMClientAsync

      public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

      Create a new instance of AMRMClientAsync.

      Parameters:
      intervalMs - heartbeat interval in milliseconds between AM and RM
      callbackHandler - callback handler that processes responses from the ResourceManager

    • createAMRMClientAsync

      public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

      Create a new instance of AMRMClientAsync.

      Parameters:
      client - the AMRMClient instance
      intervalMs - heartbeat interval in milliseconds between AM and RM
      callbackHandler - callback handler that processes responses from the ResourceManager

    • createAMRMClientAsync

      @Deprecated public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
      Deprecated.
      Use createAMRMClientAsync(int, AMRMClientAsync.AbstractCallbackHandler) instead.

    • createAMRMClientAsync

      @Deprecated public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
      Deprecated.
      Use createAMRMClientAsync(AMRMClient, int, AMRMClientAsync.AbstractCallbackHandler) instead.

    • setHeartbeatInterval

      public void setHeartbeatInterval(int interval)

    • getMatchingRequests

      public abstract List<? extends Collection<T>> getMatchingRequests(Priority priority, String resourceName, Resource capability)

    • addSchedulingRequests

      @Public @Unstable public void addSchedulingRequests(Collection<SchedulingRequest> schedulingRequests)
      Add a Collection of SchedulingRequests. The AMRMClient will ensure that all requests in the same batch are sent in the same allocate call.
      Parameters:
      schedulingRequests - Collection of Scheduling Requests.

    • getMatchingRequests

      public List<? extends Collection<T>> getMatchingRequests(Priority priority, String resourceName, ExecutionType executionType, Resource capability)
      Returns all matching ContainerRequests that match the given Priority, ResourceName, ExecutionType and Capability. NOTE: This matches only requests that were made by the client WITHOUT the allocationRequestId specified.
      Parameters:
      priority - Priority.
      resourceName - Location.
      executionType - ExecutionType.
      capability - Capability.
      Returns:
      All matching ContainerRequests

    • getMatchingRequests

      public Collection<T> getMatchingRequests(long allocationRequestId)
      Returns all matching ContainerRequests that match the given AllocationRequestId. NOTE: This matches only requests that were made by the client WITH the allocationRequestId specified.
      Parameters:
      allocationRequestId - AllocationRequestId.
      Returns:
      All matching ContainerRequests

    • registerApplicationMaster

      public abstract RegisterApplicationMasterResponse registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl) throws YarnException, IOException
      Registers this application master with the resource manager. On successful registration, starts the heartbeating thread.
      Throws:
      YarnException
      IOException

    • registerApplicationMaster

      @Public @Unstable public RegisterApplicationMasterResponse registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl, Map<Set<String>,PlacementConstraint> placementConstraints) throws YarnException, IOException
      Register the application master. This must be called before any other interaction
      Parameters:
      appHostName - Name of the host on which master is running
      appHostPort - Port master is listening on
      appTrackingUrl - URL at which the master info can be seen
      placementConstraints - Placement Constraints mappings.
      Returns:
      RegisterApplicationMasterResponse
      Throws:
      YarnException
      IOException

    • unregisterApplicationMaster

      public abstract void unregisterApplicationMaster(FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl) throws YarnException, IOException
      Unregister the application master. This must be called in the end.
      Parameters:
      appStatus - Success/Failure status of the master
      appMessage - Diagnostics message on failure
      appTrackingUrl - New URL to get master info
      Throws:
      YarnException
      IOException

    • addContainerRequest

      public abstract void addContainerRequest(T req)
      Request containers for resources before calling allocate
      Parameters:
      req - Resource request

    • removeContainerRequest

      public abstract void removeContainerRequest(T req)
      Remove previous container request. The previous container request may have already been sent to the ResourceManager. So even after the remove request the app must be prepared to receive an allocation for the previous request even after the remove request
      Parameters:
      req - Resource request

    • requestContainerResourceChange

      @Deprecated public void requestContainerResourceChange(Container container, Resource capability)
      Deprecated.
      use requestContainerUpdate(Container, UpdateContainerRequest)
      Request container resource change before calling allocate. Any previous pending resource change request of the same container will be removed. Application that calls this method is expected to maintain the Containers that are returned from previous successful allocations or resource changes. By passing in the existing container and a target resource capability to this method, the application requests the ResourceManager to change the existing resource allocation to the target resource allocation.
      Parameters:
      container - The container returned from the last successful resource allocation or resource change
      capability - The target resource capability of the container

    • requestContainerUpdate

      public abstract void requestContainerUpdate(Container container, UpdateContainerRequest updateContainerRequest)
      Request a container update before calling allocate. Any previous pending update request of the same container will be removed.
      Parameters:
      container - The container returned from the last successful resource allocation or update
      updateContainerRequest - The UpdateContainerRequest.

    • releaseAssignedContainer

      public abstract void releaseAssignedContainer(ContainerId containerId)
      Release containers assigned by the Resource Manager. If the app cannot use the container or wants to give up the container then it can release them. The app needs to make new requests for the released resource capability if it still needs it. eg. it released non-local resources
      Parameters:
      containerId -

    • getAvailableResources

      public abstract Resource getAvailableResources()
      Get the currently available resources in the cluster. A valid value is available after a call to allocate has been made
      Returns:
      Currently available resources

    • getClusterNodeCount

      public abstract int getClusterNodeCount()
      Get the current number of nodes in the cluster. A valid values is available after a call to allocate has been made
      Returns:
      Current number of nodes in the cluster

    • registerTimelineV2Client

      public void registerTimelineV2Client(org.apache.hadoop.yarn.client.api.TimelineV2Client timelineClient) throws YarnException
      Register TimelineClient to AMRMClient.
      Parameters:
      timelineClient -
      Throws:
      YarnException - when this method is invoked even when ATS V2 is not configured.

    • getRegisteredTimelineV2Client

      public org.apache.hadoop.yarn.client.api.TimelineV2Client getRegisteredTimelineV2Client()
      Get registered timeline client.
      Returns:
      the registered timeline client

    • updateBlacklist

      public abstract void updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals)
      Update application's blacklist with addition or removal resources.
      Parameters:
      blacklistAdditions - list of resources which should be added to the application blacklist
      blacklistRemovals - list of resources which should be removed from the application blacklist

    • updateTrackingUrl

      @Public @Unstable public void updateTrackingUrl(String trackingUrl)
      Update application's tracking url on next heartbeat.
      Parameters:
      trackingUrl - new tracking url for this application

    • waitFor

      public void waitFor(Supplier<Boolean> check) throws InterruptedException
      Wait for check to return true for each 1000 ms. See also waitFor(java.util.function.Supplier, int) and waitFor(java.util.function.Supplier, int, int)
      Parameters:
      check - the condition for which it should wait
      Throws:
      InterruptedException

    • waitFor

      public void waitFor(Supplier<Boolean> check, int checkEveryMillis) throws InterruptedException
      Wait for check to return true for each checkEveryMillis ms. See also waitFor(java.util.function.Supplier, int, int)
      Parameters:
      check - user defined checker
      checkEveryMillis - interval to call check
      Throws:
      InterruptedException

    • waitFor

      public void waitFor(Supplier<Boolean> check, int checkEveryMillis, int logInterval) throws InterruptedException
      Wait for check to return true for each checkEveryMillis ms. In the main loop, this method will log the message "waiting in main loop" for each logInterval times iteration to confirm the thread is alive.
      Parameters:
      check - user defined checker
      checkEveryMillis - interval to call check
      logInterval - interval to log for each
      Throws:
      InterruptedException