Package org.apache.hadoop.yarn.client.api.impl

Class AMRMClientImpl<T extends AMRMClient.ContainerRequest>

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.client.api.AMRMClient<T>
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl<T>
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
@Private @Unstable public class AMRMClientImpl<T extends AMRMClient.ContainerRequest> extends AMRMClient<T>

  • Field Details

    • appHostName

      protected String appHostName

    • appHostPort

      protected int appHostPort

    • appTrackingUrl

      protected String appTrackingUrl

    • newTrackingUrl

      protected String newTrackingUrl

    • rmClient

      protected org.apache.hadoop.yarn.api.ApplicationMasterProtocol rmClient

    • clusterAvailableResources

      protected org.apache.hadoop.yarn.api.records.Resource clusterAvailableResources

    • clusterNodeCount

      protected int clusterNodeCount

    • blacklistedNodes

      protected final Set<String> blacklistedNodes

    • blacklistAdditions

      protected final Set<String> blacklistAdditions

    • blacklistRemovals

      protected final Set<String> blacklistRemovals

    • resourceProfilesMap

      protected Map<String,org.apache.hadoop.yarn.api.records.Resource> resourceProfilesMap

    • ask

      protected final Set<org.apache.hadoop.yarn.api.records.ResourceRequest> ask

    • release

      protected final Set<org.apache.hadoop.yarn.api.records.ContainerId> release

    • pendingRelease

      protected Set<org.apache.hadoop.yarn.api.records.ContainerId> pendingRelease

    • change

      protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> change

    • pendingChange

      protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> pendingChange

  • Constructor Details

    • AMRMClientImpl

      public AMRMClientImpl()

  • Method Details

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      Overrides:
      serviceInit in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStart

      protected void serviceStart() throws Exception
      Overrides:
      serviceStart in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStop

      protected void serviceStop() throws Exception
      Overrides:
      serviceStop in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • registerApplicationMaster

      public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: AMRMClient
      Register the application master. This must be called before any other interaction
      Specified by:
      registerApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      appHostName - Name of the host on which master is running
      appHostPort - Port master is listening on
      appTrackingUrl - URL at which the master info can be seen
      Returns:
      RegisterApplicationMasterResponse
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException
      IOException

    • registerApplicationMaster

      public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl, Map<Set<String>,org.apache.hadoop.yarn.api.resource.PlacementConstraint> placementConstraintsMap) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: AMRMClient
      Register the application master. This must be called before any other interaction
      Overrides:
      registerApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      appHostName - Name of the host on which master is running
      appHostPort - Port master is listening on
      appTrackingUrl - URL at which the master info can be seen
      placementConstraintsMap - Placement Constraints mappings.
      Returns:
      RegisterApplicationMasterResponse
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException
      IOException

    • addSchedulingRequests

      public void addSchedulingRequests(Collection<org.apache.hadoop.yarn.api.records.SchedulingRequest> newSchedulingRequests)
      Description copied from class: AMRMClient
      Add a Collection of SchedulingRequests. The AMRMClient will ensure that all requests in the same batch are sent in the same allocate call.
      Overrides:
      addSchedulingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      newSchedulingRequests - Collection of Scheduling Requests.

    • allocate

      public org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse allocate(float progressIndicator) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: AMRMClient
      Request additional containers and receive new container allocations. Requests made via addContainerRequest are sent to the ResourceManager. New containers assigned to the master are retrieved. Status of completed containers and node health updates are also retrieved. This also doubles up as a heartbeat to the ResourceManager and must be made periodically. The call may not always return any new allocations of containers. App should not make concurrent allocate requests. May cause request loss.

      Note : If the user has not removed container requests that have already been satisfied, then the re-register may end up sending the entire container requests to the RM (including matched requests). Which would mean the RM could end up giving it a lot of new allocated containers.

      Specified by:
      allocate in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      progressIndicator - Indicates progress made by the master
      Returns:
      the response of the allocate request
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException
      IOException

    • removePendingReleaseRequests

      protected void removePendingReleaseRequests(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainersStatuses)

    • removePendingChangeRequests

      protected void removePendingChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdatedContainer> changedContainers)

    • populateNMTokens

      @Private @VisibleForTesting protected void populateNMTokens(List<org.apache.hadoop.yarn.api.records.NMToken> nmTokens)

    • unregisterApplicationMaster

      public void unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: AMRMClient
      Unregister the application master. This must be called in the end.
      Specified by:
      unregisterApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      appStatus - Success/Failure status of the master
      appMessage - Diagnostics message on failure
      appTrackingUrl - New URL to get master info
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException
      IOException

    • addContainerRequest

      public void addContainerRequest(T req)
      Description copied from class: AMRMClient
      Request containers for resources before calling allocate
      Specified by:
      addContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      req - Resource request

    • removeContainerRequest

      public void removeContainerRequest(T req)
      Description copied from class: AMRMClient
      Remove previous container request. The previous container request may have already been sent to the ResourceManager. So even after the remove request the app must be prepared to receive an allocation for the previous request even after the remove request
      Specified by:
      removeContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      req - Resource request

    • requestContainerUpdate

      public void requestContainerUpdate(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.UpdateContainerRequest updateContainerRequest)
      Description copied from class: AMRMClient
      Request a container update before calling allocate. Any previous pending update request of the same container will be removed.
      Specified by:
      requestContainerUpdate in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      container - The container returned from the last successful resource allocation or update
      updateContainerRequest - The UpdateContainerRequest.

    • releaseAssignedContainer

      public void releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
      Description copied from class: AMRMClient
      Release containers assigned by the Resource Manager. If the app cannot use the container or wants to give up the container then it can release them. The app needs to make new requests for the released resource capability if it still needs it. eg. it released non-local resources
      Specified by:
      releaseAssignedContainer in class AMRMClient<T extends AMRMClient.ContainerRequest>

    • getAvailableResources

      public org.apache.hadoop.yarn.api.records.Resource getAvailableResources()
      Description copied from class: AMRMClient
      Get the currently available resources in the cluster. A valid value is available after a call to allocate has been made
      Specified by:
      getAvailableResources in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Returns:
      Currently available resources

    • getClusterNodeCount

      public int getClusterNodeCount()
      Description copied from class: AMRMClient
      Get the current number of nodes in the cluster. A valid values is available after a call to allocate has been made
      Specified by:
      getClusterNodeCount in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Returns:
      Current number of nodes in the cluster

    • getMatchingRequests

      public Collection<T> getMatchingRequests(long allocationRequestId)
      Description copied from class: AMRMClient
      Get outstanding ContainerRequests matching the given allocationRequestId. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. NOTE: This API only matches Container requests that were created by the client WITH the allocationRequestId being set to a non-default value.
      Specified by:
      getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      allocationRequestId - Allocation Request Id
      Returns:
      Collection of request matching the parameters

    • getMatchingRequests

      public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.Resource capability)
      Description copied from class: AMRMClient
      Get outstanding ContainerRequests matching the given parameters. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. Each collection in the list contains requests with identical Resource size that fit in the given capability. In a collection, requests will be returned in the same order as they were added. NOTE: This API only matches Container requests that were created by the client WITHOUT the allocationRequestId being set.
      Specified by:
      getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Returns:
      Collection of request matching the parameters

    • getMatchingRequests

      public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.ExecutionType executionType, org.apache.hadoop.yarn.api.records.Resource capability, String profile)
      Overrides:
      getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>

    • getMatchingRequests

      public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.ExecutionType executionType, org.apache.hadoop.yarn.api.records.Resource capability)
      Description copied from class: AMRMClient
      Get outstanding ContainerRequests matching the given parameters. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. Each collection in the list contains requests with identical Resource size that fit in the given capability. In a collection, requests will be returned in the same order as they were added. specify an ExecutionType. NOTE: This API only matches Container requests that were created by the client WITHOUT the allocationRequestId being set.
      Overrides:
      getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      priority - Priority
      resourceName - Location
      executionType - ExecutionType
      capability - Capability
      Returns:
      Collection of request matching the parameters

    • updateBlacklist

      public void updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals)
      Description copied from class: AMRMClient
      Update application's blacklist with addition or removal resources.
      Specified by:
      updateBlacklist in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      blacklistAdditions - list of resources which should be added to the application blacklist
      blacklistRemovals - list of resources which should be removed from the application blacklist

    • updateTrackingUrl

      public void updateTrackingUrl(String trackingUrl)
      Description copied from class: AMRMClient
      Update application's tracking url on next heartbeat.
      Overrides:
      updateTrackingUrl in class AMRMClient<T extends AMRMClient.ContainerRequest>
      Parameters:
      trackingUrl - new tracking url for this application