Package org.apache.hadoop.yarn.client.api.impl

Class NMClientImpl

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.client.api.NMClient
org.apache.hadoop.yarn.client.api.impl.NMClientImpl
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
@Private @Unstable public class NMClientImpl extends NMClient

This class implements NMClient. All the APIs are blocking.

By default, this client stops all the running containers that are started by it when it stops. It can be disabled via cleanupRunningContainersOnStop(boolean), in which case containers will continue to run even after this client is stopped and till the application runs at which point ResourceManager will forcefully kill them.

Note that the blocking APIs ensure the RPC calls to NodeManager are executed immediately, and the responses are received before these APIs return. However, when startContainer(org.apache.hadoop.yarn.api.records.Container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext) or stopContainer(org.apache.hadoop.yarn.api.records.ContainerId, org.apache.hadoop.yarn.api.records.NodeId) returns, NodeManager may still need some time to either start or stop the container because of its asynchronous implementation. Therefore, getContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId, org.apache.hadoop.yarn.api.records.NodeId) is likely to return a transit container status if it is executed immediately after startContainer(org.apache.hadoop.yarn.api.records.Container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext) or stopContainer(org.apache.hadoop.yarn.api.records.ContainerId, org.apache.hadoop.yarn.api.records.NodeId).

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    protected static class 
    NMClientImpl.StartedContainer
     

    Nested classes/interfaces inherited from class org.apache.hadoop.yarn.client.api.NMClient

    NMClient.UpgradeOp

    Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

    org.apache.hadoop.service.Service.STATE

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected ConcurrentMap<org.apache.hadoop.yarn.api.records.ContainerId,NMClientImpl.StartedContainer>
    startedContainers
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    NMClientImpl()
     
    NMClientImpl(String name)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected void
    cleanupRunningContainers()
     
    void
    cleanupRunningContainersOnStop(boolean enabled)
    Set whether the containers that are started by this client, and are still running should be stopped when the client stops.
    void
    commitLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId)
    Commit last reInitialization of the specified container.
    AtomicBoolean
    getCleanupRunningContainers()
     
    org.apache.hadoop.yarn.api.records.ContainerStatus
    getContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId)
    Query the status of a container.
    List<org.apache.hadoop.yarn.api.records.LocalizationStatus>
    getLocalizationStatuses(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId)
    Get the localization statuses of a container.
    org.apache.hadoop.yarn.api.records.NodeId
    getNodeIdOfStartedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
    Get the NodeId of the node on which container is running.
    void
    increaseContainerResource(org.apache.hadoop.yarn.api.records.Container container)
    Deprecated.
    void
    localize(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
    Localize resources for a container.
    void
    reInitializeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContex, boolean autoCommit)
    Re-Initialize the Container.
    void
    restartContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
    Restart the specified container.
    void
    rollbackLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId)
    Rollback last reInitialization of the specified container.
    protected void
    serviceInit(org.apache.hadoop.conf.Configuration conf)
     
    protected void
    serviceStop()
     
    Map<String,ByteBuffer>
    startContainer(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContext)
    Start an allocated container.
    void
    stopContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId)
    Stop an started container.
    void
    updateContainerResource(org.apache.hadoop.yarn.api.records.Container container)
    Update the resources of a container.

    Methods inherited from class org.apache.hadoop.yarn.client.api.NMClient

    createNMClient, createNMClient, getNMTokenCache, setNMTokenCache

    Methods inherited from class org.apache.hadoop.service.AbstractService

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceStart, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

  • Constructor Details

    • NMClientImpl

      public NMClientImpl()

    • NMClientImpl

      public NMClientImpl(String name)

  • Method Details

    • serviceStop

      protected void serviceStop() throws Exception
      Overrides:
      serviceStop in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • cleanupRunningContainers

      protected void cleanupRunningContainers()

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      Overrides:
      serviceInit in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • cleanupRunningContainersOnStop

      public void cleanupRunningContainersOnStop(boolean enabled)
      Description copied from class: NMClient

      Set whether the containers that are started by this client, and are still running should be stopped when the client stops. By default, the feature should be enabled.

      However, containers will be stopped only when service is stopped. i.e. after AbstractService.stop().
      Specified by:
      cleanupRunningContainersOnStop in class NMClient
      Parameters:
      enabled - whether the feature is enabled or not

    • startContainer

      public Map<String,ByteBuffer> startContainer(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContext) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Start an allocated container.

      The ApplicationMaster or other applications that use the client must provide the details of the allocated container, including the Id, the assigned node's Id and the token via Container. In addition, the AM needs to provide the ContainerLaunchContext as well.

      Specified by:
      startContainer in class NMClient
      Parameters:
      container - the allocated container
      containerLaunchContext - the context information needed by the NodeManager to launch the container
      Returns:
      a map between the auxiliary service names and their outputs
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • increaseContainerResource

      @Deprecated public void increaseContainerResource(org.apache.hadoop.yarn.api.records.Container container) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Deprecated.
      Description copied from class: NMClient

      Increase the resource of a container.

      The ApplicationMaster or other applications that use the client must provide the details of the container, including the Id and the target resource encapsulated in the updated container token via Container.

      Specified by:
      increaseContainerResource in class NMClient
      Parameters:
      container - the container with updated token.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • updateContainerResource

      public void updateContainerResource(org.apache.hadoop.yarn.api.records.Container container) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Update the resources of a container.

      The ApplicationMaster or other applications that use the client must provide the details of the container, including the Id and the target resource encapsulated in the updated container token via Container.

      Specified by:
      updateContainerResource in class NMClient
      Parameters:
      container - the container with updated token.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • stopContainer

      public void stopContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Stop an started container.

      Specified by:
      stopContainer in class NMClient
      Parameters:
      containerId - the Id of the started container
      nodeId - the Id of the NodeManager
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • getContainerStatus

      public org.apache.hadoop.yarn.api.records.ContainerStatus getContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Query the status of a container.

      Specified by:
      getContainerStatus in class NMClient
      Parameters:
      containerId - the Id of the started container
      nodeId - the Id of the NodeManager
      Returns:
      the status of a container.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • reInitializeContainer

      public void reInitializeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContex, boolean autoCommit) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Re-Initialize the Container.

      Specified by:
      reInitializeContainer in class NMClient
      Parameters:
      containerId - the Id of the container to Re-Initialize.
      containerLaunchContex - the updated ContainerLaunchContext.
      autoCommit - commit re-initialization automatically ?
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • restartContainer

      public void restartContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Restart the specified container.

      Specified by:
      restartContainer in class NMClient
      Parameters:
      containerId - the Id of the container to restart.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • rollbackLastReInitialization

      public void rollbackLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Rollback last reInitialization of the specified container.

      Specified by:
      rollbackLastReInitialization in class NMClient
      Parameters:
      containerId - the Id of the container to restart.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • commitLastReInitialization

      public void commitLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient

      Commit last reInitialization of the specified container.

      Specified by:
      commitLastReInitialization in class NMClient
      Parameters:
      containerId - the Id of the container to commit reInitialize.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.

    • getCleanupRunningContainers

      public AtomicBoolean getCleanupRunningContainers()

    • getNodeIdOfStartedContainer

      public org.apache.hadoop.yarn.api.records.NodeId getNodeIdOfStartedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
      Description copied from class: NMClient
      Get the NodeId of the node on which container is running. It returns null if the container if container is not found or if it is not running.
      Overrides:
      getNodeIdOfStartedContainer in class NMClient
      Parameters:
      containerId - Container Id of the container.
      Returns:
      NodeId of the container on which it is running.

    • localize

      public void localize(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient
      Localize resources for a container.
      Overrides:
      localize in class NMClient
      Parameters:
      containerId - the ID of the container
      nodeId - node Id of the container
      localResources - resources to localize
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException
      IOException

    • getLocalizationStatuses

      public List<org.apache.hadoop.yarn.api.records.LocalizationStatus> getLocalizationStatuses(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
      Description copied from class: NMClient
      Get the localization statuses of a container.
      Overrides:
      getLocalizationStatuses in class NMClient
      Parameters:
      containerId - the Id of the container
      nodeId - node Id of the container
      Returns:
      the status of a container.
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - YarnException.
      IOException - IOException.