org.apache.hadoop.yarn.client.api.impl

Class NMClientImpl

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.client.api.NMClient

org.apache.hadoop.yarn.client.api.impl.NMClientImpl

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class NMClientImpl
extends NMClient

This class implements NMClient. All the APIs are blocking.

By default, this client stops all the running containers that are started by it when it stops. It can be disabled via cleanupRunningContainersOnStop(boolean), in which case containers will continue to run even after this client is stopped and till the application runs at which point ResourceManager will forcefully kill them.

Note that the blocking APIs ensure the RPC calls to NodeManager are executed immediately, and the responses are received before these APIs return. However, when startContainer(org.apache.hadoop.yarn.api.records.Container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext) or stopContainer(org.apache.hadoop.yarn.api.records.ContainerId, org.apache.hadoop.yarn.api.records.NodeId) returns, NodeManager may still need some time to either start or stop the container because of its asynchronous implementation. Therefore, getContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId, org.apache.hadoop.yarn.api.records.NodeId) is likely to return a transit container status if it is executed immediately after startContainer(org.apache.hadoop.yarn.api.records.Container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext) or stopContainer(org.apache.hadoop.yarn.api.records.ContainerId, org.apache.hadoop.yarn.api.records.NodeId).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	NMClientImpl.StartedContainer

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.client.api.NMClient

NMClient.UpgradeOp

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected ConcurrentMap<org.apache.hadoop.yarn.api.records.ContainerId,NMClientImpl.StartedContainer>	startedContainers

Constructor Summary

Constructors

Constructor and Description
NMClientImpl()
NMClientImpl(String name)

Method Summary

Modifier and Type	Method and Description
protected void	cleanupRunningContainers()
void	cleanupRunningContainersOnStop(boolean enabled) Set whether the containers that are started by this client, and are still running should be stopped when the client stops.
void	commitLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) Commit last reInitialization of the specified container.
AtomicBoolean	getCleanupRunningContainers()
org.apache.hadoop.yarn.api.records.ContainerStatus	getContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId) Query the status of a container.
org.apache.hadoop.yarn.api.records.NodeId	getNodeIdOfStartedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Get the NodeId of the node on which container is running.
void	increaseContainerResource(org.apache.hadoop.yarn.api.records.Container container) Deprecated.
void	reInitializeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContex, boolean autoCommit) Re-Initialize the Container.
void	restartContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Restart the specified container.
void	rollbackLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) Rollback last reInitialization of the specified container.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStop()
Map<String,ByteBuffer>	startContainer(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContext) Start an allocated container.
void	stopContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId) Stop an started container.
void	updateContainerResource(org.apache.hadoop.yarn.api.records.Container container) Update the resources of a container.

Methods inherited from class org.apache.hadoop.yarn.client.api.NMClient

createNMClient, createNMClient, getNMTokenCache, setNMTokenCache

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceStart, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

startedContainers

protected ConcurrentMap<org.apache.hadoop.yarn.api.records.ContainerId,NMClientImpl.StartedContainer> startedContainers

Constructor Detail

NMClientImpl

public NMClientImpl()

NMClientImpl

public NMClientImpl(String name)

Method Detail

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

cleanupRunningContainers

protected void cleanupRunningContainers()

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

cleanupRunningContainersOnStop

public void cleanupRunningContainersOnStop(boolean enabled)

Description copied from class: NMClient

Set whether the containers that are started by this client, and are still running should be stopped when the client stops. By default, the feature should be enabled.

However, containers will be stopped only when service is stopped. i.e. after AbstractService.stop().

Specified by:

cleanupRunningContainersOnStop in class NMClient

Parameters:

enabled - whether the feature is enabled or not

startContainer

public Map<String,ByteBuffer> startContainer(org.apache.hadoop.yarn.api.records.Container container,
                                             org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContext)
                                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                                             IOException

Description copied from class: NMClient

Start an allocated container.

The ApplicationMaster or other applications that use the client must provide the details of the allocated container, including the Id, the assigned node's Id and the token via Container. In addition, the AM needs to provide the ContainerLaunchContext as well.

Specified by:

startContainer in class NMClient

Parameters:

container - the allocated container

containerLaunchContext - the context information needed by the NodeManager to launch the container

Returns:

a map between the auxiliary service names and their outputs

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

increaseContainerResource

@Deprecated
public void increaseContainerResource(org.apache.hadoop.yarn.api.records.Container container)
                                           throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                  IOException

Deprecated.

Description copied from class: NMClient

Increase the resource of a container.

The ApplicationMaster or other applications that use the client must provide the details of the container, including the Id and the target resource encapsulated in the updated container token via Container.

Specified by:

increaseContainerResource in class NMClient

Parameters:

container - the container with updated token.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

updateContainerResource

public void updateContainerResource(org.apache.hadoop.yarn.api.records.Container container)
                             throws org.apache.hadoop.yarn.exceptions.YarnException,
                                    IOException

Description copied from class: NMClient

Update the resources of a container.

The ApplicationMaster or other applications that use the client must provide the details of the container, including the Id and the target resource encapsulated in the updated container token via Container.

Specified by:

updateContainerResource in class NMClient

Parameters:

container - the container with updated token.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

stopContainer

public void stopContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                          org.apache.hadoop.yarn.api.records.NodeId nodeId)
                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                          IOException

Description copied from class: NMClient

Stop an started container.

Specified by:

stopContainer in class NMClient

Parameters:

containerId - the Id of the started container

nodeId - the Id of the NodeManager

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

getContainerStatus

public org.apache.hadoop.yarn.api.records.ContainerStatus getContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                                                             org.apache.hadoop.yarn.api.records.NodeId nodeId)
                                                                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                             IOException

Description copied from class: NMClient

Query the status of a container.

Specified by:

getContainerStatus in class NMClient

Parameters:

containerId - the Id of the started container

nodeId - the Id of the NodeManager

Returns:

the status of a container.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

reInitializeContainer

public void reInitializeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                  org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContex,
                                  boolean autoCommit)
                           throws org.apache.hadoop.yarn.exceptions.YarnException,
                                  IOException

Description copied from class: NMClient

Re-Initialize the Container.

Specified by:

reInitializeContainer in class NMClient

Parameters:

containerId - the Id of the container to Re-Initialize.

containerLaunchContex - the updated ContainerLaunchContext.

autoCommit - commit re-initialization automatically ?

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

restartContainer

public void restartContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                             IOException

Description copied from class: NMClient

Restart the specified container.

Specified by:

restartContainer in class NMClient

Parameters:

containerId - the Id of the container to restart.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

rollbackLastReInitialization

public void rollbackLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                                  throws org.apache.hadoop.yarn.exceptions.YarnException,
                                         IOException

Description copied from class: NMClient

Rollback last reInitialization of the specified container.

Specified by:

rollbackLastReInitialization in class NMClient

Parameters:

containerId - the Id of the container to restart.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

commitLastReInitialization

public void commitLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                                throws org.apache.hadoop.yarn.exceptions.YarnException,
                                       IOException

Description copied from class: NMClient

Commit last reInitialization of the specified container.

Specified by:

commitLastReInitialization in class NMClient

Parameters:

containerId - the Id of the container to commit reInitialize.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YarnException.

IOException - IOException.

getCleanupRunningContainers

public AtomicBoolean getCleanupRunningContainers()

getNodeIdOfStartedContainer

public org.apache.hadoop.yarn.api.records.NodeId getNodeIdOfStartedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from class: NMClient

Get the NodeId of the node on which container is running. It returns null if the container if container is not found or if it is not running.

Overrides:

getNodeIdOfStartedContainer in class NMClient

Parameters:

containerId - Container Id of the container.

Returns:

NodeId of the container on which it is running.