org.apache.hadoop.yarn.client.api.async

Class NMClientAsync

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.client.api.async.NMClientAsync

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

Direct Known Subclasses:

NMClientAsyncImpl

@InterfaceAudience.Public
 @InterfaceStability.Stable
public abstract class NMClientAsync
extends org.apache.hadoop.service.AbstractService

NMClientAsync handles communication with all the NodeManagers and provides asynchronous updates on getting responses from them. It maintains a thread pool to communicate with individual NMs where a number of worker threads process requests to NMs by using NMClientImpl. The max size of the thread pool is configurable through YarnConfiguration.NM_CLIENT_ASYNC_THREAD_POOL_MAX_SIZE. It should be used in conjunction with a CallbackHandler. For example

 
 class MyCallbackHandler extends NMClientAsync.AbstractCallbackHandler {
   public void onContainerStarted(ContainerId containerId,
       Map<String, ByteBuffer> allServiceResponse) {
     [post process after the container is started, process the response]
   }

   public void onContainerResourceIncreased(ContainerId containerId,
       Resource resource) {
     [post process after the container resource is increased]
   }

   public void onContainerStatusReceived(ContainerId containerId,
       ContainerStatus containerStatus) {
     [make use of the status of the container]
   }

   public void onContainerStopped(ContainerId containerId) {
     [post process after the container is stopped]
   }

   public void onStartContainerError(
       ContainerId containerId, Throwable t) {
     [handle the raised exception]
   }

   public void onGetContainerStatusError(
       ContainerId containerId, Throwable t) {
     [handle the raised exception]
   }

   public void onStopContainerError(
       ContainerId containerId, Throwable t) {
     [handle the raised exception]
   }
 }
 
 

The client's life-cycle should be managed like the following:

 
 NMClientAsync asyncClient = 
     NMClientAsync.createNMClientAsync(new MyCallbackhandler());
 asyncClient.init(conf);
 asyncClient.start();
 asyncClient.startContainer(container, containerLaunchContext);
 [... wait for container being started]
 asyncClient.getContainerStatus(container.getId(), container.getNodeId(),
     container.getContainerToken());
 [... handle the status in the callback instance]
 asyncClient.stopContainer(container.getId(), container.getNodeId(),
     container.getContainerToken());
 [... wait for container being stopped]
 asyncClient.stop();
 
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	NMClientAsync.AbstractCallbackHandler The callback abstract class.
static interface	NMClientAsync.CallbackHandler Deprecated. Use NMClientAsync.AbstractCallbackHandler instead. The callback interface needs to be implemented by NMClientAsync users. The APIs are called when responses from NodeManager are available. Once a callback happens, the users can chose to act on it in blocking or non-blocking manner. If the action on callback is done in a blocking manner, some of the threads performing requests on NodeManagers may get blocked depending on how many threads in the pool are busy. The implementation of the callback function should not throw the unexpected exception. Otherwise, NMClientAsync will just catch, log and then ignore it.

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected NMClientAsync.CallbackHandler	callbackHandler
protected NMClient	client

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	NMClientAsync(NMClientAsync.AbstractCallbackHandler callbackHandler)
protected	NMClientAsync(NMClientAsync.CallbackHandler callbackHandler) Deprecated. Use NMClientAsync(AbstractCallbackHandler) instead.
protected	NMClientAsync(String name, NMClientAsync.AbstractCallbackHandler callbackHandler)
protected	NMClientAsync(String name, NMClientAsync.CallbackHandler callbackHandler) Deprecated. Use NMClientAsync(String, AbstractCallbackHandler) instead.
protected	NMClientAsync(String name, NMClient client, NMClientAsync.AbstractCallbackHandler callbackHandler)
protected	NMClientAsync(String name, NMClient client, NMClientAsync.CallbackHandler callbackHandler) Deprecated.

Method Summary

Modifier and Type	Method and Description
abstract void	commitLastReInitializationAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId) Commit last reInitialization of the specified container.
static NMClientAsync	createNMClientAsync(NMClientAsync.AbstractCallbackHandler callbackHandler)
static NMClientAsync	createNMClientAsync(NMClientAsync.CallbackHandler callbackHandler) Deprecated. Use createNMClientAsync(AbstractCallbackHandler) instead.
NMClientAsync.CallbackHandler	getCallbackHandler()
NMClient	getClient()
abstract void	getContainerStatusAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId)
abstract void	increaseContainerResourceAsync(org.apache.hadoop.yarn.api.records.Container container) Deprecated.
abstract void	reInitializeContainerAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContex, boolean autoCommit) Re-Initialize the Container.
abstract void	restartContainerAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId) Restart the specified container.
abstract void	rollbackLastReInitializationAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId) Rollback last reInitialization of the specified container.
void	setCallbackHandler(NMClientAsync.CallbackHandler callbackHandler)
void	setClient(NMClient client)
abstract void	startContainerAsync(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContext)
abstract void	stopContainerAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.NodeId nodeId)
abstract void	updateContainerResourceAsync(org.apache.hadoop.yarn.api.records.Container container) Update the resources of a container.

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, serviceStart, serviceStop, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

client

protected NMClient client

callbackHandler

protected NMClientAsync.CallbackHandler callbackHandler

Constructor Detail

NMClientAsync

protected NMClientAsync(NMClientAsync.AbstractCallbackHandler callbackHandler)

NMClientAsync

protected NMClientAsync(String name,
                        NMClientAsync.AbstractCallbackHandler callbackHandler)

NMClientAsync

protected NMClientAsync(String name,
                        NMClient client,
                        NMClientAsync.AbstractCallbackHandler callbackHandler)

NMClientAsync

@Deprecated
protected NMClientAsync(NMClientAsync.CallbackHandler callbackHandler)

Deprecated. Use NMClientAsync(AbstractCallbackHandler) instead.

NMClientAsync

@Deprecated
protected NMClientAsync(String name,
                                    NMClientAsync.CallbackHandler callbackHandler)

Deprecated. Use NMClientAsync(String, AbstractCallbackHandler) instead.

NMClientAsync

@InterfaceAudience.Private
 @Deprecated
protected NMClientAsync(String name,
                                                                NMClient client,
                                                                NMClientAsync.CallbackHandler callbackHandler)

Deprecated.

Method Detail

createNMClientAsync

public static NMClientAsync createNMClientAsync(NMClientAsync.AbstractCallbackHandler callbackHandler)

createNMClientAsync

@Deprecated
public static NMClientAsync createNMClientAsync(NMClientAsync.CallbackHandler callbackHandler)

Deprecated. Use createNMClientAsync(AbstractCallbackHandler) instead.

startContainerAsync

public abstract void startContainerAsync(org.apache.hadoop.yarn.api.records.Container container,
                                         org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContext)

increaseContainerResourceAsync

@Deprecated
public abstract void increaseContainerResourceAsync(org.apache.hadoop.yarn.api.records.Container container)

Deprecated.

updateContainerResourceAsync

public abstract void updateContainerResourceAsync(org.apache.hadoop.yarn.api.records.Container container)

Update the resources of a container.

The ApplicationMaster or other applications that use the client must provide the details of the container, including the Id and the target resource encapsulated in the updated container token via Container.

Parameters:

container - the container with updated token.

reInitializeContainerAsync

public abstract void reInitializeContainerAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                                org.apache.hadoop.yarn.api.records.ContainerLaunchContext containerLaunchContex,
                                                boolean autoCommit)

Re-Initialize the Container.

Parameters:

containerId - the Id of the container to Re-Initialize.

containerLaunchContex - the updated ContainerLaunchContext.

autoCommit - commit re-initialization automatically ?

restartContainerAsync

public abstract void restartContainerAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Restart the specified container.

Parameters:

containerId - the Id of the container to restart.

rollbackLastReInitializationAsync

public abstract void rollbackLastReInitializationAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Rollback last reInitialization of the specified container.

Parameters:

containerId - the Id of the container to restart.

commitLastReInitializationAsync

public abstract void commitLastReInitializationAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Commit last reInitialization of the specified container.

Parameters:

containerId - the Id of the container to commit reInitialize.

stopContainerAsync

public abstract void stopContainerAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                        org.apache.hadoop.yarn.api.records.NodeId nodeId)

getContainerStatusAsync

public abstract void getContainerStatusAsync(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                             org.apache.hadoop.yarn.api.records.NodeId nodeId)

getClient

public NMClient getClient()

setClient

public void setClient(NMClient client)

getCallbackHandler

public NMClientAsync.CallbackHandler getCallbackHandler()

setCallbackHandler

public void setCallbackHandler(NMClientAsync.CallbackHandler callbackHandler)