org.apache.hadoop.yarn.client.api.impl

Class AMRMClientImpl<T extends AMRMClient.ContainerRequest>

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.client.api.AMRMClient<T>

org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl<T>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class AMRMClientImpl<T extends AMRMClient.ContainerRequest>
extends AMRMClient<T>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.client.api.AMRMClient

AMRMClient.ContainerRequest

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected String	appHostName
protected int	appHostPort
protected String	appTrackingUrl
protected Set<org.apache.hadoop.yarn.api.records.ResourceRequest>	ask
protected Set<String>	blacklistAdditions
protected Set<String>	blacklistedNodes
protected Set<String>	blacklistRemovals
protected Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>>	change
protected org.apache.hadoop.yarn.api.records.Resource	clusterAvailableResources
protected int	clusterNodeCount
protected Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>>	pendingChange
protected Set<org.apache.hadoop.yarn.api.records.ContainerId>	pendingRelease
protected Set<org.apache.hadoop.yarn.api.records.ContainerId>	release
protected Map<String,org.apache.hadoop.yarn.api.records.Resource>	resourceProfilesMap
protected org.apache.hadoop.yarn.api.ApplicationMasterProtocol	rmClient

Constructor Summary

Constructors

Constructor and Description
AMRMClientImpl()

Method Summary

Modifier and Type	Method and Description
void	addContainerRequest(T req) Request containers for resources before calling allocate
void	addSchedulingRequests(Collection<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests) Add a Collection of SchedulingRequests.
org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse	allocate(float progressIndicator) Request additional containers and receive new container allocations.
org.apache.hadoop.yarn.api.records.Resource	getAvailableResources() Get the currently available resources in the cluster.
int	getClusterNodeCount() Get the current number of nodes in the cluster.
Collection<T>	getMatchingRequests(long allocationRequestId) Get outstanding ContainerRequests matching the given allocationRequestId.
List<? extends Collection<T>>	getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.ExecutionType executionType, org.apache.hadoop.yarn.api.records.Resource capability) Get outstanding ContainerRequests matching the given parameters.
List<? extends Collection<T>>	getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.ExecutionType executionType, org.apache.hadoop.yarn.api.records.Resource capability, String profile)
List<? extends Collection<T>>	getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.Resource capability) Get outstanding ContainerRequests matching the given parameters.
protected void	populateNMTokens(List<org.apache.hadoop.yarn.api.records.NMToken> nmTokens)
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl) Register the application master.
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl, Map<Set<String>,org.apache.hadoop.yarn.api.resource.PlacementConstraint> placementConstraintsMap) Register the application master.
void	releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Release containers assigned by the Resource Manager.
void	removeContainerRequest(T req) Remove previous container request.
protected void	removePendingChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdatedContainer> changedContainers)
protected void	removePendingReleaseRequests(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainersStatuses)
void	requestContainerUpdate(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.UpdateContainerRequest updateContainerRequest) Request a container update before calling allocate.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
protected void	serviceStop()
void	unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl) Unregister the application master.
void	updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals) Update application's blacklist with addition or removal resources.

Methods inherited from class org.apache.hadoop.yarn.client.api.AMRMClient

createAMRMClient, getNMTokenCache, getRegisteredTimelineV2Client, registerTimelineV2Client, requestContainerResourceChange, setNMTokenCache, waitFor, waitFor, waitFor

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

appHostName

protected String appHostName

appHostPort

protected int appHostPort

appTrackingUrl

protected String appTrackingUrl

rmClient

protected org.apache.hadoop.yarn.api.ApplicationMasterProtocol rmClient

clusterAvailableResources

protected org.apache.hadoop.yarn.api.records.Resource clusterAvailableResources

clusterNodeCount

protected int clusterNodeCount

blacklistedNodes

protected final Set<String> blacklistedNodes

blacklistAdditions

protected final Set<String> blacklistAdditions

blacklistRemovals

protected final Set<String> blacklistRemovals

resourceProfilesMap

protected Map<String,org.apache.hadoop.yarn.api.records.Resource> resourceProfilesMap

ask

protected final Set<org.apache.hadoop.yarn.api.records.ResourceRequest> ask

release

protected final Set<org.apache.hadoop.yarn.api.records.ContainerId> release

pendingRelease

protected Set<org.apache.hadoop.yarn.api.records.ContainerId> pendingRelease

change

protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> change

pendingChange

protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> pendingChange

Constructor Detail

AMRMClientImpl

public AMRMClientImpl()

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

registerApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName,
                                                                                                              int appHostPort,
                                                                                                              String appTrackingUrl)
                                                                                                       throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                              IOException

Description copied from class: AMRMClient

Register the application master. This must be called before any other interaction

Specified by:

registerApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

appHostName - Name of the host on which master is running

appHostPort - Port master is listening on

appTrackingUrl - URL at which the master info can be seen

Returns:

RegisterApplicationMasterResponse

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

registerApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName,
                                                                                                              int appHostPort,
                                                                                                              String appTrackingUrl,
                                                                                                              Map<Set<String>,org.apache.hadoop.yarn.api.resource.PlacementConstraint> placementConstraintsMap)
                                                                                                       throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                              IOException

Description copied from class: AMRMClient

Register the application master. This must be called before any other interaction

Overrides:

registerApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

appHostName - Name of the host on which master is running

appHostPort - Port master is listening on

appTrackingUrl - URL at which the master info can be seen

placementConstraintsMap - Placement Constraints mappings.

Returns:

RegisterApplicationMasterResponse

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

addSchedulingRequests

public void addSchedulingRequests(Collection<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests)

Description copied from class: AMRMClient

Add a Collection of SchedulingRequests. The AMRMClient will ensure that all requests in the same batch are sent in the same allocate call.

Overrides:

addSchedulingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

schedulingRequests - Collection of Scheduling Requests.

allocate

public org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse allocate(float progressIndicator)
                                                                     throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                            IOException

Description copied from class: AMRMClient

Request additional containers and receive new container allocations. Requests made via addContainerRequest are sent to the ResourceManager. New containers assigned to the master are retrieved. Status of completed containers and node health updates are also retrieved. This also doubles up as a heartbeat to the ResourceManager and must be made periodically. The call may not always return any new allocations of containers. App should not make concurrent allocate requests. May cause request loss.

Note : If the user has not removed container requests that have already been satisfied, then the re-register may end up sending the entire container requests to the RM (including matched requests). Which would mean the RM could end up giving it a lot of new allocated containers.

Specified by:

allocate in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

progressIndicator - Indicates progress made by the master

Returns:

the response of the allocate request

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

removePendingReleaseRequests

protected void removePendingReleaseRequests(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainersStatuses)

removePendingChangeRequests

protected void removePendingChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdatedContainer> changedContainers)

populateNMTokens

@InterfaceAudience.Private
protected void populateNMTokens(List<org.apache.hadoop.yarn.api.records.NMToken> nmTokens)

unregisterApplicationMaster

public void unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus,
                                        String appMessage,
                                        String appTrackingUrl)
                                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                                        IOException

Description copied from class: AMRMClient

Unregister the application master. This must be called in the end.

Specified by:

unregisterApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

appStatus - Success/Failure status of the master

appMessage - Diagnostics message on failure

appTrackingUrl - New URL to get master info

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

addContainerRequest

public void addContainerRequest(T req)

Description copied from class: AMRMClient

Request containers for resources before calling allocate

Specified by:

addContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

req - Resource request

removeContainerRequest

public void removeContainerRequest(T req)

Description copied from class: AMRMClient

Remove previous container request. The previous container request may have already been sent to the ResourceManager. So even after the remove request the app must be prepared to receive an allocation for the previous request even after the remove request

Specified by:

removeContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

req - Resource request

requestContainerUpdate

public void requestContainerUpdate(org.apache.hadoop.yarn.api.records.Container container,
                                   org.apache.hadoop.yarn.api.records.UpdateContainerRequest updateContainerRequest)

Description copied from class: AMRMClient

Request a container update before calling allocate. Any previous pending update request of the same container will be removed.

Specified by:

requestContainerUpdate in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

container - The container returned from the last successful resource allocation or update

updateContainerRequest - The UpdateContainerRequest.

releaseAssignedContainer

public void releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from class: AMRMClient

Release containers assigned by the Resource Manager. If the app cannot use the container or wants to give up the container then it can release them. The app needs to make new requests for the released resource capability if it still needs it. eg. it released non-local resources

Specified by:

releaseAssignedContainer in class AMRMClient<T extends AMRMClient.ContainerRequest>

getAvailableResources

public org.apache.hadoop.yarn.api.records.Resource getAvailableResources()

Description copied from class: AMRMClient

Get the currently available resources in the cluster. A valid value is available after a call to allocate has been made

Specified by:

getAvailableResources in class AMRMClient<T extends AMRMClient.ContainerRequest>

Returns:

Currently available resources

getClusterNodeCount

public int getClusterNodeCount()

Description copied from class: AMRMClient

Get the current number of nodes in the cluster. A valid values is available after a call to allocate has been made

Specified by:

getClusterNodeCount in class AMRMClient<T extends AMRMClient.ContainerRequest>

Returns:

Current number of nodes in the cluster

getMatchingRequests

public Collection<T> getMatchingRequests(long allocationRequestId)

Description copied from class: AMRMClient

Get outstanding ContainerRequests matching the given allocationRequestId. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. NOTE: This API only matches Container requests that were created by the client WITH the allocationRequestId being set to a non-default value.

Specified by:

getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

allocationRequestId - Allocation Request Id

Returns:

Collection of request matching the parameters

getMatchingRequests

public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
                                                         String resourceName,
                                                         org.apache.hadoop.yarn.api.records.Resource capability)

Description copied from class: AMRMClient

Get outstanding ContainerRequests matching the given parameters. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. Each collection in the list contains requests with identical Resource size that fit in the given capability. In a collection, requests will be returned in the same order as they were added. NOTE: This API only matches Container requests that were created by the client WITHOUT the allocationRequestId being set.

Specified by:

getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>

Returns:

Collection of request matching the parameters

getMatchingRequests

public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
                                                         String resourceName,
                                                         org.apache.hadoop.yarn.api.records.ExecutionType executionType,
                                                         org.apache.hadoop.yarn.api.records.Resource capability,
                                                         String profile)

Overrides:

getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>

getMatchingRequests

public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
                                                         String resourceName,
                                                         org.apache.hadoop.yarn.api.records.ExecutionType executionType,
                                                         org.apache.hadoop.yarn.api.records.Resource capability)

Description copied from class: AMRMClient

Get outstanding ContainerRequests matching the given parameters. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. Each collection in the list contains requests with identical Resource size that fit in the given capability. In a collection, requests will be returned in the same order as they were added. specify an ExecutionType. NOTE: This API only matches Container requests that were created by the client WITHOUT the allocationRequestId being set.

Overrides:

getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

priority - Priority

resourceName - Location

executionType - ExecutionType

capability - Capability

Returns:

Collection of request matching the parameters

updateBlacklist

public void updateBlacklist(List<String> blacklistAdditions,
                            List<String> blacklistRemovals)

Description copied from class: AMRMClient

Update application's blacklist with addition or removal resources.

Specified by:

updateBlacklist in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

blacklistAdditions - list of resources which should be added to the application blacklist

blacklistRemovals - list of resources which should be removed from the application blacklist