YARN is scalable very easily: any new NodeManager could join to the configured ResourceManager and start to execute jobs. But to achieve full elasticity we need a decommissioning process which helps to remove existing nodes and down-scale the cluster.
YARN Nodes could be decommissioned NORMAL or GRACEFUL.
Normal Decommission of YARN Nodes means an immediate shutdown.
Graceful Decommission of YARN Nodes is the mechanism to decommission NMs while minimize the impact to running applications. Once a node is in DECOMMISSIONING state, RM won’t schedule new containers on it and will wait for running containers and applications to complete (or until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED.
To do a normal decommissioning:
In the next sections we will cover some more detailed usage (for example: using graceful decommissioning with timeout).
yarn rmadmin -refreshNodes [-g [timeout in seconds] -client|server] notifies NodesListManager to detect and handle include and exclude hosts changes. NodesListManager loads excluded hosts from the exclude file as specified through the yarn.resourcemanager.nodes.exclude-path configuration in yarn-site.xml. (Note: It is unnecessary to restart RM in case of changing the exclude-path as this config will be read again for every refreshNodes command)
The format of the file could be plain text or XML depending the extension of the file. Only the XML format supports per node timout for graceful decommissioning.
NodesListManager inspects and compares status of RMNodes in resource manager and the exclude list, and apply necessary actions based on following rules:
Accordingly, RECOMMISSION, GRACEFUL_DECOMMISSION or DECOMMISSION RMNodeEvent will be sent to the RMNode.
To support flexible graceful decommission of nodes using different timeout through single or multiple refreshNodes requests, HostsFileReader supports optional timeout value after each hostname (or ip) in the exclude host file.
The effective decommissioning timeout to use for a particular host is based on following priorities:
In case of server side timeout:
In case of client side timout (see bellow):
NodesListManager decides the effective timeout to use and set it on individual RMNode. The timeout could also be dynamically adjusted through yarn rmadmin -refreshNodes -g [timeout in seconds] command. NodesListManager will resolve the effective timeout to use and update RMNode as necessary of the new timeout. Change of timeout does not reset the ongoing decommissioning but only affect the evaluation of whether the node has reached decommissioning timeout.
Here is a sample excludes file in xml format.
<?xml version="1.0"?> <hosts> <host><name>host1</name></host> <host><name>host2</name><timeout>123</timeout></host> <host><name>host3</name><timeout>-1</timeout></host> <host><name>host4, host5,host6</name><timeout>1800</timeout></host> </hosts>
If the file extension of the exclude file is not xml, standard one-line-per-host format is used without timeout support.
host1 host2 host3
Note: In the future more file formats are planned with timeout support. Follow the YARN-5536 if you are interested.
Important to mention, that the timeout is not persited. In case of a RM restart/failover the node will be immediatelly decommission. (Follow the YARN-5464 for changes in this behavior).
Timeout of Graceful decommissioning could be tracked on server or client side. The -client|server indicates if the timeout tracking should be handled by the client or the ResourceManager. The client side tracking is blocking, while the server-side tracking is not.
Upon receiving GRACEFUL_DECOMMISSION event, the RMNode will save the decommissioning timeout if specified, update metrics for graceful decommission and preserve its original total capacity, and transition into DECOMMISSIONING state.
Resources will be dynamically and periodically updated on DECOMMISSIONING RMNode so that scheduler won’t be scheduling new containers on them due to no available resources.
DecommissioningNodeWatcher is the YARN component that tracks DECOMMISSIONING nodes status automatically and asynchronously after client/admin made the graceful decommission request. NM periodically send RM heart beat with it latest container status. DecommissioningNodeWatcher tracks heartbeat updates on all DECOMMISSIONING nodes to decide when, after all running containers on the node have completed, will be transitioned into DECOMMISSIONED state after which NodeManager will be told to shutdown.
Under MR application, a node, after completes all its containers, may still serve it map output data during the duration of the application for reducers. The YARN graceful decommission mechanism keeps such DECOMMISSIONING nodes until all involved applications complete. It could be however undesirable under long-running applications scenario where a bunch of “idle” nodes might stay around for long period of time. DecommissioningNodeWatcher balances such concern with a timeout — a DECOMMISSIONING node will be DECOMMISSIONED no later than decommissioning timeout regardless of running containers or applications. If running containers finished earlier, it continues waiting for applications to finish until the decommissioning timeout. When decommissioning timeout reaches, the node will be decommissioned regardless. The node will be deactivated and owning tasks will be rescheduled as necessary.
Status of all decommissioning node are logged periodically (every 20 seconds) in resource manager logs. Following are the sub-status of a decommissioning node:
| Property | Value |
|---|---|
| yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs | Timeout in seconds for YARN node graceful decommission. This is the maximal time to wait for running containers and applications to complete before transition a DECOMMISSIONING node into DECOMMISSIONED. The default value is 3600 seconds. Negative value (like -1) is handled as infinite timeout. |
| yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs | Period in seconds of the poll timer task inside DecommissioningNodesWatcher to identify and take care of DECOMMISSIONING nodes missing regular heart beat. The default value is 20 seconds. |
| yarn.resourcemanager.nodes.exclude-path | Path to file with nodes to exclude. |
| yarn.resourcemanager.nodes.include-path | Path to file with nodes to include. |