Resource types support in YARN helps to extend the YARN resource model to a more flexible model which makes it easier to add new countable resourcetypes. This solution also helps the users to submit jobs with ease to specify the resources they need.
Resource Manager will load a new configuration file named resource-types.xml to determine the set of resource types for which scheduling is enabled. Sample XML will look like below.
<configuration>
<property>
<name>yarn.resource-types</name>
<value>resource1, resource2</value>
</property>
<property>
<name>yarn.resource-types.resource1.units</name>
<value>G</value>
</property>
</configuration>
Similarly, a new configuration file node-resources.xml will also be loaded by Node Manager where the resource capabilities of a node can be specified.
<configuration> <property> <name>yarn.nodemanager.resource-type.resource1</name> <value>5G</value> </property> <property> <name>yarn.nodemanager.resource-type.resource2</name> <value>2m</value> </property> </configuration>
Node Manager will use these custom resource types and will register it’s capability to Resource Manager.
Please note that, resource-types.xml and node-resources.xml file also need to be placed in conf directory if new resources are to be added to YARN.
In resource-types.xml
| Configuration Property | Value | Description |
|---|---|---|
| yarn.resource-types | resource1 | Custom resource |
| yarn.resource-types.resource1.units | G | Default unit for resource1 type |
In node-resources.xml
| Configuration Property | Value | Description |
|---|---|---|
| yarn.nodemanager.resource-type.resource1 | 5G | Resource capability for resource named ‘resource1’. |