org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies

Class DominantResourceFairnessPolicy

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.DominantResourceFairnessPolicy

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class DominantResourceFairnessPolicy
extends SchedulingPolicy

Makes scheduling decisions by trying to equalize dominant resource usage. A schedulable's dominant resource usage is the largest ratio of resource usage to capacity among the resource types it is using.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DominantResourceFairnessPolicy.DominantResourceFairnessComparator

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME

Fields inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy

DEFAULT_POLICY, DEPTH_ANY, DEPTH_INTERMEDIATE, DEPTH_LEAF, DEPTH_PARENT, DEPTH_ROOT

Constructor Summary

Constructors

Constructor and Description
DominantResourceFairnessPolicy()

Method Summary

Methods

Modifier and Type	Method and Description
boolean	checkIfAMResourceUsageOverLimit(org.apache.hadoop.yarn.api.records.Resource usage, org.apache.hadoop.yarn.api.records.Resource maxAMResource) Check if a leaf queue's AM resource usage over its limit under this policy
boolean	checkIfUsageOverFairShare(org.apache.hadoop.yarn.api.records.Resource usage, org.apache.hadoop.yarn.api.records.Resource fairShare) Check if the resource usage is over the fair share under this policy
void	computeShares(Collection<? extends Schedulable> schedulables, org.apache.hadoop.yarn.api.records.Resource totalResources) Computes and updates the shares of Schedulables as per the SchedulingPolicy, to be used later for scheduling decisions.
void	computeSteadyShares(Collection<? extends FSQueue> queues, org.apache.hadoop.yarn.api.records.Resource totalResources) Computes and updates the steady shares of FSQueues as per the SchedulingPolicy.
byte	getApplicableDepth() Specifies the depths in the hierarchy, this SchedulingPolicy applies to
Comparator<Schedulable>	getComparator() The comparator returned by this method is to be used for sorting the Schedulables in that queue.
org.apache.hadoop.yarn.api.records.Resource	getHeadroom(org.apache.hadoop.yarn.api.records.Resource queueFairShare, org.apache.hadoop.yarn.api.records.Resource queueUsage, org.apache.hadoop.yarn.api.records.Resource maxAvailable) Get headroom by calculating the min of clusterAvailable and (queueFairShare - queueUsage) resources that are applicable to this policy.
String	getName()
org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator() The ResourceCalculator returned by this method should be used for any calculations involving resources.
void	initialize(org.apache.hadoop.yarn.api.records.Resource clusterCapacity)

Methods inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy

getInstance, isApplicableTo, parse

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME

public static final String NAME

See Also:

Constant Field Values

Constructor Detail

DominantResourceFairnessPolicy

public DominantResourceFairnessPolicy()

Method Detail

getName

public String getName()

Specified by:

getName in class SchedulingPolicy

Returns:

returns the name of SchedulingPolicy

getApplicableDepth

public byte getApplicableDepth()

Description copied from class: SchedulingPolicy

Specifies the depths in the hierarchy, this SchedulingPolicy applies to

Specified by:

getApplicableDepth in class SchedulingPolicy

Returns:

depth equal to one of fields SchedulingPolicy#DEPTH_*

getComparator

public Comparator<Schedulable> getComparator()

Description copied from class: SchedulingPolicy

The comparator returned by this method is to be used for sorting the Schedulables in that queue.

Specified by:

getComparator in class SchedulingPolicy

Returns:

the comparator to sort by

getResourceCalculator

public org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

Description copied from class: SchedulingPolicy

The ResourceCalculator returned by this method should be used for any calculations involving resources.

Specified by:

getResourceCalculator in class SchedulingPolicy

Returns:

ResourceCalculator instance to use

computeShares

public void computeShares(Collection<? extends Schedulable> schedulables,
                 org.apache.hadoop.yarn.api.records.Resource totalResources)

Description copied from class: SchedulingPolicy

Computes and updates the shares of Schedulables as per the SchedulingPolicy, to be used later for scheduling decisions. The shares computed are instantaneous and only consider queues with running applications.

Specified by:

computeShares in class SchedulingPolicy

Parameters:

schedulables - Schedulables whose shares are to be updated

totalResources - Total Resources in the cluster

computeSteadyShares

public void computeSteadyShares(Collection<? extends FSQueue> queues,
                       org.apache.hadoop.yarn.api.records.Resource totalResources)

Description copied from class: SchedulingPolicy

Computes and updates the steady shares of FSQueues as per the SchedulingPolicy. The steady share does not differentiate between queues with and without running applications under them. The steady share is not used for scheduling, it is displayed on the Web UI for better visibility.

Specified by:

computeSteadyShares in class SchedulingPolicy

Parameters:

queues - FSQueues whose shares are to be updated

totalResources - Total Resources in the cluster

checkIfUsageOverFairShare

public boolean checkIfUsageOverFairShare(org.apache.hadoop.yarn.api.records.Resource usage,
                                org.apache.hadoop.yarn.api.records.Resource fairShare)

Description copied from class: SchedulingPolicy

Check if the resource usage is over the fair share under this policy

Specified by:

checkIfUsageOverFairShare in class SchedulingPolicy

Parameters:

usage - Resource the resource usage

fairShare - Resource the fair share

Returns:

true if check passes (is over) or false otherwise

checkIfAMResourceUsageOverLimit

public boolean checkIfAMResourceUsageOverLimit(org.apache.hadoop.yarn.api.records.Resource usage,
                                      org.apache.hadoop.yarn.api.records.Resource maxAMResource)

Description copied from class: SchedulingPolicy

Check if a leaf queue's AM resource usage over its limit under this policy

Specified by:

checkIfAMResourceUsageOverLimit in class SchedulingPolicy

Parameters:

usage - Resource the resource used by application masters

maxAMResource - Resource the maximum allowed resource for application masters

Returns:

true if AM resource usage is over the limit

getHeadroom

public org.apache.hadoop.yarn.api.records.Resource getHeadroom(org.apache.hadoop.yarn.api.records.Resource queueFairShare,
                                                      org.apache.hadoop.yarn.api.records.Resource queueUsage,
                                                      org.apache.hadoop.yarn.api.records.Resource maxAvailable)

Description copied from class: SchedulingPolicy

Get headroom by calculating the min of clusterAvailable and (queueFairShare - queueUsage) resources that are applicable to this policy. For eg if only memory then leave other resources such as CPU to same as clusterAvailable.

Specified by:

getHeadroom in class SchedulingPolicy

Parameters:

queueFairShare - fairshare in the queue

queueUsage - resources used in the queue

maxAvailable - available resource in cluster for this queue

Returns:

calculated headroom

initialize

public void initialize(org.apache.hadoop.yarn.api.records.Resource clusterCapacity)

Overrides:

initialize in class SchedulingPolicy