org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair

Class SchedulingPolicy

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy

Direct Known Subclasses:

DominantResourceFairnessPolicy, FairSharePolicy, FifoPolicy

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public abstract class SchedulingPolicy
extends Object

The SchedulingPolicy is used by the fair scheduler mainly to determine what a queue's fair share and steady fair share should be as well as calculating available headroom. This determines how resources can be shared between running applications within a queue.

Every queue has a policy, including parents and children. If a child queue doesn't specify one, it inherits the parent's policy. The policy for a child queue must be compatible with the policy of the parent queue; there are some combinations that aren't allowed. See isChildPolicyAllowed(SchedulingPolicy). The policy for a queue is specified by setting property schedulingPolicy in the fair scheduler configuration file. The default policy is FairSharePolicy if not specified.

Field Summary

Fields

Modifier and Type	Field and Description
static SchedulingPolicy	DEFAULT_POLICY

Constructor Summary

Constructors

Constructor and Description
SchedulingPolicy()

Method Summary

Modifier and Type	Method and Description
abstract boolean	checkIfUsageOverFairShare(org.apache.hadoop.yarn.api.records.Resource usage, org.apache.hadoop.yarn.api.records.Resource fairShare) Check if the resource usage is over the fair share under this policy.
abstract void	computeShares(Collection<? extends Schedulable> schedulables, org.apache.hadoop.yarn.api.records.Resource totalResources) Computes and updates the shares of Schedulables as per the SchedulingPolicy, to be used later for scheduling decisions.
abstract void	computeSteadyShares(Collection<? extends FSQueue> queues, org.apache.hadoop.yarn.api.records.Resource totalResources) Computes and updates the steady shares of FSQueues as per the SchedulingPolicy.
abstract Comparator<Schedulable>	getComparator() The comparator returned by this method is to be used for sorting the Schedulables in that queue.
abstract org.apache.hadoop.yarn.api.records.Resource	getHeadroom(org.apache.hadoop.yarn.api.records.Resource queueFairShare, org.apache.hadoop.yarn.api.records.Resource queueUsage, org.apache.hadoop.yarn.api.records.Resource maxAvailable) Get headroom by calculating the min of clusterAvailable and (queueFairShare - queueUsage) resources that are applicable to this policy.
static SchedulingPolicy	getInstance(Class<? extends SchedulingPolicy> clazz) Returns a SchedulingPolicy instance corresponding to the passed clazz.
abstract String	getName()
abstract org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator() The ResourceCalculator returned by this method should be used for any calculations involving resources.
void	initialize(FSContext fsContext) Initialize the scheduling policy with a FSContext object, which has a pointer to the cluster resources among other information.
void	initialize(org.apache.hadoop.yarn.api.records.Resource clusterCapacity) Deprecated. Since it doesn't track cluster resource changes, replaced by initialize(FSContext).
boolean	isChildPolicyAllowed(SchedulingPolicy childPolicy) Check whether the policy of a child queue is allowed.
static SchedulingPolicy	parse(String policy) Returns SchedulingPolicy instance corresponding to the SchedulingPolicy passed as a string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_POLICY

public static final SchedulingPolicy DEFAULT_POLICY

Constructor Detail

SchedulingPolicy

public SchedulingPolicy()

Method Detail

getInstance

public static SchedulingPolicy getInstance(Class<? extends SchedulingPolicy> clazz)

Returns a SchedulingPolicy instance corresponding to the passed clazz.

Parameters:

clazz - a class that extends SchedulingPolicy

Returns:

a SchedulingPolicy instance

parse

public static SchedulingPolicy parse(String policy)
                              throws AllocationConfigurationException

Returns SchedulingPolicy instance corresponding to the SchedulingPolicy passed as a string. The policy can be "fair" for FairSharePolicy, "fifo" for FifoPolicy, or "drf" for DominantResourceFairnessPolicy. For a custom SchedulingPolicys in the RM classpath, the policy should be canonical class name of the SchedulingPolicy.

Parameters:

policy - canonical class name or "drf" or "fair" or "fifo"

Returns:

a SchedulingPolicy instance parsed from given policy

Throws:

AllocationConfigurationException

initialize

@Deprecated
public void initialize(org.apache.hadoop.yarn.api.records.Resource clusterCapacity)

Deprecated. Since it doesn't track cluster resource changes, replaced by initialize(FSContext).

Initialize the scheduling policy with cluster resources.

Parameters:

clusterCapacity - cluster resources

initialize

public void initialize(FSContext fsContext)

Initialize the scheduling policy with a FSContext object, which has a pointer to the cluster resources among other information.

Parameters:

fsContext - a FSContext object which has a pointer to the cluster resources

getResourceCalculator

public abstract org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

The ResourceCalculator returned by this method should be used for any calculations involving resources.

Returns:

ResourceCalculator instance to use

getName

public abstract String getName()

Returns:

returns the name of SchedulingPolicy

getComparator

public abstract Comparator<Schedulable> getComparator()

The comparator returned by this method is to be used for sorting the Schedulables in that queue.

Returns:

the comparator to sort by

computeShares

public abstract void computeShares(Collection<? extends Schedulable> schedulables,
                                   org.apache.hadoop.yarn.api.records.Resource totalResources)

Computes and updates the shares of Schedulables as per the SchedulingPolicy, to be used later for scheduling decisions. The shares computed are instantaneous and only consider queues with running applications.

Parameters:

schedulables - Schedulables whose shares are to be updated

totalResources - Total Resources in the cluster

computeSteadyShares

public abstract void computeSteadyShares(Collection<? extends FSQueue> queues,
                                         org.apache.hadoop.yarn.api.records.Resource totalResources)

Computes and updates the steady shares of FSQueues as per the SchedulingPolicy. The steady share does not differentiate between queues with and without running applications under them. The steady share is not used for scheduling, it is displayed on the Web UI for better visibility.

Parameters:

queues - FSQueues whose shares are to be updated

totalResources - Total Resources in the cluster

checkIfUsageOverFairShare

public abstract boolean checkIfUsageOverFairShare(org.apache.hadoop.yarn.api.records.Resource usage,
                                                  org.apache.hadoop.yarn.api.records.Resource fairShare)

Check if the resource usage is over the fair share under this policy.

Parameters:

usage - Resource the resource usage

fairShare - Resource the fair share

Returns:

true if check passes (is over) or false otherwise

getHeadroom

public abstract org.apache.hadoop.yarn.api.records.Resource getHeadroom(org.apache.hadoop.yarn.api.records.Resource queueFairShare,
                                                                        org.apache.hadoop.yarn.api.records.Resource queueUsage,
                                                                        org.apache.hadoop.yarn.api.records.Resource maxAvailable)

Get headroom by calculating the min of clusterAvailable and (queueFairShare - queueUsage) resources that are applicable to this policy. For eg if only memory then leave other resources such as CPU to same as clusterAvailable.

Parameters:

queueFairShare - fairshare in the queue

queueUsage - resources used in the queue

maxAvailable - available resource in cluster for this queue

Returns:

calculated headroom

isChildPolicyAllowed

public boolean isChildPolicyAllowed(SchedulingPolicy childPolicy)

Check whether the policy of a child queue is allowed.

Parameters:

childPolicy - the policy of child queue

Returns:

true if the child policy is allowed; false otherwise