org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair

Class SchedulingPolicy

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy

Direct Known Subclasses:

DominantResourceFairnessPolicy, FairSharePolicy, FifoPolicy

@InterfaceAudience.Public
@InterfaceStability.Evolving
public abstract class SchedulingPolicy
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static SchedulingPolicy	DEFAULT_POLICY
static byte	DEPTH_ANY
static byte	DEPTH_INTERMEDIATE
static byte	DEPTH_LEAF
static byte	DEPTH_PARENT
static byte	DEPTH_ROOT

Constructor Summary

Constructors

Constructor and Description
SchedulingPolicy()

Method Summary

Methods

Modifier and Type	Method and Description
abstract boolean	checkIfAMResourceUsageOverLimit(org.apache.hadoop.yarn.api.records.Resource usage, org.apache.hadoop.yarn.api.records.Resource maxAMResource) Check if a leaf queue's AM resource usage over its limit under this policy
abstract boolean	checkIfUsageOverFairShare(org.apache.hadoop.yarn.api.records.Resource usage, org.apache.hadoop.yarn.api.records.Resource fairShare) Check if the resource usage is over the fair share under this policy
abstract void	computeShares(Collection<? extends Schedulable> schedulables, org.apache.hadoop.yarn.api.records.Resource totalResources) Computes and updates the shares of Schedulables as per the SchedulingPolicy, to be used later for scheduling decisions.
abstract void	computeSteadyShares(Collection<? extends FSQueue> queues, org.apache.hadoop.yarn.api.records.Resource totalResources) Computes and updates the steady shares of FSQueues as per the SchedulingPolicy.
abstract byte	getApplicableDepth() Specifies the depths in the hierarchy, this SchedulingPolicy applies to
abstract Comparator<Schedulable>	getComparator() The comparator returned by this method is to be used for sorting the Schedulables in that queue.
abstract org.apache.hadoop.yarn.api.records.Resource	getHeadroom(org.apache.hadoop.yarn.api.records.Resource queueFairShare, org.apache.hadoop.yarn.api.records.Resource queueUsage, org.apache.hadoop.yarn.api.records.Resource maxAvailable) Get headroom by calculating the min of clusterAvailable and (queueFairShare - queueUsage) resources that are applicable to this policy.
static SchedulingPolicy	getInstance(Class<? extends SchedulingPolicy> clazz) Returns a SchedulingPolicy instance corresponding to the passed clazz
abstract String	getName()
abstract org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator() The ResourceCalculator returned by this method should be used for any calculations involving resources.
void	initialize(org.apache.hadoop.yarn.api.records.Resource clusterCapacity)
static boolean	isApplicableTo(SchedulingPolicy policy, byte depth) Checks if the specified SchedulingPolicy can be used for a queue at the specified depth in the hierarchy
static SchedulingPolicy	parse(String policy) Returns SchedulingPolicy instance corresponding to the SchedulingPolicy passed as a string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_POLICY

public static final SchedulingPolicy DEFAULT_POLICY

DEPTH_LEAF

public static final byte DEPTH_LEAF

See Also:

Constant Field Values

DEPTH_INTERMEDIATE

public static final byte DEPTH_INTERMEDIATE

See Also:

Constant Field Values

DEPTH_ROOT

public static final byte DEPTH_ROOT

See Also:

Constant Field Values

DEPTH_PARENT

public static final byte DEPTH_PARENT

See Also:

Constant Field Values

DEPTH_ANY

public static final byte DEPTH_ANY

See Also:

Constant Field Values

Constructor Detail

SchedulingPolicy

public SchedulingPolicy()

Method Detail

getInstance

public static SchedulingPolicy getInstance(Class<? extends SchedulingPolicy> clazz)

Returns a SchedulingPolicy instance corresponding to the passed clazz

parse

public static SchedulingPolicy parse(String policy)
                              throws AllocationConfigurationException

Returns SchedulingPolicy instance corresponding to the SchedulingPolicy passed as a string. The policy can be "fair" for FairSharePolicy, "fifo" for FifoPolicy, or "drf" for DominantResourceFairnessPolicy. For a custom SchedulingPolicys in the RM classpath, the policy should be canonical class name of the SchedulingPolicy.

Parameters:

policy - canonical class name or "drf" or "fair" or "fifo"

Throws:

AllocationConfigurationException

initialize

public void initialize(org.apache.hadoop.yarn.api.records.Resource clusterCapacity)

getResourceCalculator

public abstract org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

The ResourceCalculator returned by this method should be used for any calculations involving resources.

Returns:

ResourceCalculator instance to use

getName

public abstract String getName()

Returns:

returns the name of SchedulingPolicy

getApplicableDepth

public abstract byte getApplicableDepth()

Specifies the depths in the hierarchy, this SchedulingPolicy applies to

Returns:

depth equal to one of fields SchedulingPolicy#DEPTH_*

isApplicableTo

public static boolean isApplicableTo(SchedulingPolicy policy,
                     byte depth)

Checks if the specified SchedulingPolicy can be used for a queue at the specified depth in the hierarchy

Parameters:

policy - SchedulingPolicy we are checking the depth-applicability for

depth - queue's depth in the hierarchy

Returns:

true if policy is applicable to passed depth, false otherwise

getComparator

public abstract Comparator<Schedulable> getComparator()

The comparator returned by this method is to be used for sorting the Schedulables in that queue.

Returns:

the comparator to sort by

computeShares

public abstract void computeShares(Collection<? extends Schedulable> schedulables,
                 org.apache.hadoop.yarn.api.records.Resource totalResources)

Computes and updates the shares of Schedulables as per the SchedulingPolicy, to be used later for scheduling decisions. The shares computed are instantaneous and only consider queues with running applications.

Parameters:

schedulables - Schedulables whose shares are to be updated

totalResources - Total Resources in the cluster

computeSteadyShares

public abstract void computeSteadyShares(Collection<? extends FSQueue> queues,
                       org.apache.hadoop.yarn.api.records.Resource totalResources)

Computes and updates the steady shares of FSQueues as per the SchedulingPolicy. The steady share does not differentiate between queues with and without running applications under them. The steady share is not used for scheduling, it is displayed on the Web UI for better visibility.

Parameters:

queues - FSQueues whose shares are to be updated

totalResources - Total Resources in the cluster

checkIfUsageOverFairShare

public abstract boolean checkIfUsageOverFairShare(org.apache.hadoop.yarn.api.records.Resource usage,
                                org.apache.hadoop.yarn.api.records.Resource fairShare)

Check if the resource usage is over the fair share under this policy

Parameters:

usage - Resource the resource usage

fairShare - Resource the fair share

Returns:

true if check passes (is over) or false otherwise

checkIfAMResourceUsageOverLimit

public abstract boolean checkIfAMResourceUsageOverLimit(org.apache.hadoop.yarn.api.records.Resource usage,
                                      org.apache.hadoop.yarn.api.records.Resource maxAMResource)

Check if a leaf queue's AM resource usage over its limit under this policy

Parameters:

usage - Resource the resource used by application masters

maxAMResource - Resource the maximum allowed resource for application masters

Returns:

true if AM resource usage is over the limit

getHeadroom

public abstract org.apache.hadoop.yarn.api.records.Resource getHeadroom(org.apache.hadoop.yarn.api.records.Resource queueFairShare,
                                                      org.apache.hadoop.yarn.api.records.Resource queueUsage,
                                                      org.apache.hadoop.yarn.api.records.Resource maxAvailable)

Get headroom by calculating the min of clusterAvailable and (queueFairShare - queueUsage) resources that are applicable to this policy. For eg if only memory then leave other resources such as CPU to same as clusterAvailable.

Parameters:

queueFairShare - fairshare in the queue

queueUsage - resources used in the queue

maxAvailable - available resource in cluster for this queue

Returns:

calculated headroom