org.apache.hadoop.yarn.util.resource

Class DominantResourceCalculator

java.lang.Object

org.apache.hadoop.yarn.util.resource.ResourceCalculator

org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class DominantResourceCalculator
extends ResourceCalculator

A ResourceCalculator which uses the concept of dominant resource to compare multi-dimensional resources. Essentially the idea is that the in a multi-resource environment, the resource allocation should be determined by the dominant share of an entity (user or queue), which is the maximum share that the entity has been allocated of any resource. In a nutshell, it seeks to maximize the minimum dominant share across all entities. For example, if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, it attempts to equalize CPU share of user A with Memory-share of user B. In the single resource case, it reduces to max-min fairness for that resource. See the Dominant Resource Fairness paper for more details: www.cs.berkeley.edu/~matei/papers/2011/nsdi_drf.pdf

Constructor Summary

Constructors

Constructor and Description
DominantResourceCalculator()

Method Summary

Modifier and Type	Method and Description
int	compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType) On a cluster with capacity clusterResource, compare lhs and rhs.
long	computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required) Compute the number of containers which can be allocated given available and required resources.
float	divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator) Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.
org.apache.hadoop.yarn.api.records.Resource	divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator) Divide-and-ceil numerator by denominator.
org.apache.hadoop.yarn.api.records.Resource	divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator) Divide-and-ceil numerator by denominator.
org.apache.hadoop.yarn.api.records.Resource	divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, long denominator)
boolean	fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger) Check if a smaller resource can be contained by bigger resource.
Set<String>	getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available) Get insufficient resource names via comparing required resource and capacity resource.
boolean	isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r) Determine if all resources are zero.
boolean	isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource) Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
boolean	isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource) Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
boolean	isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r) Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
float	minRatio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b) Computes the ratio of resource a over resource b.
org.apache.hadoop.yarn.api.records.Resource	multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor) Multiply resource r by factor by and normalize down using step-factor stepFactor.
org.apache.hadoop.yarn.api.records.Resource	multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor) Multiply resource r by factor by and normalize up using step-factor stepFactor.
org.apache.hadoop.yarn.api.records.Resource	multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor) Multiply resource r by factor by and normalize up using step-factor stepFactor.
org.apache.hadoop.yarn.api.records.Resource	normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor) Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.
org.apache.hadoop.yarn.api.records.Resource	normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor) Get resource rand normalize down using step-factor stepFactor.
float	ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b) Ratio of resource a to resource b.
org.apache.hadoop.yarn.api.records.Resource	roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor) Round-down resource r given factor stepFactor.
org.apache.hadoop.yarn.api.records.Resource	roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor) Round-up resource r given factor stepFactor.

Methods inherited from class org.apache.hadoop.yarn.util.resource.ResourceCalculator

compare, divideAndCeil, divideAndCeil, divideAndCeil, divideAndCeil, divideSafelyAsFloat, roundDown, roundDown, roundUp, roundUp

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DominantResourceCalculator

public DominantResourceCalculator()

Method Detail

compare

public int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                   org.apache.hadoop.yarn.api.records.Resource lhs,
                   org.apache.hadoop.yarn.api.records.Resource rhs,
                   boolean singleType)

Description copied from class: ResourceCalculator

On a cluster with capacity clusterResource, compare lhs and rhs. Consider all resources unless singleType is set to true. When singleType is set to true, consider only one resource as per the ResourceCalculator implementation; the DefaultResourceCalculator considers memory and DominantResourceCalculator considers the dominant resource.

Specified by:

compare in class ResourceCalculator

Parameters:

clusterResource - cluster capacity

lhs - First Resource to compare

rhs - Second Resource to compare

singleType - Whether to consider a single resource type or all resource types

Returns:

-1 if lhs is smaller, 0 if equal and 1 if it is larger

computeAvailableContainers

public long computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available,
                                       org.apache.hadoop.yarn.api.records.Resource required)

Description copied from class: ResourceCalculator

Compute the number of containers which can be allocated given available and required resources.

Specified by:

computeAvailableContainers in class ResourceCalculator

Parameters:

available - available resources

required - required resources

Returns:

number of containers which can be allocated

divide

public float divide(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                    org.apache.hadoop.yarn.api.records.Resource numerator,
                    org.apache.hadoop.yarn.api.records.Resource denominator)

Description copied from class: ResourceCalculator

Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.

Specified by:

divide in class ResourceCalculator

Parameters:

clusterResource - cluster resources

numerator - numerator

denominator - denominator

Returns:

numerator/denominator using specific policy

isInvalidDivisor

public boolean isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)

Description copied from class: ResourceCalculator

Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)

Specified by:

isInvalidDivisor in class ResourceCalculator

Parameters:

r - resource

Returns:

true if divisor is invalid (should not be used), false else

isAllInvalidDivisor

public boolean isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)

Description copied from class: ResourceCalculator

Determine if all resources are zero.

Specified by:

isAllInvalidDivisor in class ResourceCalculator

Parameters:

r - resource

Returns:

true if all divisors are invalid (should not be used), false else

ratio

public float ratio(org.apache.hadoop.yarn.api.records.Resource a,
                   org.apache.hadoop.yarn.api.records.Resource b)

Description copied from class: ResourceCalculator

Ratio of resource a to resource b.

Specified by:

ratio in class ResourceCalculator

Parameters:

a - resource

b - resource

Returns:

ratio of resource a to resource b

minRatio

@InterfaceStability.Unstable
public float minRatio(org.apache.hadoop.yarn.api.records.Resource a,
                                                   org.apache.hadoop.yarn.api.records.Resource b)

Computes the ratio of resource a over resource b. However, different from ratio(Resource, Resource), this returns the min-share of the resources. For example, ratio(Resource(10, 50), Resource(100, 100)) would return 0.5, whereas minRatio(Resource(10, 50), Resource(100, 100)) would return 0.1.

Parameters:

a - the numerator resource.

b - the denominator resource.

Returns:

the min-share ratio of the resources.

divideAndCeil

public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
                                                                 int denominator)

Description copied from class: ResourceCalculator

Divide-and-ceil numerator by denominator.

Specified by:

divideAndCeil in class ResourceCalculator

Parameters:

numerator - numerator resource

denominator - denominator

Returns:

resultant resource

divideAndCeil

public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
                                                                 long denominator)

divideAndCeil

public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
                                                                 float denominator)

Description copied from class: ResourceCalculator

Divide-and-ceil numerator by denominator.

Specified by:

divideAndCeil in class ResourceCalculator

Parameters:

numerator - numerator resource

denominator - denominator

Returns:

resultant resource

normalize

public org.apache.hadoop.yarn.api.records.Resource normalize(org.apache.hadoop.yarn.api.records.Resource r,
                                                             org.apache.hadoop.yarn.api.records.Resource minimumResource,
                                                             org.apache.hadoop.yarn.api.records.Resource maximumResource,
                                                             org.apache.hadoop.yarn.api.records.Resource stepFactor)

Description copied from class: ResourceCalculator

Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.

Specified by:

normalize in class ResourceCalculator

Parameters:

r - resource

minimumResource - minimum value

maximumResource - the upper bound of the resource to be allocated

stepFactor - the increment for resources to be allocated

Returns:

normalized resource

roundUp

public org.apache.hadoop.yarn.api.records.Resource roundUp(org.apache.hadoop.yarn.api.records.Resource r,
                                                           org.apache.hadoop.yarn.api.records.Resource stepFactor)

Description copied from class: ResourceCalculator

Round-up resource r given factor stepFactor.

Specified by:

roundUp in class ResourceCalculator

Parameters:

r - resource

stepFactor - step-factor

Returns:

rounded resource

roundDown

public org.apache.hadoop.yarn.api.records.Resource roundDown(org.apache.hadoop.yarn.api.records.Resource r,
                                                             org.apache.hadoop.yarn.api.records.Resource stepFactor)

Description copied from class: ResourceCalculator

Round-down resource r given factor stepFactor.

Specified by:

roundDown in class ResourceCalculator

Parameters:

r - resource

stepFactor - step-factor

Returns:

rounded resource

multiplyAndNormalizeUp

public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
                                                                          double[] by,
                                                                          org.apache.hadoop.yarn.api.records.Resource stepFactor)

Description copied from class: ResourceCalculator

Multiply resource r by factor by and normalize up using step-factor stepFactor.

Specified by:

multiplyAndNormalizeUp in class ResourceCalculator

Parameters:

r - resource to be multiplied

by - multiplier array for all resource types

stepFactor - factor by which to normalize up

Returns:

resulting normalized resource

multiplyAndNormalizeUp

public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
                                                                          double by,
                                                                          org.apache.hadoop.yarn.api.records.Resource stepFactor)

Description copied from class: ResourceCalculator

Multiply resource r by factor by and normalize up using step-factor stepFactor.

Specified by:

multiplyAndNormalizeUp in class ResourceCalculator

Parameters:

r - resource to be multiplied

by - multiplier

stepFactor - factor by which to normalize up

Returns:

resulting normalized resource

multiplyAndNormalizeDown

public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
                                                                            double by,
                                                                            org.apache.hadoop.yarn.api.records.Resource stepFactor)

Description copied from class: ResourceCalculator

Multiply resource r by factor by and normalize down using step-factor stepFactor.

Specified by:

multiplyAndNormalizeDown in class ResourceCalculator

Parameters:

r - resource to be multiplied

by - multiplier

stepFactor - factor by which to normalize down

Returns:

resulting normalized resource

fitsIn

public boolean fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller,
                      org.apache.hadoop.yarn.api.records.Resource bigger)

Description copied from class: ResourceCalculator

Check if a smaller resource can be contained by bigger resource.

Specified by:

fitsIn in class ResourceCalculator

Parameters:

smaller - smaller resource.

bigger - bigger resource.

Returns:

if true, smaller resource can be contained by bigger resource; false otherwise.

normalizeDown

public org.apache.hadoop.yarn.api.records.Resource normalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
                                                                 org.apache.hadoop.yarn.api.records.Resource stepFactor)

Description copied from class: ResourceCalculator

Get resource rand normalize down using step-factor stepFactor.

Specified by:

normalizeDown in class ResourceCalculator

Parameters:

r - resource to be multiplied

stepFactor - factor by which to normalize down

Returns:

resulting normalized resource

isAnyMajorResourceZeroOrNegative

public boolean isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)

Description copied from class: ResourceCalculator

Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.

Specified by:

isAnyMajorResourceZeroOrNegative in class ResourceCalculator

Parameters:

resource - resource

Returns:

returns true if any resource is zero.

isAnyMajorResourceAboveZero

public boolean isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)

Description copied from class: ResourceCalculator

Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.

Specified by:

isAnyMajorResourceAboveZero in class ResourceCalculator

Parameters:

resource - resource

Returns:

returns true if any resource is > 0

getInsufficientResourceNames

public Set<String> getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required,
                                                org.apache.hadoop.yarn.api.records.Resource available)

Description copied from class: ResourceCalculator

Get insufficient resource names via comparing required resource and capacity resource.

Specified by:

getInsufficientResourceNames in class ResourceCalculator

Parameters:

required - - required resource

available - - available resource

Returns:

insufficient resource names