org.apache.hadoop.yarn.util.resource

Class ResourceCalculator

java.lang.Object

org.apache.hadoop.yarn.util.resource.ResourceCalculator

Direct Known Subclasses:

DefaultResourceCalculator, DominantResourceCalculator

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public abstract class ResourceCalculator
extends Object

A set of Resource comparison and manipulation interfaces.

Constructor Summary

Constructors

Constructor and Description
ResourceCalculator()

Method Summary

Modifier and Type	Method and Description
int	compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs) On a cluster with capacity clusterResource, compare lhs and rhs considering all resources.
abstract int	compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType) On a cluster with capacity clusterResource, compare lhs and rhs.
abstract long	computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required) Compute the number of containers which can be allocated given available and required resources.
abstract float	divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator) Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.
static int	divideAndCeil(int a, float b)
static int	divideAndCeil(int a, int b)
static long	divideAndCeil(long a, float b)
static long	divideAndCeil(long a, long b)
abstract org.apache.hadoop.yarn.api.records.Resource	divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator) Divide-and-ceil numerator by denominator.
abstract org.apache.hadoop.yarn.api.records.Resource	divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator) Divide-and-ceil numerator by denominator.
static float	divideSafelyAsFloat(long lhs, long rhs) Divides lhs by rhs.
abstract boolean	fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger) Check if a smaller resource can be contained by bigger resource.
abstract Set<String>	getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available) Get insufficient resource names via comparing required resource and capacity resource.
abstract boolean	isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r) Determine if all resources are zero.
abstract boolean	isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource) Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
abstract boolean	isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource) Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
abstract boolean	isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r) Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
abstract org.apache.hadoop.yarn.api.records.Resource	multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor) Multiply resource r by factor by and normalize down using step-factor stepFactor.
abstract org.apache.hadoop.yarn.api.records.Resource	multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor) Multiply resource r by factor by and normalize up using step-factor stepFactor.
abstract org.apache.hadoop.yarn.api.records.Resource	multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor) Multiply resource r by factor by and normalize up using step-factor stepFactor.
abstract org.apache.hadoop.yarn.api.records.Resource	normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor) Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.
abstract org.apache.hadoop.yarn.api.records.Resource	normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor) Get resource rand normalize down using step-factor stepFactor.
abstract float	ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b) Ratio of resource a to resource b.
static int	roundDown(int a, int b)
static long	roundDown(long a, long b)
abstract org.apache.hadoop.yarn.api.records.Resource	roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor) Round-down resource r given factor stepFactor.
static int	roundUp(int a, int b)
static long	roundUp(long a, long b)
abstract org.apache.hadoop.yarn.api.records.Resource	roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor) Round-up resource r given factor stepFactor.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ResourceCalculator

public ResourceCalculator()

Method Detail

compare

public abstract int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                            org.apache.hadoop.yarn.api.records.Resource lhs,
                            org.apache.hadoop.yarn.api.records.Resource rhs,
                            boolean singleType)

On a cluster with capacity clusterResource, compare lhs and rhs. Consider all resources unless singleType is set to true. When singleType is set to true, consider only one resource as per the ResourceCalculator implementation; the DefaultResourceCalculator considers memory and DominantResourceCalculator considers the dominant resource.

Parameters:

clusterResource - cluster capacity

lhs - First Resource to compare

rhs - Second Resource to compare

singleType - Whether to consider a single resource type or all resource types

Returns:

-1 if lhs is smaller, 0 if equal and 1 if it is larger

compare

public int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                   org.apache.hadoop.yarn.api.records.Resource lhs,
                   org.apache.hadoop.yarn.api.records.Resource rhs)

On a cluster with capacity clusterResource, compare lhs and rhs considering all resources.

Parameters:

clusterResource - cluster capacity

lhs - First Resource to compare

rhs - Second Resource to compare

Returns:

-1 if lhs is smaller, 0 if equal and 1 if it is larger

divideAndCeil

public static int divideAndCeil(int a,
                                int b)

divideAndCeil

public static int divideAndCeil(int a,
                                float b)

divideAndCeil

public static long divideAndCeil(long a,
                                 long b)

divideAndCeil

public static long divideAndCeil(long a,
                                 float b)

divideSafelyAsFloat

public static float divideSafelyAsFloat(long lhs,
                                        long rhs)

Divides lhs by rhs.

Parameters:

lhs - left number.

rhs - right number.

Returns:

If both lhs and rhs are having a value of 0, then we return 0. This is to avoid division by zero and return NaN as a result. If lhs is zero but rhs is not, Float.infinity will be returned as the result.

roundUp

public static int roundUp(int a,
                          int b)

roundUp

public static long roundUp(long a,
                           long b)

roundDown

public static long roundDown(long a,
                             long b)

roundDown

public static int roundDown(int a,
                            int b)

computeAvailableContainers

public abstract long computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available,
                                                org.apache.hadoop.yarn.api.records.Resource required)

Compute the number of containers which can be allocated given available and required resources.

Parameters:

available - available resources

required - required resources

Returns:

number of containers which can be allocated

multiplyAndNormalizeUp

public abstract org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
                                                                                   double by,
                                                                                   org.apache.hadoop.yarn.api.records.Resource stepFactor)

Multiply resource r by factor by and normalize up using step-factor stepFactor.

Parameters:

r - resource to be multiplied

by - multiplier

stepFactor - factor by which to normalize up

Returns:

resulting normalized resource

multiplyAndNormalizeUp

public abstract org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
                                                                                   double[] by,
                                                                                   org.apache.hadoop.yarn.api.records.Resource stepFactor)

Multiply resource r by factor by and normalize up using step-factor stepFactor.

Parameters:

r - resource to be multiplied

by - multiplier array for all resource types

stepFactor - factor by which to normalize up

Returns:

resulting normalized resource

multiplyAndNormalizeDown

public abstract org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
                                                                                     double by,
                                                                                     org.apache.hadoop.yarn.api.records.Resource stepFactor)

Multiply resource r by factor by and normalize down using step-factor stepFactor.

Parameters:

r - resource to be multiplied

by - multiplier

stepFactor - factor by which to normalize down

Returns:

resulting normalized resource

normalize

public abstract org.apache.hadoop.yarn.api.records.Resource normalize(org.apache.hadoop.yarn.api.records.Resource r,
                                                                      org.apache.hadoop.yarn.api.records.Resource minimumResource,
                                                                      org.apache.hadoop.yarn.api.records.Resource maximumResource,
                                                                      org.apache.hadoop.yarn.api.records.Resource stepFactor)

Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.

Parameters:

r - resource

minimumResource - minimum value

maximumResource - the upper bound of the resource to be allocated

stepFactor - the increment for resources to be allocated

Returns:

normalized resource

roundUp

public abstract org.apache.hadoop.yarn.api.records.Resource roundUp(org.apache.hadoop.yarn.api.records.Resource r,
                                                                    org.apache.hadoop.yarn.api.records.Resource stepFactor)

Round-up resource r given factor stepFactor.

Parameters:

r - resource

stepFactor - step-factor

Returns:

rounded resource

roundDown

public abstract org.apache.hadoop.yarn.api.records.Resource roundDown(org.apache.hadoop.yarn.api.records.Resource r,
                                                                      org.apache.hadoop.yarn.api.records.Resource stepFactor)

Round-down resource r given factor stepFactor.

Parameters:

r - resource

stepFactor - step-factor

Returns:

rounded resource

divide

public abstract float divide(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                             org.apache.hadoop.yarn.api.records.Resource numerator,
                             org.apache.hadoop.yarn.api.records.Resource denominator)

Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.

Parameters:

clusterResource - cluster resources

numerator - numerator

denominator - denominator

Returns:

numerator/denominator using specific policy

isInvalidDivisor

public abstract boolean isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)

Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)

Parameters:

r - resource

Returns:

true if divisor is invalid (should not be used), false else

isAllInvalidDivisor

public abstract boolean isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)

Determine if all resources are zero.

Parameters:

r - resource

Returns:

true if all divisors are invalid (should not be used), false else

ratio

public abstract float ratio(org.apache.hadoop.yarn.api.records.Resource a,
                            org.apache.hadoop.yarn.api.records.Resource b)

Ratio of resource a to resource b.

Parameters:

a - resource

b - resource

Returns:

ratio of resource a to resource b

divideAndCeil

public abstract org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
                                                                          int denominator)

Divide-and-ceil numerator by denominator.

Parameters:

numerator - numerator resource

denominator - denominator

Returns:

resultant resource

divideAndCeil

public abstract org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
                                                                          float denominator)

Divide-and-ceil numerator by denominator.

Parameters:

numerator - numerator resource

denominator - denominator

Returns:

resultant resource

fitsIn

public abstract boolean fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller,
                               org.apache.hadoop.yarn.api.records.Resource bigger)

Check if a smaller resource can be contained by bigger resource.

Parameters:

smaller - smaller resource.

bigger - bigger resource.

Returns:

if true, smaller resource can be contained by bigger resource; false otherwise.

isAnyMajorResourceZeroOrNegative

public abstract boolean isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)

Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.

Parameters:

resource - resource

Returns:

returns true if any resource is zero.

normalizeDown

public abstract org.apache.hadoop.yarn.api.records.Resource normalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
                                                                          org.apache.hadoop.yarn.api.records.Resource stepFactor)

Get resource rand normalize down using step-factor stepFactor.

Parameters:

r - resource to be multiplied

stepFactor - factor by which to normalize down

Returns:

resulting normalized resource

isAnyMajorResourceAboveZero

public abstract boolean isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)

Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.

Parameters:

resource - resource

Returns:

returns true if any resource is > 0

getInsufficientResourceNames

public abstract Set<String> getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required,
                                                         org.apache.hadoop.yarn.api.records.Resource available)

Get insufficient resource names via comparing required resource and capacity resource.

Parameters:

required - - required resource

available - - available resource

Returns:

insufficient resource names