Package org.apache.hadoop.yarn.util.resource

Class ResourceCalculator

java.lang.Object
org.apache.hadoop.yarn.util.resource.ResourceCalculator
Direct Known Subclasses:
DefaultResourceCalculator, DominantResourceCalculator
@Private @Unstable public abstract class ResourceCalculator extends Object
A set of Resource comparison and manipulation interfaces.

  • Constructor Summary

    Constructors
    Constructor
    Description
    ResourceCalculator()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs)
    On a cluster with capacity clusterResource, compare lhs and rhs considering all resources.
    abstract int
    compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType)
    On a cluster with capacity clusterResource, compare lhs and rhs.
    abstract long
    computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required)
    Compute the number of containers which can be allocated given available and required resources.
    abstract float
    divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator)
    Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.
    static int
    divideAndCeil(int a, float b)
     
    static int
    divideAndCeil(int a, int b)
     
    static long
    divideAndCeil(long a, float b)
     
    static long
    divideAndCeil(long a, long b)
     
    abstract org.apache.hadoop.yarn.api.records.Resource
    divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator)
    Divide-and-ceil numerator by denominator.
    abstract org.apache.hadoop.yarn.api.records.Resource
    divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator)
    Divide-and-ceil numerator by denominator.
    static float
    divideSafelyAsFloat(long lhs, long rhs)
    Divides lhs by rhs.
    abstract boolean
    fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger)
    Check if a smaller resource can be contained by bigger resource.
    abstract Set<String>
    getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available)
    Get insufficient resource names via comparing required resource and capacity resource.
    abstract boolean
    isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
    Determine if all resources are zero.
    abstract boolean
    isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
    Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
    abstract boolean
    isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
    Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
    abstract boolean
    isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
    Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
    abstract org.apache.hadoop.yarn.api.records.Resource
    multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Multiply resource r by factor by and normalize down using step-factor stepFactor.
    abstract org.apache.hadoop.yarn.api.records.Resource
    multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Multiply resource r by factor by and normalize up using step-factor stepFactor.
    abstract org.apache.hadoop.yarn.api.records.Resource
    multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Multiply resource r by factor by and normalize up using step-factor stepFactor.
    abstract org.apache.hadoop.yarn.api.records.Resource
    normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.
    abstract org.apache.hadoop.yarn.api.records.Resource
    normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Get resource rand normalize down using step-factor stepFactor.
    abstract float
    ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
    Ratio of resource a to resource b.
    static int
    roundDown(int a, int b)
     
    static long
    roundDown(long a, long b)
     
    abstract org.apache.hadoop.yarn.api.records.Resource
    roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Round-down resource r given factor stepFactor.
    static int
    roundUp(int a, int b)
     
    static long
    roundUp(long a, long b)
     
    abstract org.apache.hadoop.yarn.api.records.Resource
    roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Round-up resource r given factor stepFactor.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • ResourceCalculator

      public ResourceCalculator()

  • Method Details

    • compare

      public abstract int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType)
      On a cluster with capacity clusterResource, compare lhs and rhs. Consider all resources unless singleType is set to true. When singleType is set to true, consider only one resource as per the ResourceCalculator implementation; the DefaultResourceCalculator considers memory and DominantResourceCalculator considers the dominant resource.
      Parameters:
      clusterResource - cluster capacity
      lhs - First Resource to compare
      rhs - Second Resource to compare
      singleType - Whether to consider a single resource type or all resource types
      Returns:
      -1 if lhs is smaller, 0 if equal and 1 if it is larger

    • compare

      public int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs)
      On a cluster with capacity clusterResource, compare lhs and rhs considering all resources.
      Parameters:
      clusterResource - cluster capacity
      lhs - First Resource to compare
      rhs - Second Resource to compare
      Returns:
      -1 if lhs is smaller, 0 if equal and 1 if it is larger

    • divideAndCeil

      public static int divideAndCeil(int a, int b)

    • divideAndCeil

      public static int divideAndCeil(int a, float b)

    • divideAndCeil

      public static long divideAndCeil(long a, long b)

    • divideAndCeil

      public static long divideAndCeil(long a, float b)

    • divideSafelyAsFloat

      public static float divideSafelyAsFloat(long lhs, long rhs)
      Divides lhs by rhs.
      Parameters:
      lhs - left number.
      rhs - right number.
      Returns:
      If both lhs and rhs are having a value of 0, then we return 0. This is to avoid division by zero and return NaN as a result. If lhs is zero but rhs is not, Float.infinity will be returned as the result.

    • roundUp

      public static int roundUp(int a, int b)

    • roundUp

      public static long roundUp(long a, long b)

    • roundDown

      public static long roundDown(long a, long b)

    • roundDown

      public static int roundDown(int a, int b)

    • computeAvailableContainers

      public abstract long computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required)
      Compute the number of containers which can be allocated given available and required resources.
      Parameters:
      available - available resources
      required - required resources
      Returns:
      number of containers which can be allocated

    • multiplyAndNormalizeUp

      public abstract org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Multiply resource r by factor by and normalize up using step-factor stepFactor.
      Parameters:
      r - resource to be multiplied
      by - multiplier
      stepFactor - factor by which to normalize up
      Returns:
      resulting normalized resource

    • multiplyAndNormalizeUp

      public abstract org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Multiply resource r by factor by and normalize up using step-factor stepFactor.
      Parameters:
      r - resource to be multiplied
      by - multiplier array for all resource types
      stepFactor - factor by which to normalize up
      Returns:
      resulting normalized resource

    • multiplyAndNormalizeDown

      public abstract org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Multiply resource r by factor by and normalize down using step-factor stepFactor.
      Parameters:
      r - resource to be multiplied
      by - multiplier
      stepFactor - factor by which to normalize down
      Returns:
      resulting normalized resource

    • normalize

      public abstract org.apache.hadoop.yarn.api.records.Resource normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.
      Parameters:
      r - resource
      minimumResource - minimum value
      maximumResource - the upper bound of the resource to be allocated
      stepFactor - the increment for resources to be allocated
      Returns:
      normalized resource

    • roundUp

      public abstract org.apache.hadoop.yarn.api.records.Resource roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Round-up resource r given factor stepFactor.
      Parameters:
      r - resource
      stepFactor - step-factor
      Returns:
      rounded resource

    • roundDown

      public abstract org.apache.hadoop.yarn.api.records.Resource roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Round-down resource r given factor stepFactor.
      Parameters:
      r - resource
      stepFactor - step-factor
      Returns:
      rounded resource

    • divide

      public abstract float divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator)
      Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.
      Parameters:
      clusterResource - cluster resources
      numerator - numerator
      denominator - denominator
      Returns:
      numerator/denominator using specific policy

    • isInvalidDivisor

      public abstract boolean isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
      Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
      Parameters:
      r - resource
      Returns:
      true if divisor is invalid (should not be used), false else

    • isAllInvalidDivisor

      public abstract boolean isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
      Determine if all resources are zero.
      Parameters:
      r - resource
      Returns:
      true if all divisors are invalid (should not be used), false else

    • ratio

      public abstract float ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
      Ratio of resource a to resource b.
      Parameters:
      a - resource
      b - resource
      Returns:
      ratio of resource a to resource b

    • divideAndCeil

      public abstract org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator)
      Divide-and-ceil numerator by denominator.
      Parameters:
      numerator - numerator resource
      denominator - denominator
      Returns:
      resultant resource

    • divideAndCeil

      public abstract org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator)
      Divide-and-ceil numerator by denominator.
      Parameters:
      numerator - numerator resource
      denominator - denominator
      Returns:
      resultant resource

    • fitsIn

      public abstract boolean fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger)
      Check if a smaller resource can be contained by bigger resource.
      Parameters:
      smaller - smaller resource.
      bigger - bigger resource.
      Returns:
      if true, smaller resource can be contained by bigger resource; false otherwise.

    • isAnyMajorResourceZeroOrNegative

      public abstract boolean isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
      Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
      Parameters:
      resource - resource
      Returns:
      returns true if any resource is zero.

    • normalizeDown

      public abstract org.apache.hadoop.yarn.api.records.Resource normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Get resource rand normalize down using step-factor stepFactor.
      Parameters:
      r - resource to be multiplied
      stepFactor - factor by which to normalize down
      Returns:
      resulting normalized resource

    • isAnyMajorResourceAboveZero

      public abstract boolean isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
      Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
      Parameters:
      resource - resource
      Returns:
      returns true if any resource is > 0

    • getInsufficientResourceNames

      public abstract Set<String> getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available)
      Get insufficient resource names via comparing required resource and capacity resource.
      Parameters:
      required - - required resource
      available - - available resource
      Returns:
      insufficient resource names