Package org.apache.hadoop.yarn.util.resource

Class DominantResourceCalculator

java.lang.Object
org.apache.hadoop.yarn.util.resource.ResourceCalculator
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
@Private @Unstable public class DominantResourceCalculator extends ResourceCalculator
A ResourceCalculator which uses the concept of dominant resource to compare multi-dimensional resources. Essentially the idea is that the in a multi-resource environment, the resource allocation should be determined by the dominant share of an entity (user or queue), which is the maximum share that the entity has been allocated of any resource. In a nutshell, it seeks to maximize the minimum dominant share across all entities. For example, if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, it attempts to equalize CPU share of user A with Memory-share of user B. In the single resource case, it reduces to max-min fairness for that resource. See the Dominant Resource Fairness paper for more details: www.cs.berkeley.edu/~matei/papers/2011/nsdi_drf.pdf

  • Constructor Summary

    Constructors
    Constructor
    Description
    DominantResourceCalculator()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType)
    On a cluster with capacity clusterResource, compare lhs and rhs.
    long
    computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required)
    Compute the number of containers which can be allocated given available and required resources.
    float
    divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator)
    Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.
    org.apache.hadoop.yarn.api.records.Resource
    divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator)
    Divide-and-ceil numerator by denominator.
    org.apache.hadoop.yarn.api.records.Resource
    divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator)
    Divide-and-ceil numerator by denominator.
    org.apache.hadoop.yarn.api.records.Resource
    divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, long denominator)
     
    boolean
    fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger)
    Check if a smaller resource can be contained by bigger resource.
    Set<String>
    getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available)
    Get insufficient resource names via comparing required resource and capacity resource.
    boolean
    isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
    Determine if all resources are zero.
    boolean
    isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
    Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
    boolean
    isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
    Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
    boolean
    isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
    Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
    float
    minRatio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
    Computes the ratio of resource a over resource b.
    org.apache.hadoop.yarn.api.records.Resource
    multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Multiply resource r by factor by and normalize down using step-factor stepFactor.
    org.apache.hadoop.yarn.api.records.Resource
    multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Multiply resource r by factor by and normalize up using step-factor stepFactor.
    org.apache.hadoop.yarn.api.records.Resource
    multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Multiply resource r by factor by and normalize up using step-factor stepFactor.
    org.apache.hadoop.yarn.api.records.Resource
    normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.
    org.apache.hadoop.yarn.api.records.Resource
    normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Get resource rand normalize down using step-factor stepFactor.
    float
    ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
    Ratio of resource a to resource b.
    org.apache.hadoop.yarn.api.records.Resource
    roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Round-down resource r given factor stepFactor.
    org.apache.hadoop.yarn.api.records.Resource
    roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Round-up resource r given factor stepFactor.

    Methods inherited from class org.apache.hadoop.yarn.util.resource.ResourceCalculator

    compare, divideAndCeil, divideAndCeil, divideAndCeil, divideAndCeil, divideSafelyAsFloat, roundDown, roundDown, roundUp, roundUp

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • DominantResourceCalculator

      public DominantResourceCalculator()

  • Method Details

    • compare

      public int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType)
      Description copied from class: ResourceCalculator
      On a cluster with capacity clusterResource, compare lhs and rhs. Consider all resources unless singleType is set to true. When singleType is set to true, consider only one resource as per the ResourceCalculator implementation; the DefaultResourceCalculator considers memory and DominantResourceCalculator considers the dominant resource.
      Specified by:
      compare in class ResourceCalculator
      Parameters:
      clusterResource - cluster capacity
      lhs - First Resource to compare
      rhs - Second Resource to compare
      singleType - Whether to consider a single resource type or all resource types
      Returns:
      -1 if lhs is smaller, 0 if equal and 1 if it is larger

    • computeAvailableContainers

      public long computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required)
      Description copied from class: ResourceCalculator
      Compute the number of containers which can be allocated given available and required resources.
      Specified by:
      computeAvailableContainers in class ResourceCalculator
      Parameters:
      available - available resources
      required - required resources
      Returns:
      number of containers which can be allocated

    • divide

      public float divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator)
      Description copied from class: ResourceCalculator
      Divide resource numerator by resource denominator using specified policy (domination, average, fairness etc.); hence overall clusterResource is provided for context.
      Specified by:
      divide in class ResourceCalculator
      Parameters:
      clusterResource - cluster resources
      numerator - numerator
      denominator - denominator
      Returns:
      numerator/denominator using specific policy

    • isInvalidDivisor

      public boolean isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
      Description copied from class: ResourceCalculator
      Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
      Specified by:
      isInvalidDivisor in class ResourceCalculator
      Parameters:
      r - resource
      Returns:
      true if divisor is invalid (should not be used), false else

    • isAllInvalidDivisor

      public boolean isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
      Description copied from class: ResourceCalculator
      Determine if all resources are zero.
      Specified by:
      isAllInvalidDivisor in class ResourceCalculator
      Parameters:
      r - resource
      Returns:
      true if all divisors are invalid (should not be used), false else

    • ratio

      public float ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
      Description copied from class: ResourceCalculator
      Ratio of resource a to resource b.
      Specified by:
      ratio in class ResourceCalculator
      Parameters:
      a - resource
      b - resource
      Returns:
      ratio of resource a to resource b

    • minRatio

      @Unstable public float minRatio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
      Computes the ratio of resource a over resource b. However, different from ratio(Resource, Resource), this returns the min-share of the resources. For example, ratio(Resource(10, 50), Resource(100, 100)) would return 0.5, whereas minRatio(Resource(10, 50), Resource(100, 100)) would return 0.1.
      Parameters:
      a - the numerator resource.
      b - the denominator resource.
      Returns:
      the min-share ratio of the resources.

    • divideAndCeil

      public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator)
      Description copied from class: ResourceCalculator
      Divide-and-ceil numerator by denominator.
      Specified by:
      divideAndCeil in class ResourceCalculator
      Parameters:
      numerator - numerator resource
      denominator - denominator
      Returns:
      resultant resource

    • divideAndCeil

      public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, long denominator)

    • divideAndCeil

      public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator)
      Description copied from class: ResourceCalculator
      Divide-and-ceil numerator by denominator.
      Specified by:
      divideAndCeil in class ResourceCalculator
      Parameters:
      numerator - numerator resource
      denominator - denominator
      Returns:
      resultant resource

    • normalize

      public org.apache.hadoop.yarn.api.records.Resource normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Description copied from class: ResourceCalculator
      Normalize resource r given the base minimumResource and verify against max allowed maximumResource using a step factor for the normalization.
      Specified by:
      normalize in class ResourceCalculator
      Parameters:
      r - resource
      minimumResource - minimum value
      maximumResource - the upper bound of the resource to be allocated
      stepFactor - the increment for resources to be allocated
      Returns:
      normalized resource

    • roundUp

      public org.apache.hadoop.yarn.api.records.Resource roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Description copied from class: ResourceCalculator
      Round-up resource r given factor stepFactor.
      Specified by:
      roundUp in class ResourceCalculator
      Parameters:
      r - resource
      stepFactor - step-factor
      Returns:
      rounded resource

    • roundDown

      public org.apache.hadoop.yarn.api.records.Resource roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Description copied from class: ResourceCalculator
      Round-down resource r given factor stepFactor.
      Specified by:
      roundDown in class ResourceCalculator
      Parameters:
      r - resource
      stepFactor - step-factor
      Returns:
      rounded resource

    • multiplyAndNormalizeUp

      public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Description copied from class: ResourceCalculator
      Multiply resource r by factor by and normalize up using step-factor stepFactor.
      Specified by:
      multiplyAndNormalizeUp in class ResourceCalculator
      Parameters:
      r - resource to be multiplied
      by - multiplier array for all resource types
      stepFactor - factor by which to normalize up
      Returns:
      resulting normalized resource

    • multiplyAndNormalizeUp

      public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Description copied from class: ResourceCalculator
      Multiply resource r by factor by and normalize up using step-factor stepFactor.
      Specified by:
      multiplyAndNormalizeUp in class ResourceCalculator
      Parameters:
      r - resource to be multiplied
      by - multiplier
      stepFactor - factor by which to normalize up
      Returns:
      resulting normalized resource

    • multiplyAndNormalizeDown

      public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Description copied from class: ResourceCalculator
      Multiply resource r by factor by and normalize down using step-factor stepFactor.
      Specified by:
      multiplyAndNormalizeDown in class ResourceCalculator
      Parameters:
      r - resource to be multiplied
      by - multiplier
      stepFactor - factor by which to normalize down
      Returns:
      resulting normalized resource

    • fitsIn

      public boolean fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger)
      Description copied from class: ResourceCalculator
      Check if a smaller resource can be contained by bigger resource.
      Specified by:
      fitsIn in class ResourceCalculator
      Parameters:
      smaller - smaller resource.
      bigger - bigger resource.
      Returns:
      if true, smaller resource can be contained by bigger resource; false otherwise.

    • normalizeDown

      public org.apache.hadoop.yarn.api.records.Resource normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
      Description copied from class: ResourceCalculator
      Get resource rand normalize down using step-factor stepFactor.
      Specified by:
      normalizeDown in class ResourceCalculator
      Parameters:
      r - resource to be multiplied
      stepFactor - factor by which to normalize down
      Returns:
      resulting normalized resource

    • isAnyMajorResourceZeroOrNegative

      public boolean isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
      Description copied from class: ResourceCalculator
      Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
      Specified by:
      isAnyMajorResourceZeroOrNegative in class ResourceCalculator
      Parameters:
      resource - resource
      Returns:
      returns true if any resource is zero.

    • isAnyMajorResourceAboveZero

      public boolean isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
      Description copied from class: ResourceCalculator
      Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
      Specified by:
      isAnyMajorResourceAboveZero in class ResourceCalculator
      Parameters:
      resource - resource
      Returns:
      returns true if any resource is > 0

    • getInsufficientResourceNames

      public Set<String> getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available)
      Description copied from class: ResourceCalculator
      Get insufficient resource names via comparing required resource and capacity resource.
      Specified by:
      getInsufficientResourceNames in class ResourceCalculator
      Parameters:
      required - - required resource
      available - - available resource
      Returns:
      insufficient resource names