@InterfaceAudience.Private @InterfaceStability.Unstable public class DominantResourceCalculator extends ResourceCalculator
ResourceCalculator
which uses the concept of
dominant resource to compare multi-dimensional resources.
Essentially the idea is that the in a multi-resource environment,
the resource allocation should be determined by the dominant share
of an entity (user or queue), which is the maximum share that the
entity has been allocated of any resource.
In a nutshell, it seeks to maximize the minimum dominant share across
all entities.
For example, if user A runs CPU-heavy tasks and user B runs
memory-heavy tasks, it attempts to equalize CPU share of user A
with Memory-share of user B.
In the single resource case, it reduces to max-min fairness for that resource.
See the Dominant Resource Fairness paper for more details:
www.cs.berkeley.edu/~matei/papers/2011/nsdi_drf.pdfConstructor and Description |
---|
DominantResourceCalculator() |
Modifier and Type | Method and Description |
---|---|
int |
compare(org.apache.hadoop.yarn.api.records.Resource clusterResource,
org.apache.hadoop.yarn.api.records.Resource lhs,
org.apache.hadoop.yarn.api.records.Resource rhs,
boolean singleType)
On a cluster with capacity
clusterResource , compare lhs
and rhs . |
long |
computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available,
org.apache.hadoop.yarn.api.records.Resource required)
Compute the number of containers which can be allocated given
available and required resources. |
float |
divide(org.apache.hadoop.yarn.api.records.Resource clusterResource,
org.apache.hadoop.yarn.api.records.Resource numerator,
org.apache.hadoop.yarn.api.records.Resource denominator)
Divide resource
numerator by resource denominator
using specified policy (domination, average, fairness etc.); hence overall
clusterResource is provided for context. |
org.apache.hadoop.yarn.api.records.Resource |
divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
float denominator)
Divide-and-ceil
numerator by denominator . |
org.apache.hadoop.yarn.api.records.Resource |
divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
int denominator)
Divide-and-ceil
numerator by denominator . |
org.apache.hadoop.yarn.api.records.Resource |
divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
long denominator) |
boolean |
fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller,
org.apache.hadoop.yarn.api.records.Resource bigger)
Check if a smaller resource can be contained by bigger resource.
|
boolean |
isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
Check if resource has any major resource types (which are all NodeManagers
included) has a > 0 value.
|
boolean |
isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
Check if resource has any major resource types (which are all NodeManagers
included) a zero value or negative value.
|
boolean |
isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
Determine if a resource is not suitable for use as a divisor
(will result in divide by 0, etc)
|
org.apache.hadoop.yarn.api.records.Resource |
multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
double by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Multiply resource
r by factor by
and normalize down using step-factor stepFactor . |
org.apache.hadoop.yarn.api.records.Resource |
multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
double[] by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Multiply resource
r by factor by
and normalize up using step-factor stepFactor . |
org.apache.hadoop.yarn.api.records.Resource |
multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
double by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Multiply resource
r by factor by
and normalize up using step-factor stepFactor . |
org.apache.hadoop.yarn.api.records.Resource |
normalize(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource minimumResource,
org.apache.hadoop.yarn.api.records.Resource maximumResource,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Normalize resource
r given the base
minimumResource and verify against max allowed
maximumResource using a step factor for the normalization. |
org.apache.hadoop.yarn.api.records.Resource |
normalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Get resource
r and normalize down using step-factor
stepFactor . |
float |
ratio(org.apache.hadoop.yarn.api.records.Resource a,
org.apache.hadoop.yarn.api.records.Resource b)
Ratio of resource
a to resource b . |
org.apache.hadoop.yarn.api.records.Resource |
roundDown(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Round-down resource
r given factor stepFactor . |
org.apache.hadoop.yarn.api.records.Resource |
roundUp(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Round-up resource
r given factor stepFactor . |
compare, divideAndCeil, divideAndCeil, divideAndCeil, divideAndCeil, roundDown, roundDown, roundUp, roundUp
public int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType)
ResourceCalculator
clusterResource
, compare lhs
and rhs
. Consider all resources unless singleType
is set
to true. When singleType
is set to true, consider only one
resource as per the ResourceCalculator
implementation; the
DefaultResourceCalculator
considers memory and
DominantResourceCalculator
considers the dominant resource.compare
in class ResourceCalculator
clusterResource
- cluster capacitylhs
- First Resource
to comparerhs
- Second Resource
to comparesingleType
- Whether to consider a single resource type or all
resource typeslhs
is smaller, 0 if equal and 1 if it is largerpublic long computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required)
ResourceCalculator
available
and required
resources.computeAvailableContainers
in class ResourceCalculator
available
- available resourcesrequired
- required resourcespublic float divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator)
ResourceCalculator
numerator
by resource denominator
using specified policy (domination, average, fairness etc.); hence overall
clusterResource
is provided for context.divide
in class ResourceCalculator
clusterResource
- cluster resourcesnumerator
- numeratordenominator
- denominatornumerator
/denominator
using specific policypublic boolean isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
ResourceCalculator
isInvalidDivisor
in class ResourceCalculator
r
- resourcepublic float ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
ResourceCalculator
a
to resource b
.ratio
in class ResourceCalculator
a
- resourceb
- resourcea
to resource b
public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator)
ResourceCalculator
numerator
by denominator
.divideAndCeil
in class ResourceCalculator
numerator
- numerator resourcedenominator
- denominatorpublic org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, long denominator)
public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator)
ResourceCalculator
numerator
by denominator
.divideAndCeil
in class ResourceCalculator
numerator
- numerator resourcedenominator
- denominatorpublic org.apache.hadoop.yarn.api.records.Resource normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculator
r
given the base
minimumResource
and verify against max allowed
maximumResource
using a step factor for the normalization.normalize
in class ResourceCalculator
r
- resourceminimumResource
- minimum valuemaximumResource
- the upper bound of the resource to be allocatedstepFactor
- the increment for resources to be allocatedpublic org.apache.hadoop.yarn.api.records.Resource roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculator
r
given factor stepFactor
.roundUp
in class ResourceCalculator
r
- resourcestepFactor
- step-factorpublic org.apache.hadoop.yarn.api.records.Resource roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculator
r
given factor stepFactor
.roundDown
in class ResourceCalculator
r
- resourcestepFactor
- step-factorpublic org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculator
r
by factor by
and normalize up using step-factor stepFactor
.multiplyAndNormalizeUp
in class ResourceCalculator
r
- resource to be multipliedby
- multiplier array for all resource typesstepFactor
- factor by which to normalize uppublic org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculator
r
by factor by
and normalize up using step-factor stepFactor
.multiplyAndNormalizeUp
in class ResourceCalculator
r
- resource to be multipliedby
- multiplierstepFactor
- factor by which to normalize uppublic org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculator
r
by factor by
and normalize down using step-factor stepFactor
.multiplyAndNormalizeDown
in class ResourceCalculator
r
- resource to be multipliedby
- multiplierstepFactor
- factor by which to normalize downpublic boolean fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger)
ResourceCalculator
fitsIn
in class ResourceCalculator
public org.apache.hadoop.yarn.api.records.Resource normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculator
r
and normalize down using step-factor
stepFactor
.normalizeDown
in class ResourceCalculator
r
- resource to be multipliedstepFactor
- factor by which to normalize downpublic boolean isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
ResourceCalculator
isAnyMajorResourceZeroOrNegative
in class ResourceCalculator
resource
- resourcepublic boolean isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
ResourceCalculator
isAnyMajorResourceAboveZero
in class ResourceCalculator
resource
- resourceCopyright © 2008–2019 Apache Software Foundation. All rights reserved.