Package org.apache.hadoop.yarn.api.records

Class Resource

java.lang.Object
org.apache.hadoop.yarn.api.records.Resource
All Implemented Interfaces:
Comparable<Resource>
@Public @Stable public abstract class Resource extends Object implements Comparable<Resource>

Resource models a set of computer resources in the cluster.

Currently it models both memory and CPU.

The unit for memory is megabytes. CPU is modeled with virtual cores (vcores), a unit for expressing parallelism. A node's capacity should be configured with virtual cores equal to its number of physical cores. A container should be requested with the number of cores it can saturate, i.e. the average number of threads it expects to have runnable at a time.

Virtual cores take integer values and thus currently CPU-scheduling is very coarse. A complementary axis for CPU requests that represents processing power will likely be added in the future to enable finer-grained resource configuration.

Typically, applications request Resource of suitable capability to run their component tasks.

See Also:

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    MEMORY_INDEX
     
    protected org.apache.hadoop.yarn.api.records.ResourceInformation[]
    resources
     
    static final int
    VCORES_INDEX
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    Resource()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected static int
    castToIntSafely(long value)
    Convert long to int for a resource value safely.
    int
    compareTo(Resource other)
     
    static void
    copy(Resource source, Resource dest)
     
    boolean
    equals(Object obj)
     
    List<org.apache.hadoop.yarn.api.records.ResourceInformation>
    getAllResourcesListCopy()
    Get list of resource information, this will be used by JAXB.
    String
    getFormattedString()
    This method is to get memory in terms of KB|MB|GB.
    String
    getFormattedString(long memory)
     
    abstract int
    getMemory()
    Deprecated.
    long
    getMemorySize()
    Get memory of the resource.
    org.apache.hadoop.yarn.api.records.ResourceInformation
    getResourceInformation(int index)
    Get ResourceInformation for a specified resource from a given index.
    org.apache.hadoop.yarn.api.records.ResourceInformation
    getResourceInformation(String resource)
    Get ResourceInformation for a specified resource.
    org.apache.hadoop.yarn.api.records.ResourceInformation[]
    getResources()
    Get ResourceInformation for all resources.
    long
    getResourceValue(String resource)
    Get the value for a specified resource.
    abstract int
    getVirtualCores()
    Get number of virtual cpu cores of the resource.
    int
    hashCode()
     
    protected static org.apache.hadoop.yarn.api.records.ResourceInformation
    newDefaultInformation(String name, String unit, long value)
    Create ResourceInformation with basic fields.
    static Resource
    newInstance(int memory, int vCores)
     
    static Resource
    newInstance(long memory, int vCores)
     
    static Resource
    newInstance(long memory, int vCores, Map<String,Long> others)
    Create a new Resource instance with the given CPU and memory values and additional resource values as set in the others parameter.
    static Resource
    newInstance(Resource resource)
     
    abstract void
    setMemory(int memory)
    Deprecated.
    void
    setMemorySize(long memory)
    Set memory of the resource.
    void
    setResourceInformation(int index, org.apache.hadoop.yarn.api.records.ResourceInformation resourceInformation)
    Set the ResourceInformation object for a particular resource.
    void
    setResourceInformation(String resource, org.apache.hadoop.yarn.api.records.ResourceInformation resourceInformation)
    Set the ResourceInformation object for a particular resource.
    protected void
    setResources(org.apache.hadoop.yarn.api.records.ResourceInformation[] resources)
     
    void
    setResourceValue(int index, long value)
    Set the value of a resource in the ResourceInformation object.
    void
    setResourceValue(String resource, long value)
    Set the value of a resource in the ResourceInformation object.
    abstract void
    setVirtualCores(int vCores)
    Set number of virtual cpu cores of the resource.
    protected void
    throwExceptionWhenArrayOutOfBound(int index)
     
    String
    toFormattedString()
     
    String
    toString()
     

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

  • Field Details

    • resources

      protected org.apache.hadoop.yarn.api.records.ResourceInformation[] resources

    • MEMORY_INDEX

      @Private public static final int MEMORY_INDEX
      See Also:

    • VCORES_INDEX

      @Private public static final int VCORES_INDEX
      See Also:

  • Constructor Details

    • Resource

      public Resource()

  • Method Details

    • newInstance

      @Public @Stable public static Resource newInstance(int memory, int vCores)

    • newInstance

      @Public @Stable public static Resource newInstance(long memory, int vCores)

    • newInstance

      @Public @Stable public static Resource newInstance(long memory, int vCores, Map<String,Long> others)
      Create a new Resource instance with the given CPU and memory values and additional resource values as set in the others parameter. Note that the CPU and memory settings in the others parameter will be ignored.
      Parameters:
      memory - the memory value
      vCores - the CPU value
      others - a map of other resource values indexed by resource name
      Returns:
      a Resource instance with the given resource values

    • newInstance

      @Private @Unstable public static Resource newInstance(Resource resource)

    • copy

      @Private @Unstable public static void copy(Resource source, Resource dest)

    • getMemory

      @Public @Deprecated public abstract int getMemory()
      Deprecated.
      This method is DEPRECATED: Use getMemorySize() instead Get memory of the resource. Note - while memory has never had a unit specified, all YARN configurations have specified memory in MB. The assumption has been that the daemons and applications are always using the same units. With the introduction of the ResourceInformation class we have support for units - so this function will continue to return memory but in the units of MB
      Returns:
      memory(in MB) of the resource

    • getMemorySize

      @Public @Stable public long getMemorySize()
      Get memory of the resource. Note - while memory has never had a unit specified, all YARN configurations have specified memory in MB. The assumption has been that the daemons and applications are always using the same units. With the introduction of the ResourceInformation class we have support for units - so this function will continue to return memory but in the units of MB
      Returns:
      memory of the resource

    • setMemory

      @Public @Deprecated public abstract void setMemory(int memory)
      Deprecated.
      Set memory of the resource. Note - while memory has never had a unit specified, all YARN configurations have specified memory in MB. The assumption has been that the daemons and applications are always using the same units. With the introduction of the ResourceInformation class we have support for units - so this function will continue to set memory but the assumption is that the value passed is in units of MB.
      Parameters:
      memory - memory(in MB) of the resource

    • setMemorySize

      @Public @Stable public void setMemorySize(long memory)
      Set memory of the resource.
      Parameters:
      memory - memory of the resource

    • getVirtualCores

      @Public @Evolving public abstract int getVirtualCores()
      Get number of virtual cpu cores of the resource. Virtual cores are a unit for expressing CPU parallelism. A node's capacity should be configured with virtual cores equal to its number of physical cores. A container should be requested with the number of cores it can saturate, i.e. the average number of threads it expects to have runnable at a time.
      Returns:
      num of virtual cpu cores of the resource

    • setVirtualCores

      @Public @Evolving public abstract void setVirtualCores(int vCores)
      Set number of virtual cpu cores of the resource. Virtual cores are a unit for expressing CPU parallelism. A node's capacity should be configured with virtual cores equal to its number of physical cores. A container should be requested with the number of cores it can saturate, i.e. the average number of threads it expects to have runnable at a time.
      Parameters:
      vCores - number of virtual cpu cores of the resource

    • getResources

      @Private @Unstable public org.apache.hadoop.yarn.api.records.ResourceInformation[] getResources()
      Get ResourceInformation for all resources.
      Returns:
      Map of resource name to ResourceInformation

    • getAllResourcesListCopy

      @Private @Unstable public List<org.apache.hadoop.yarn.api.records.ResourceInformation> getAllResourcesListCopy()
      Get list of resource information, this will be used by JAXB.
      Returns:
      list of resources copy.

    • getResourceInformation

      @Public @Unstable public org.apache.hadoop.yarn.api.records.ResourceInformation getResourceInformation(String resource)
      Get ResourceInformation for a specified resource.
      Parameters:
      resource - name of the resource
      Returns:
      the ResourceInformation object for the resource

    • getResourceInformation

      @Private @Unstable public org.apache.hadoop.yarn.api.records.ResourceInformation getResourceInformation(int index) throws ResourceNotFoundException
      Get ResourceInformation for a specified resource from a given index.
      Parameters:
      index - of the resource
      Returns:
      the ResourceInformation object for the resource
      Throws:
      ResourceNotFoundException - if the resource can't be found

    • getResourceValue

      @Public @Unstable public long getResourceValue(String resource)
      Get the value for a specified resource. No information about the units is returned.
      Parameters:
      resource - name of the resource
      Returns:
      the value for the resource

    • setResourceInformation

      @Public @Unstable public void setResourceInformation(String resource, org.apache.hadoop.yarn.api.records.ResourceInformation resourceInformation)
      Set the ResourceInformation object for a particular resource.
      Parameters:
      resource - the resource for which the ResourceInformation is provided
      resourceInformation - ResourceInformation object

    • setResourceInformation

      @Private @Unstable public void setResourceInformation(int index, org.apache.hadoop.yarn.api.records.ResourceInformation resourceInformation) throws ResourceNotFoundException
      Set the ResourceInformation object for a particular resource.
      Parameters:
      index - the resource index for which the ResourceInformation is provided
      resourceInformation - ResourceInformation object
      Throws:
      ResourceNotFoundException - if the resource is not found

    • setResourceValue

      @Public @Unstable public void setResourceValue(String resource, long value)
      Set the value of a resource in the ResourceInformation object. The unit of the value is assumed to be the one in the ResourceInformation object.
      Parameters:
      resource - the resource for which the value is provided.
      value - the value to set

    • setResourceValue

      @Private @Unstable public void setResourceValue(int index, long value) throws ResourceNotFoundException
      Set the value of a resource in the ResourceInformation object. The unit of the value is assumed to be the one in the ResourceInformation object.
      Parameters:
      index - the resource index for which the value is provided.
      value - the value to set
      Throws:
      ResourceNotFoundException - if the resource is not found

    • throwExceptionWhenArrayOutOfBound

      protected void throwExceptionWhenArrayOutOfBound(int index)

    • equals

      public boolean equals(Object obj)
      Overrides:
      equals in class Object

    • compareTo

      public int compareTo(Resource other)
      Specified by:
      compareTo in interface Comparable<Resource>

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • toFormattedString

      public String toFormattedString()

    • getFormattedString

      public String getFormattedString()
      This method is to get memory in terms of KB|MB|GB.
      Returns:
      string containing all resources

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • castToIntSafely

      protected static int castToIntSafely(long value)
      Convert long to int for a resource value safely. This method assumes resource value is positive.
      Parameters:
      value - long resource value
      Returns:
      int resource value

    • newDefaultInformation

      protected static org.apache.hadoop.yarn.api.records.ResourceInformation newDefaultInformation(String name, String unit, long value)
      Create ResourceInformation with basic fields.
      Parameters:
      name - Resource Type Name
      unit - Default unit of provided resource type
      value - Value associated with giveb resource
      Returns:
      ResourceInformation object

    • setResources

      @VisibleForTesting protected void setResources(org.apache.hadoop.yarn.api.records.ResourceInformation[] resources)

    • getFormattedString

      public String getFormattedString(long memory)