org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Class AbstractCSQueue

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

All Implemented Interfaces:

CSQueue, Queue

Direct Known Subclasses:

LeafQueue, ParentQueue

public abstract class AbstractCSQueue
extends Object
implements CSQueue

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.yarn.security.YarnAuthorizationProvider	authorizer
protected CapacitySchedulerContext	csContext
protected org.apache.hadoop.yarn.security.PrivilegedEntity	queueEntity

Constructor Summary

Constructors

Constructor and Description
AbstractCSQueue(CapacitySchedulerContext cs, String queueName, CSQueue parent, CSQueue old)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	accessibleToPartition(String nodePartition)
void	decPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec) When an outstanding resource is fulfilled or canceled, calling this will decrease pending resource in a queue.
void	decReservedResource(String partition, org.apache.hadoop.yarn.api.records.Resource reservedRes) Decrement Reserved Capacity
void	decUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec, SchedulerApplicationAttempt application) When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
float	getAbsoluteCapacity() Get capacity of the parent of the queue as a function of the cumulative capacity in the cluster.
float	getAbsoluteMaximumCapacity() Get maximum-capacity of the queue as a funciton of the cumulative capacity of the cluster.
float	getAbsoluteUsedCapacity() Get the current absolute used capacity of the queue relative to the entire cluster.
Set<String>	getAccessibleNodeLabels() Get labels can be accessed of this queue labels={*}, means this queue can access any label labels={ }, means this queue cannot access any label except node without label labels={a, b, c} means this queue can access a or b or c
Map<org.apache.hadoop.yarn.security.AccessType,org.apache.hadoop.security.authorize.AccessControlList>	getACLs()
float	getCapacity() Get the configured capacity of the queue.
org.apache.hadoop.yarn.api.records.Priority	getDefaultApplicationPriority() Get the Default Application Priority for this queue
String	getDefaultNodeLabelExpression() Get default label expression of this queue.
Iterator<RMContainer>	getKillableContainers(String partition)
org.apache.hadoop.yarn.api.records.Resource	getMaximumAllocation()
float	getMaximumCapacity() Get the configured maximum-capacity of the queue.
CSQueueMetrics	getMetrics() Get the queue metrics
org.apache.hadoop.yarn.api.records.Resource	getMinimumAllocation()
Set<String>	getNodeLabelsForQueue() Get valid Node Labels for this queue
int	getNumContainers()
CSQueue	getParent() Get the parent Queue.
boolean	getPreemptionDisabled() Check whether disable_preemption property is set for this queue
org.apache.hadoop.yarn.security.PrivilegedEntity	getPrivilegedEntity()
QueueCapacities	getQueueCapacities() Get QueueCapacities of this queue
protected org.apache.hadoop.yarn.api.records.QueueInfo	getQueueInfo()
String	getQueueName() Get the queue name.
String	getQueuePath() Get the full name of the queue, including the heirarchy.
ResourceUsage	getQueueResourceUsage() Get ResourceUsage of this queue
org.apache.hadoop.yarn.api.records.QueueStatistics	getQueueStatistics()
boolean	getReservationContinueLooking()
org.apache.hadoop.yarn.api.records.QueueState	getState() Get the current run-state of the queue
org.apache.hadoop.yarn.api.records.Resource	getTotalKillableResource(String partition)
float	getUsedCapacity() Get the current used capacity of nodes without label(s) of the queue and it's children (if any).
org.apache.hadoop.yarn.api.records.Resource	getUsedResources() Get the currently utilized resources which allocated at nodes without any labels in the cluster by the queue and children (if any).
Map<String,Float>	getUserWeights() Get a map of usernames and weights
boolean	hasAccess(org.apache.hadoop.yarn.api.records.QueueACL acl, org.apache.hadoop.security.UserGroupInformation user) Check if the user has permission to perform the operation
boolean	hasChildQueues()
void	incPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc) When new outstanding resource is asked, calling this will increase pending resource in a queue.
void	incReservedResource(String partition, org.apache.hadoop.yarn.api.records.Resource reservedRes) Increment Reserved Capacity
void	incUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc, SchedulerApplicationAttempt application) When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
protected void	releaseResource(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource resource, String nodePartition, boolean changeContainerResource)
void	setAbsoluteUsedCapacity(float absUsedCapacity) Set absolute used capacity of the queue.
void	setParent(CSQueue newParentQueue) Set the parent Queue.
protected void	setupConfigurableCapacities()
void	setUsedCapacity(float usedCapacity) Set used capacity of the queue.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue

assignContainers, attachContainer, collectSchedulerApplications, completedContainer, decreaseContainer, detachContainer, finishApplication, finishApplicationAttempt, getActiveUsersManager, getChildQueues, getNumApplications, reinitialize, submitApplication, submitApplicationAttempt, unreserveIncreasedContainer, updateClusterResource

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.Queue

getQueueInfo, getQueueUserAclInfo, recoverContainer

Field Detail

queueEntity

protected final org.apache.hadoop.yarn.security.PrivilegedEntity queueEntity

csContext

protected CapacitySchedulerContext csContext

authorizer

protected org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer

Constructor Detail

AbstractCSQueue

public AbstractCSQueue(CapacitySchedulerContext cs,
               String queueName,
               CSQueue parent,
               CSQueue old)
                throws IOException

Throws:

IOException

Method Detail

setupConfigurableCapacities

protected void setupConfigurableCapacities()

getQueuePath

public String getQueuePath()

Description copied from interface: CSQueue

Get the full name of the queue, including the heirarchy.

Specified by:

getQueuePath in interface CSQueue

Returns:

the full name of the queue

getCapacity

public float getCapacity()

Description copied from interface: CSQueue

Get the configured capacity of the queue.

Specified by:

getCapacity in interface CSQueue

Returns:

configured queue capacity

getAbsoluteCapacity

public float getAbsoluteCapacity()

Description copied from interface: CSQueue

Get capacity of the parent of the queue as a function of the cumulative capacity in the cluster.

Specified by:

getAbsoluteCapacity in interface CSQueue

Returns:

capacity of the parent of the queue as a function of the cumulative capacity in the cluster

getAbsoluteMaximumCapacity

public float getAbsoluteMaximumCapacity()

Description copied from interface: CSQueue

Get maximum-capacity of the queue as a funciton of the cumulative capacity of the cluster.

Specified by:

getAbsoluteMaximumCapacity in interface CSQueue

Returns:

maximum-capacity of the queue as a funciton of the cumulative capacity of the cluster

getAbsoluteUsedCapacity

public float getAbsoluteUsedCapacity()

Description copied from interface: CSQueue

Get the current absolute used capacity of the queue relative to the entire cluster.

Specified by:

getAbsoluteUsedCapacity in interface CSQueue

Returns:

queue absolute used capacity

getMaximumCapacity

public float getMaximumCapacity()

Description copied from interface: CSQueue

Get the configured maximum-capacity of the queue.

Specified by:

getMaximumCapacity in interface CSQueue

Returns:

the configured maximum-capacity of the queue

getUsedCapacity

public float getUsedCapacity()

Description copied from interface: CSQueue

Get the current used capacity of nodes without label(s) of the queue and it's children (if any).

Specified by:

getUsedCapacity in interface CSQueue

Returns:

queue used capacity

getUsedResources

public org.apache.hadoop.yarn.api.records.Resource getUsedResources()

Description copied from interface: CSQueue

Get the currently utilized resources which allocated at nodes without any labels in the cluster by the queue and children (if any).

Specified by:

getUsedResources in interface CSQueue

Returns:

used resources by the queue and it's children

getNumContainers

public int getNumContainers()

getState

public org.apache.hadoop.yarn.api.records.QueueState getState()

Description copied from interface: CSQueue

Get the current run-state of the queue

Specified by:

getState in interface CSQueue

Returns:

current run-state

getMetrics

public CSQueueMetrics getMetrics()

Description copied from interface: Queue

Get the queue metrics

Specified by:

getMetrics in interface Queue

Returns:

the queue metrics

getQueueName

public String getQueueName()

Description copied from interface: CSQueue

Get the queue name.

Specified by:

getQueueName in interface CSQueue

Specified by:

getQueueName in interface Queue

Returns:

the queue name

getPrivilegedEntity

public org.apache.hadoop.yarn.security.PrivilegedEntity getPrivilegedEntity()

getParent

public CSQueue getParent()

Description copied from interface: CSQueue

Get the parent Queue.

Specified by:

getParent in interface CSQueue

Returns:

the parent queue

setParent

public void setParent(CSQueue newParentQueue)

Description copied from interface: CSQueue

Set the parent Queue.

Specified by:

setParent in interface CSQueue

Parameters:

newParentQueue - new parent queue

getAccessibleNodeLabels

public Set<String> getAccessibleNodeLabels()

Description copied from interface: Queue

Get labels can be accessed of this queue labels={*}, means this queue can access any label labels={ }, means this queue cannot access any label except node without label labels={a, b, c} means this queue can access a or b or c

Specified by:

getAccessibleNodeLabels in interface Queue

Returns:

labels

hasAccess

public boolean hasAccess(org.apache.hadoop.yarn.api.records.QueueACL acl,
                org.apache.hadoop.security.UserGroupInformation user)

Description copied from interface: CSQueue

Check if the user has permission to perform the operation

Specified by:

hasAccess in interface CSQueue

Specified by:

hasAccess in interface Queue

Parameters:

acl - ACL

user - user

Returns:

true if the user has the permission, false otherwise

setUsedCapacity

public void setUsedCapacity(float usedCapacity)

Description copied from interface: CSQueue

Set used capacity of the queue.

Specified by:

setUsedCapacity in interface CSQueue

Parameters:

usedCapacity - used capacity of the queue

setAbsoluteUsedCapacity

public void setAbsoluteUsedCapacity(float absUsedCapacity)

Description copied from interface: CSQueue

Set absolute used capacity of the queue.

Specified by:

setAbsoluteUsedCapacity in interface CSQueue

Parameters:

absUsedCapacity - absolute used capacity of the queue

getDefaultNodeLabelExpression

public String getDefaultNodeLabelExpression()

Description copied from interface: Queue

Get default label expression of this queue. If label expression of ApplicationSubmissionContext and label expression of Resource Request not set, this will be used.

Specified by:

getDefaultNodeLabelExpression in interface Queue

Returns:

default label expression

getQueueInfo

protected org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo()

getQueueStatistics

public org.apache.hadoop.yarn.api.records.QueueStatistics getQueueStatistics()

getMaximumAllocation

@InterfaceAudience.Private
public org.apache.hadoop.yarn.api.records.Resource getMaximumAllocation()

getMinimumAllocation

@InterfaceAudience.Private
public org.apache.hadoop.yarn.api.records.Resource getMinimumAllocation()

releaseResource

protected void releaseResource(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                   org.apache.hadoop.yarn.api.records.Resource resource,
                   String nodePartition,
                   boolean changeContainerResource)

getReservationContinueLooking

@InterfaceAudience.Private
public boolean getReservationContinueLooking()

getACLs

@InterfaceAudience.Private
public Map<org.apache.hadoop.yarn.security.AccessType,org.apache.hadoop.security.authorize.AccessControlList> getACLs()

getPreemptionDisabled

@InterfaceAudience.Private
public boolean getPreemptionDisabled()

Description copied from interface: CSQueue

Check whether disable_preemption property is set for this queue

Specified by:

getPreemptionDisabled in interface CSQueue

Returns:

true if disable_preemption is set, false if not

getQueueCapacities

@InterfaceAudience.Private
public QueueCapacities getQueueCapacities()

Description copied from interface: CSQueue

Get QueueCapacities of this queue

Specified by:

getQueueCapacities in interface CSQueue

Returns:

queueCapacities

getQueueResourceUsage

@InterfaceAudience.Private
public ResourceUsage getQueueResourceUsage()

Description copied from interface: CSQueue

Get ResourceUsage of this queue

Specified by:

getQueueResourceUsage in interface CSQueue

Returns:

resourceUsage

hasChildQueues

public boolean hasChildQueues()

incReservedResource

public void incReservedResource(String partition,
                       org.apache.hadoop.yarn.api.records.Resource reservedRes)

Description copied from interface: Queue

Increment Reserved Capacity

Specified by:

incReservedResource in interface Queue

Parameters:

partition - asked by application

reservedRes - reserved resource asked

decReservedResource

public void decReservedResource(String partition,
                       org.apache.hadoop.yarn.api.records.Resource reservedRes)

Description copied from interface: Queue

Decrement Reserved Capacity

Specified by:

decReservedResource in interface Queue

Parameters:

partition - asked by application

reservedRes - reserved resource asked

incPendingResource

public void incPendingResource(String nodeLabel,
                      org.apache.hadoop.yarn.api.records.Resource resourceToInc)

Description copied from interface: Queue

When new outstanding resource is asked, calling this will increase pending resource in a queue.

Specified by:

incPendingResource in interface Queue

Parameters:

nodeLabel - asked by application

resourceToInc - new resource asked

decPendingResource

public void decPendingResource(String nodeLabel,
                      org.apache.hadoop.yarn.api.records.Resource resourceToDec)

Description copied from interface: CSQueue

When an outstanding resource is fulfilled or canceled, calling this will decrease pending resource in a queue.

Specified by:

decPendingResource in interface CSQueue

Specified by:

decPendingResource in interface Queue

Parameters:

nodeLabel - asked by application

resourceToDec - new resource asked

incUsedResource

public void incUsedResource(String nodeLabel,
                   org.apache.hadoop.yarn.api.records.Resource resourceToInc,
                   SchedulerApplicationAttempt application)

Description copied from interface: CSQueue

When partition of node updated, we will update queue's resource usage if it has container(s) running on that.

Specified by:

incUsedResource in interface CSQueue

decUsedResource

public void decUsedResource(String nodeLabel,
                   org.apache.hadoop.yarn.api.records.Resource resourceToDec,
                   SchedulerApplicationAttempt application)

Description copied from interface: CSQueue

When partition of node updated, we will update queue's resource usage if it has container(s) running on that.

Specified by:

decUsedResource in interface CSQueue

accessibleToPartition

public boolean accessibleToPartition(String nodePartition)

getDefaultApplicationPriority

public org.apache.hadoop.yarn.api.records.Priority getDefaultApplicationPriority()

Description copied from interface: Queue

Get the Default Application Priority for this queue

Specified by:

getDefaultApplicationPriority in interface Queue

Returns:

default application priority

getNodeLabelsForQueue

public Set<String> getNodeLabelsForQueue()

Description copied from interface: CSQueue

Get valid Node Labels for this queue

Specified by:

getNodeLabelsForQueue in interface CSQueue

Returns:

valid node labels

getTotalKillableResource

public org.apache.hadoop.yarn.api.records.Resource getTotalKillableResource(String partition)

getKillableContainers

public Iterator<RMContainer> getKillableContainers(String partition)

getUserWeights

public Map<String,Float> getUserWeights()

Description copied from interface: CSQueue

Get a map of usernames and weights

Specified by:

getUserWeights in interface CSQueue

Returns:

map of usernames and corresponding weight