org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair

Class FSQueue

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue

All Implemented Interfaces:

Schedulable, Queue

Direct Known Subclasses:

FSLeafQueue, FSParentQueue

@InterfaceAudience.Private
@InterfaceStability.Unstable
public abstract class FSQueue
extends Object
implements Queue, Schedulable

Field Summary

Fields

Modifier and Type	Field and Description
protected FSParentQueue	parent
protected SchedulingPolicy	policy
protected org.apache.hadoop.yarn.factories.RecordFactory	recordFactory
protected FairScheduler	scheduler

Constructor Summary

Constructors

Constructor and Description
FSQueue(String name, FairScheduler scheduler, FSParentQueue parent)

Method Summary

Methods

Modifier and Type	Method and Description
protected boolean	assignContainerPreCheck(FSSchedulerNode node) Helper method to check if the queue should attempt assigning resources
boolean	canBePreempted() Recursively check if the queue can be preempted based on whether the resource usage is greater than fair share.
abstract void	collectSchedulerApplications(Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps) Adds all applications in the queue and its subqueues to the given collection.
void	decPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec) When an outstanding resource is fulfilled or canceled, calling this will decrease pending resource in a queue.
void	decReservedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec) Decrement Reserved Capacity
boolean	fitsInMaxShare(org.apache.hadoop.yarn.api.records.Resource additionalResource)
Set<String>	getAccessibleNodeLabels() Get labels can be accessed of this queue labels={*}, means this queue can access any label labels={ }, means this queue cannot access any label except node without label labels={a, b, c} means this queue can access a or b or c
abstract List<FSQueue>	getChildQueues() Gets the children of this queue, if any.
org.apache.hadoop.yarn.api.records.Priority	getDefaultApplicationPriority() Get the Default Application Priority for this queue
String	getDefaultNodeLabelExpression() Get default label expression of this queue.
org.apache.hadoop.yarn.api.records.Resource	getFairShare() Get the fair share assigned to this Schedulable.
float	getFairSharePreemptionThreshold()
long	getFairSharePreemptionTimeout()
org.apache.hadoop.yarn.api.records.Resource	getMaxShare() Maximum Resource share assigned to the schedulable.
FSQueueMetrics	getMetrics() Get the queue metrics
org.apache.hadoop.yarn.api.records.Resource	getMinShare() Minimum Resource share assigned to the schedulable.
long	getMinSharePreemptionTimeout()
String	getName() Name of job/queue, used for debugging as well as for breaking ties in scheduling order deterministically.
abstract int	getNumRunnableApps() Return the number of apps for which containers can be allocated.
FSParentQueue	getParent()
SchedulingPolicy	getPolicy()
org.apache.hadoop.yarn.api.records.Priority	getPriority() Job priority for jobs in FIFO queues; meaningless for QueueSchedulables.
org.apache.hadoop.yarn.api.records.QueueInfo	getQueueInfo(boolean includeChildQueues, boolean recursive) Get queue information
String	getQueueName() Get the queue name
org.apache.hadoop.yarn.api.records.QueueStatistics	getQueueStatistics()
long	getStartTime() Start time for jobs in FIFO queues; meaningless for QueueSchedulables.
org.apache.hadoop.yarn.api.records.Resource	getSteadyFairShare() Get the steady fair share assigned to this Schedulable.
ResourceWeights	getWeights() Job/queue weight in fair sharing.
boolean	hasAccess(org.apache.hadoop.yarn.api.records.QueueACL acl, org.apache.hadoop.security.UserGroupInformation user)
void	incPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc) When new outstanding resource is asked, calling this will increase pending resource in a queue.
void	incReservedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc) Increment Reserved Capacity
boolean	isActive() Returns true if queue has at least one app running.
abstract void	recomputeShares() Recomputes the shares for all child queues and applications based on this queue's current share
void	setFairShare(org.apache.hadoop.yarn.api.records.Resource fairShare) Assign a fair share to this Schedulable.
void	setFairSharePreemptionThreshold(float fairSharePreemptionThreshold)
void	setFairSharePreemptionTimeout(long fairSharePreemptionTimeout)
void	setMinSharePreemptionTimeout(long minSharePreemptionTimeout)
abstract void	setPolicy(SchedulingPolicy policy)
void	setSteadyFairShare(org.apache.hadoop.yarn.api.records.Resource steadyFairShare)
protected void	throwPolicyDoesnotApplyException(SchedulingPolicy policy)
String	toString() Convenient toString implementation for debugging.
void	updatePreemptionVariables() Update the min/fair share preemption timeouts and threshold for this queue.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.Queue

getActiveUsersManager, getQueueUserAclInfo, recoverContainer

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.Schedulable

assignContainer, getDemand, getResourceUsage, preemptContainer, updateDemand

Field Detail

scheduler

protected final FairScheduler scheduler

parent

protected final FSParentQueue parent

recordFactory

protected final org.apache.hadoop.yarn.factories.RecordFactory recordFactory

policy

protected SchedulingPolicy policy

Constructor Detail

FSQueue

public FSQueue(String name,
       FairScheduler scheduler,
       FSParentQueue parent)

Method Detail

getName

public String getName()

Description copied from interface: Schedulable

Name of job/queue, used for debugging as well as for breaking ties in scheduling order deterministically.

Specified by:

getName in interface Schedulable

getQueueName

public String getQueueName()

Description copied from interface: Queue

Get the queue name

Specified by:

getQueueName in interface Queue

Returns:

queue name

getPolicy

public SchedulingPolicy getPolicy()

getParent

public FSParentQueue getParent()

throwPolicyDoesnotApplyException

protected void throwPolicyDoesnotApplyException(SchedulingPolicy policy)
                                         throws AllocationConfigurationException

Throws:

AllocationConfigurationException

setPolicy

public abstract void setPolicy(SchedulingPolicy policy)
                        throws AllocationConfigurationException

Throws:

AllocationConfigurationException

getWeights

public ResourceWeights getWeights()

Description copied from interface: Schedulable

Job/queue weight in fair sharing.

Specified by:

getWeights in interface Schedulable

getMinShare

public org.apache.hadoop.yarn.api.records.Resource getMinShare()

Description copied from interface: Schedulable

Minimum Resource share assigned to the schedulable.

Specified by:

getMinShare in interface Schedulable

getMaxShare

public org.apache.hadoop.yarn.api.records.Resource getMaxShare()

Description copied from interface: Schedulable

Maximum Resource share assigned to the schedulable.

Specified by:

getMaxShare in interface Schedulable

getStartTime

public long getStartTime()

Description copied from interface: Schedulable

Start time for jobs in FIFO queues; meaningless for QueueSchedulables.

Specified by:

getStartTime in interface Schedulable

getPriority

public org.apache.hadoop.yarn.api.records.Priority getPriority()

Description copied from interface: Schedulable

Job priority for jobs in FIFO queues; meaningless for QueueSchedulables.

Specified by:

getPriority in interface Schedulable

getQueueInfo

public org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(boolean includeChildQueues,
                                                        boolean recursive)

Description copied from interface: Queue

Get queue information

Specified by:

getQueueInfo in interface Queue

Parameters:

includeChildQueues - include child queues?

recursive - recursively get child queue information?

Returns:

queue information

getQueueStatistics

public org.apache.hadoop.yarn.api.records.QueueStatistics getQueueStatistics()

getMetrics

public FSQueueMetrics getMetrics()

Description copied from interface: Queue

Get the queue metrics

Specified by:

getMetrics in interface Queue

Returns:

the queue metrics

getFairShare

public org.apache.hadoop.yarn.api.records.Resource getFairShare()

Get the fair share assigned to this Schedulable.

Specified by:

getFairShare in interface Schedulable

setFairShare

public void setFairShare(org.apache.hadoop.yarn.api.records.Resource fairShare)

Description copied from interface: Schedulable

Assign a fair share to this Schedulable.

Specified by:

setFairShare in interface Schedulable

getSteadyFairShare

public org.apache.hadoop.yarn.api.records.Resource getSteadyFairShare()

Get the steady fair share assigned to this Schedulable.

setSteadyFairShare

public void setSteadyFairShare(org.apache.hadoop.yarn.api.records.Resource steadyFairShare)

hasAccess

public boolean hasAccess(org.apache.hadoop.yarn.api.records.QueueACL acl,
                org.apache.hadoop.security.UserGroupInformation user)

Specified by:

hasAccess in interface Queue

getFairSharePreemptionTimeout

public long getFairSharePreemptionTimeout()

setFairSharePreemptionTimeout

public void setFairSharePreemptionTimeout(long fairSharePreemptionTimeout)

getMinSharePreemptionTimeout

public long getMinSharePreemptionTimeout()

setMinSharePreemptionTimeout

public void setMinSharePreemptionTimeout(long minSharePreemptionTimeout)

getFairSharePreemptionThreshold

public float getFairSharePreemptionThreshold()

setFairSharePreemptionThreshold

public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold)

canBePreempted

public boolean canBePreempted()

Recursively check if the queue can be preempted based on whether the resource usage is greater than fair share.

Returns:

true if the queue can be preempted

recomputeShares

public abstract void recomputeShares()

Recomputes the shares for all child queues and applications based on this queue's current share

updatePreemptionVariables

public void updatePreemptionVariables()

Update the min/fair share preemption timeouts and threshold for this queue.

getChildQueues

public abstract List<FSQueue> getChildQueues()

Gets the children of this queue, if any.

collectSchedulerApplications

public abstract void collectSchedulerApplications(Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps)

Adds all applications in the queue and its subqueues to the given collection.

Parameters:

apps - the collection to add the applications to

getNumRunnableApps

public abstract int getNumRunnableApps()

Return the number of apps for which containers can be allocated. Includes apps in subqueues.

assignContainerPreCheck

protected boolean assignContainerPreCheck(FSSchedulerNode node)

Helper method to check if the queue should attempt assigning resources

Returns:

true if check passes (can assign) or false otherwise

isActive

public boolean isActive()

Returns true if queue has at least one app running.

toString

public String toString()

Convenient toString implementation for debugging.

Overrides:

toString in class Object

getAccessibleNodeLabels

public Set<String> getAccessibleNodeLabels()

Description copied from interface: Queue

Get labels can be accessed of this queue labels={*}, means this queue can access any label labels={ }, means this queue cannot access any label except node without label labels={a, b, c} means this queue can access a or b or c

Specified by:

getAccessibleNodeLabels in interface Queue

Returns:

labels

getDefaultNodeLabelExpression

public String getDefaultNodeLabelExpression()

Description copied from interface: Queue

Get default label expression of this queue. If label expression of ApplicationSubmissionContext and label expression of Resource Request not set, this will be used.

Specified by:

getDefaultNodeLabelExpression in interface Queue

Returns:

default label expression

incPendingResource

public void incPendingResource(String nodeLabel,
                      org.apache.hadoop.yarn.api.records.Resource resourceToInc)

Description copied from interface: Queue

When new outstanding resource is asked, calling this will increase pending resource in a queue.

Specified by:

incPendingResource in interface Queue

Parameters:

nodeLabel - asked by application

resourceToInc - new resource asked

decPendingResource

public void decPendingResource(String nodeLabel,
                      org.apache.hadoop.yarn.api.records.Resource resourceToDec)

Description copied from interface: Queue

When an outstanding resource is fulfilled or canceled, calling this will decrease pending resource in a queue.

Specified by:

decPendingResource in interface Queue

Parameters:

nodeLabel - asked by application

resourceToDec - new resource asked

incReservedResource

public void incReservedResource(String nodeLabel,
                       org.apache.hadoop.yarn.api.records.Resource resourceToInc)

Description copied from interface: Queue

Increment Reserved Capacity

Specified by:

incReservedResource in interface Queue

Parameters:

nodeLabel - asked by application

resourceToInc - reserved resource asked

decReservedResource

public void decReservedResource(String nodeLabel,
                       org.apache.hadoop.yarn.api.records.Resource resourceToDec)

Description copied from interface: Queue

Decrement Reserved Capacity

Specified by:

decReservedResource in interface Queue

Parameters:

nodeLabel - asked by application

resourceToDec - reserved resource asked

getDefaultApplicationPriority

public org.apache.hadoop.yarn.api.records.Priority getDefaultApplicationPriority()

Description copied from interface: Queue

Get the Default Application Priority for this queue

Specified by:

getDefaultApplicationPriority in interface Queue

Returns:

default application priority

fitsInMaxShare

public boolean fitsInMaxShare(org.apache.hadoop.yarn.api.records.Resource additionalResource)