org.apache.hadoop.yarn.server.resourcemanager

Class RMServerUtils

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils

public class RMServerUtils
extends Object

Utility methods to aid serving RM data through the REST and RPC APIs

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport	DUMMY_APPLICATION_RESOURCE_USAGE_REPORT Statically defined dummy ApplicationResourceUsageREport.
protected static org.apache.hadoop.yarn.factories.RecordFactory	RECORD_FACTORY
static String	RESOURCE_OUTSIDE_ALLOWED_RANGE
static String	UPDATE_OUTSTANDING_ERROR

Constructor Summary

Constructors

Constructor and Description
RMServerUtils()

Method Summary

Modifier and Type	Method and Description
static void	checkSchedContainerChangeRequest(SchedContainerChangeRequest request, boolean increase) Validate increase/decrease request.
static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState	convertRmAppAttemptStateToYarnApplicationAttemptState(RMAppAttemptState currentState, RMAppAttemptState previousState)
static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState	createApplicationAttemptState(RMAppAttemptState rmAppAttemptState)
static org.apache.hadoop.yarn.api.records.YarnApplicationState	createApplicationState(RMAppState rmAppState)
static int	getApplicableNodeCountForAM(RMContext rmContext, org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.yarn.api.records.ResourceRequest> amReqs) Get applicable Node count for AM.
static Long	getOrDefault(Map<String,Long> map, String key, Long defaultValue)
static void	normalizeAndValidateRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, String queueName, YarnScheduler scheduler, RMContext rmContext, boolean nodeLabelsEnabled) Utility method to validate a list resource requests, by ensuring that the requested memory/vcore is non-negative and not greater than max.
static void	processRMProxyUsersConf(org.apache.hadoop.conf.Configuration conf) Find all configs whose name starts with YarnConfiguration.RM_PROXY_USER_PREFIX, and add a record for each one by replacing the prefix with ProxyUsers.CONF_HADOOP_PROXYUSER.
static List<RMNode>	queryRMNodes(RMContext context, EnumSet<org.apache.hadoop.yarn.api.records.NodeState> acceptedStates)
static ContainerUpdates	validateAndSplitUpdateResourceRequests(RMContext rmContext, org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, List<org.apache.hadoop.yarn.api.records.UpdateContainerError> updateErrors) Check if we have: - Request for same containerId and different target resource.
static void	validateApplicationTimeouts(Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,Long> timeouts)
static void	validateBlacklistRequest(org.apache.hadoop.yarn.api.records.ResourceBlacklistRequest blacklistRequest)
static void	validateContainerReleaseRequest(List<org.apache.hadoop.yarn.api.records.ContainerId> containerReleaseList, org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId) It will validate to make sure all the containers belong to correct application attempt id.
static Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,Long>	validateISO8601AndConvertToLocalTimeEpoch(Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,String> timeoutsInISO8601) Validate ISO8601 format with epoch time.
static org.apache.hadoop.security.UserGroupInformation	verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer, String method, org.slf4j.Logger LOG)
static org.apache.hadoop.security.UserGroupInformation	verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer, String method, String module, org.slf4j.Logger LOG) Utility method to verify if the current user has access based on the passed AccessControlList

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UPDATE_OUTSTANDING_ERROR

public static final String UPDATE_OUTSTANDING_ERROR

See Also:

Constant Field Values

RESOURCE_OUTSIDE_ALLOWED_RANGE

public static final String RESOURCE_OUTSIDE_ALLOWED_RANGE

See Also:

Constant Field Values

RECORD_FACTORY

protected static final org.apache.hadoop.yarn.factories.RecordFactory RECORD_FACTORY

DUMMY_APPLICATION_RESOURCE_USAGE_REPORT

public static final org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport DUMMY_APPLICATION_RESOURCE_USAGE_REPORT

Statically defined dummy ApplicationResourceUsageREport. Used as a return value when a valid report cannot be found.

Constructor Detail

RMServerUtils

public RMServerUtils()

Method Detail

queryRMNodes

public static List<RMNode> queryRMNodes(RMContext context,
                                        EnumSet<org.apache.hadoop.yarn.api.records.NodeState> acceptedStates)

validateAndSplitUpdateResourceRequests

public static ContainerUpdates validateAndSplitUpdateResourceRequests(RMContext rmContext,
                                                                      org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request,
                                                                      org.apache.hadoop.yarn.api.records.Resource maximumAllocation,
                                                                      List<org.apache.hadoop.yarn.api.records.UpdateContainerError> updateErrors)

Check if we have: - Request for same containerId and different target resource. - If targetResources violates maximum/minimumAllocation.

Parameters:

rmContext - RM context.

request - Allocate Request.

maximumAllocation - Maximum Allocation.

updateErrors - Container update errors.

Returns:

ContainerUpdateRequests.

normalizeAndValidateRequests

public static void normalizeAndValidateRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask,
                                                org.apache.hadoop.yarn.api.records.Resource maximumAllocation,
                                                String queueName,
                                                YarnScheduler scheduler,
                                                RMContext rmContext,
                                                boolean nodeLabelsEnabled)
                                         throws org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

Utility method to validate a list resource requests, by ensuring that the requested memory/vcore is non-negative and not greater than max.

Parameters:

ask - resource request.

maximumAllocation - Maximum Allocation.

queueName - queue name.

scheduler - YarnScheduler.

rmContext - RMContext.

nodeLabelsEnabled - the node labels feature enabled.

Throws:

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException - when there is invalid request.

checkSchedContainerChangeRequest

public static void checkSchedContainerChangeRequest(SchedContainerChangeRequest request,
                                                    boolean increase)
                                             throws org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

Validate increase/decrease request.

 - Throw exception when any other error happens
 

Parameters:

request - SchedContainerChangeRequest.

increase - true, add container; false, decrease container.

Throws:

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException - when there is invalid request.

validateBlacklistRequest

public static void validateBlacklistRequest(org.apache.hadoop.yarn.api.records.ResourceBlacklistRequest blacklistRequest)
                                     throws org.apache.hadoop.yarn.exceptions.InvalidResourceBlacklistRequestException

Throws:

org.apache.hadoop.yarn.exceptions.InvalidResourceBlacklistRequestException

validateContainerReleaseRequest

public static void validateContainerReleaseRequest(List<org.apache.hadoop.yarn.api.records.ContainerId> containerReleaseList,
                                                   org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
                                            throws org.apache.hadoop.yarn.exceptions.InvalidContainerReleaseException

It will validate to make sure all the containers belong to correct application attempt id. If not then it will throw InvalidContainerReleaseException

Parameters:

containerReleaseList - containers to be released as requested by application master.

appAttemptId - Application attempt Id

Throws:

org.apache.hadoop.yarn.exceptions.InvalidContainerReleaseException - an Application Master tries to release containers not belonging to it using.

verifyAdminAccess

public static org.apache.hadoop.security.UserGroupInformation verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer,
                                                                                String method,
                                                                                org.slf4j.Logger LOG)
                                                                         throws IOException

Throws:

IOException

verifyAdminAccess

public static org.apache.hadoop.security.UserGroupInformation verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer,
                                                                                String method,
                                                                                String module,
                                                                                org.slf4j.Logger LOG)
                                                                         throws IOException

Utility method to verify if the current user has access based on the passed AccessControlList

Parameters:

authorizer - the AccessControlList to check against

method - the method name to be logged

module - like AdminService or NodeLabelManager

LOG - the logger to use

Returns:

UserGroupInformation of the current user

Throws:

IOException - an I/O exception has occurred.

createApplicationState

public static org.apache.hadoop.yarn.api.records.YarnApplicationState createApplicationState(RMAppState rmAppState)

convertRmAppAttemptStateToYarnApplicationAttemptState

public static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState convertRmAppAttemptStateToYarnApplicationAttemptState(RMAppAttemptState currentState,
                                                                                                                                   RMAppAttemptState previousState)

createApplicationAttemptState

public static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState createApplicationAttemptState(RMAppAttemptState rmAppAttemptState)

processRMProxyUsersConf

public static void processRMProxyUsersConf(org.apache.hadoop.conf.Configuration conf)

Find all configs whose name starts with YarnConfiguration.RM_PROXY_USER_PREFIX, and add a record for each one by replacing the prefix with ProxyUsers.CONF_HADOOP_PROXYUSER.

Parameters:

conf - Configuration.

validateApplicationTimeouts

public static void validateApplicationTimeouts(Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,Long> timeouts)
                                        throws org.apache.hadoop.yarn.exceptions.YarnException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

validateISO8601AndConvertToLocalTimeEpoch

public static Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,Long> validateISO8601AndConvertToLocalTimeEpoch(Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,String> timeoutsInISO8601)
                                                                                                                     throws org.apache.hadoop.yarn.exceptions.YarnException

Validate ISO8601 format with epoch time.

Parameters:

timeoutsInISO8601 - format

Returns:

expire time in local epoch

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if given application timeout value is lesser than current time.

getApplicableNodeCountForAM

public static int getApplicableNodeCountForAM(RMContext rmContext,
                                              org.apache.hadoop.conf.Configuration conf,
                                              List<org.apache.hadoop.yarn.api.records.ResourceRequest> amReqs)

Get applicable Node count for AM.

Parameters:

rmContext - context

conf - configuration

amReqs - am resource requests

Returns:

applicable node count

getOrDefault

public static Long getOrDefault(Map<String,Long> map,
                                String key,
                                Long defaultValue)