org.apache.hadoop.mapreduce.v2.util

Class MRApps

java.lang.Object

org.apache.hadoop.yarn.util.Apps

org.apache.hadoop.mapreduce.v2.util.MRApps

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class MRApps
extends org.apache.hadoop.yarn.util.Apps

Helper class for MR applications

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MRApps.TaskAttemptStateUI
static class	MRApps.TaskStateUI

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.yarn.util.Apps

APP, ID

Constructor Summary

Constructors

Constructor and Description
MRApps()

Method Summary

Modifier and Type	Method and Description
static void	addLog4jSystemProperties(org.apache.hadoop.mapred.Task task, List<String> vargs, org.apache.hadoop.conf.Configuration conf) Add the JVM system properties necessary to configure ContainerLogAppender or ContainerRollingLogAppender.
static void	addToEnvironment(Map<String,String> environment, String variable, String value, org.apache.hadoop.conf.Configuration conf)
static ClassLoader	createJobClassLoader(org.apache.hadoop.conf.Configuration conf) Creates a ApplicationClassLoader if MRJobConfig.MAPREDUCE_JOB_CLASSLOADER is set to true, and the APP_CLASSPATH environment variable is set.
static String	crossPlatformifyMREnv(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.ApplicationConstants.Environment env)
static String	getChildLogLevel(org.apache.hadoop.conf.Configuration conf, boolean isMap)
static org.apache.hadoop.fs.Path	getEndJobCommitFailureFile(org.apache.hadoop.conf.Configuration conf, String user, JobId jobId)
static org.apache.hadoop.fs.Path	getEndJobCommitSuccessFile(org.apache.hadoop.conf.Configuration conf, String user, JobId jobId)
static String	getJobFile(org.apache.hadoop.conf.Configuration conf, String user, org.apache.hadoop.mapreduce.JobID jobId)
static org.apache.hadoop.fs.Path	getStagingAreaDir(org.apache.hadoop.conf.Configuration conf, String user)
static org.apache.hadoop.fs.Path	getStartJobCommitFile(org.apache.hadoop.conf.Configuration conf, String user, JobId jobId)
static String	getSystemPropertiesToLog(org.apache.hadoop.conf.Configuration conf) Return lines for system property keys and values per configuration.
static void	setClassLoader(ClassLoader classLoader, org.apache.hadoop.conf.Configuration conf) Sets the provided classloader on the given configuration and as the thread context classloader if the classloader is not null.
static void	setClasspath(Map<String,String> environment, org.apache.hadoop.conf.Configuration conf)
static void	setEnvFromInputProperty(Map<String,String> env, String propName, String defaultPropValue, org.apache.hadoop.conf.Configuration conf)
static void	setEnvFromInputString(Map<String,String> env, String envString, org.apache.hadoop.conf.Configuration conf)
static void	setJobClassLoader(org.apache.hadoop.conf.Configuration conf) Creates and sets a ApplicationClassLoader on the given configuration and as the thread context classloader, if MRJobConfig.MAPREDUCE_JOB_CLASSLOADER is set to true, and the APP_CLASSPATH environment variable is set.
static void	setupDistributedCache(org.apache.hadoop.conf.Configuration conf, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
static void	setupDistributedCacheLocal(org.apache.hadoop.conf.Configuration conf) Set up the DistributedCache related configs to make JobContextImpl.getLocalCacheFiles(Configuration) and JobContextImpl.getLocalCacheArchives(Configuration) working.
static MRApps.TaskAttemptStateUI	taskAttemptState(String attemptStateStr)
static MRApps.TaskStateUI	taskState(String taskStateStr)
static String	taskSymbol(TaskType type)
static TaskType	taskType(String symbol)
static JobId	toJobID(String jid)
static String	toString(JobId jid)
static String	toString(TaskAttemptId taid)
static String	toString(TaskId tid)
static TaskAttemptId	toTaskAttemptID(String taid)
static TaskId	toTaskID(String tid)

Methods inherited from class org.apache.hadoop.yarn.util.Apps

addToEnvironment, addToEnvironment, crossPlatformify, getEnvVarsFromInputProperty, isApplicationFinalState, setEnvFromInputProperty, setEnvFromInputString, setEnvFromInputString, shouldCountTowardsNodeBlacklisting, shouldHaveNext, throwParseException, toAppID, toAppID

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

MRApps

public MRApps()

Method Detail

toString

public static String toString(JobId jid)

toJobID

public static JobId toJobID(String jid)

toString

public static String toString(TaskId tid)

toTaskID

public static TaskId toTaskID(String tid)

toString

public static String toString(TaskAttemptId taid)

toTaskAttemptID

public static TaskAttemptId toTaskAttemptID(String taid)

taskSymbol

public static String taskSymbol(TaskType type)

taskType

public static TaskType taskType(String symbol)

taskAttemptState

public static MRApps.TaskAttemptStateUI taskAttemptState(String attemptStateStr)

taskState

public static MRApps.TaskStateUI taskState(String taskStateStr)

setClasspath

public static void setClasspath(Map<String,String> environment,
                                org.apache.hadoop.conf.Configuration conf)
                         throws IOException

Throws:

IOException

setJobClassLoader

public static void setJobClassLoader(org.apache.hadoop.conf.Configuration conf)
                              throws IOException

Creates and sets a ApplicationClassLoader on the given configuration and as the thread context classloader, if MRJobConfig.MAPREDUCE_JOB_CLASSLOADER is set to true, and the APP_CLASSPATH environment variable is set.

Parameters:

conf -

Throws:

IOException

createJobClassLoader

public static ClassLoader createJobClassLoader(org.apache.hadoop.conf.Configuration conf)
                                        throws IOException

Creates a ApplicationClassLoader if MRJobConfig.MAPREDUCE_JOB_CLASSLOADER is set to true, and the APP_CLASSPATH environment variable is set.

Parameters:

conf -

Returns:

the created job classloader, or null if the job classloader is not enabled or the APP_CLASSPATH environment variable is not set

Throws:

IOException

setClassLoader

public static void setClassLoader(ClassLoader classLoader,
                                  org.apache.hadoop.conf.Configuration conf)

Sets the provided classloader on the given configuration and as the thread context classloader if the classloader is not null.

Parameters:

classLoader -

conf -

getStagingAreaDir

public static org.apache.hadoop.fs.Path getStagingAreaDir(org.apache.hadoop.conf.Configuration conf,
                                                          String user)

getJobFile

public static String getJobFile(org.apache.hadoop.conf.Configuration conf,
                                String user,
                                org.apache.hadoop.mapreduce.JobID jobId)

getEndJobCommitSuccessFile

public static org.apache.hadoop.fs.Path getEndJobCommitSuccessFile(org.apache.hadoop.conf.Configuration conf,
                                                                   String user,
                                                                   JobId jobId)

getEndJobCommitFailureFile

public static org.apache.hadoop.fs.Path getEndJobCommitFailureFile(org.apache.hadoop.conf.Configuration conf,
                                                                   String user,
                                                                   JobId jobId)

getStartJobCommitFile

public static org.apache.hadoop.fs.Path getStartJobCommitFile(org.apache.hadoop.conf.Configuration conf,
                                                              String user,
                                                              JobId jobId)

setupDistributedCache

public static void setupDistributedCache(org.apache.hadoop.conf.Configuration conf,
                                         Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
                                  throws IOException

Throws:

IOException

setupDistributedCacheLocal

public static void setupDistributedCacheLocal(org.apache.hadoop.conf.Configuration conf)
                                       throws IOException

Set up the DistributedCache related configs to make JobContextImpl.getLocalCacheFiles(Configuration) and JobContextImpl.getLocalCacheArchives(Configuration) working.

Parameters:

conf -

Throws:

IOException

getChildLogLevel

public static String getChildLogLevel(org.apache.hadoop.conf.Configuration conf,
                                      boolean isMap)

addLog4jSystemProperties

public static void addLog4jSystemProperties(org.apache.hadoop.mapred.Task task,
                                            List<String> vargs,
                                            org.apache.hadoop.conf.Configuration conf)

Add the JVM system properties necessary to configure ContainerLogAppender or ContainerRollingLogAppender.

Parameters:

task - for map/reduce, or null for app master

vargs - the argument list to append to

conf - configuration of MR job

getSystemPropertiesToLog

public static String getSystemPropertiesToLog(org.apache.hadoop.conf.Configuration conf)

Return lines for system property keys and values per configuration.

Returns:

the formatted string for the system property lines or null if no properties are specified.

setEnvFromInputString

public static void setEnvFromInputString(Map<String,String> env,
                                         String envString,
                                         org.apache.hadoop.conf.Configuration conf)

setEnvFromInputProperty

public static void setEnvFromInputProperty(Map<String,String> env,
                                           String propName,
                                           String defaultPropValue,
                                           org.apache.hadoop.conf.Configuration conf)

addToEnvironment

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public static void addToEnvironment(Map<String,String> environment,
                                                                                            String variable,
                                                                                            String value,
                                                                                            org.apache.hadoop.conf.Configuration conf)

crossPlatformifyMREnv

public static String crossPlatformifyMREnv(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.yarn.api.ApplicationConstants.Environment env)