org.apache.hadoop.mapred

Class YARNRunner

java.lang.Object

org.apache.hadoop.mapred.YARNRunner

All Implemented Interfaces:

org.apache.hadoop.ipc.VersionedProtocol, org.apache.hadoop.mapreduce.protocol.ClientProtocol

public class YARNRunner
extends Object
implements org.apache.hadoop.mapreduce.protocol.ClientProtocol

This class enables the current JobClient (0.22 hadoop) to run on YARN.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.yarn.api.records.Priority	AM_CONTAINER_PRIORITY

Fields inherited from interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

versionID

Constructor Summary

Constructors

Constructor and Description
YARNRunner(org.apache.hadoop.conf.Configuration conf) Yarn runner incapsulates the client interface of yarn
YARNRunner(org.apache.hadoop.conf.Configuration conf, ResourceMgrDelegate resMgrDelegate) Similar to YARNRunner(Configuration) but allowing injecting ResourceMgrDelegate.
YARNRunner(org.apache.hadoop.conf.Configuration conf, ResourceMgrDelegate resMgrDelegate, ClientCache clientCache) Similar to YARNRunner(Configuration, ResourceMgrDelegate) but allowing injecting ClientCache.

Method Summary

Modifier and Type	Method and Description
void	cancelDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
void	close()
org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext	createApplicationSubmissionContext(org.apache.hadoop.conf.Configuration jobConf, String jobSubmitDir, org.apache.hadoop.security.Credentials ts) Constructs all the necessary information to start the MR AM.
org.apache.hadoop.mapreduce.TaskTrackerInfo[]	getActiveTrackers()
org.apache.hadoop.mapreduce.JobStatus[]	getAllJobs()
org.apache.hadoop.mapreduce.TaskTrackerInfo[]	getBlacklistedTrackers()
org.apache.hadoop.mapreduce.QueueInfo[]	getChildQueues(String parent)
org.apache.hadoop.mapreduce.ClusterMetrics	getClusterMetrics()
org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier>	getDelegationToken(org.apache.hadoop.io.Text renewer)
String	getFilesystemName()
org.apache.hadoop.mapreduce.Counters	getJobCounters(org.apache.hadoop.mapreduce.JobID arg0)
String	getJobHistoryDir()
org.apache.hadoop.mapreduce.JobStatus	getJobStatus(org.apache.hadoop.mapreduce.JobID jobID)
org.apache.hadoop.mapreduce.Cluster.JobTrackerStatus	getJobTrackerStatus()
org.apache.hadoop.mapreduce.v2.LogParams	getLogFileParams(org.apache.hadoop.mapreduce.JobID jobID, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID)
org.apache.hadoop.mapreduce.JobID	getNewJobID()
org.apache.hadoop.ipc.ProtocolSignature	getProtocolSignature(String protocol, long clientVersion, int clientMethodsHash)
long	getProtocolVersion(String arg0, long arg1)
org.apache.hadoop.mapreduce.QueueInfo	getQueue(String queueName)
org.apache.hadoop.mapreduce.QueueAclsInfo[]	getQueueAclsForCurrentUser()
org.apache.hadoop.security.authorize.AccessControlList	getQueueAdmins(String arg0)
org.apache.hadoop.mapreduce.QueueInfo[]	getQueues()
org.apache.hadoop.mapreduce.QueueInfo[]	getRootQueues()
String	getStagingAreaDir()
String	getSystemDir()
org.apache.hadoop.mapreduce.TaskCompletionEvent[]	getTaskCompletionEvents(org.apache.hadoop.mapreduce.JobID arg0, int arg1, int arg2)
String[]	getTaskDiagnostics(org.apache.hadoop.mapreduce.TaskAttemptID arg0)
org.apache.hadoop.mapreduce.TaskReport[]	getTaskReports(org.apache.hadoop.mapreduce.JobID jobID, org.apache.hadoop.mapreduce.TaskType taskType)
long	getTaskTrackerExpiryInterval()
void	killJob(org.apache.hadoop.mapreduce.JobID arg0)
boolean	killTask(org.apache.hadoop.mapreduce.TaskAttemptID arg0, boolean arg1)
long	renewDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
void	setJobPriority(org.apache.hadoop.mapreduce.JobID arg0, String arg1)
void	setResourceMgrDelegate(ResourceMgrDelegate resMgrDelegate)
org.apache.hadoop.mapreduce.JobStatus	submitJob(org.apache.hadoop.mapreduce.JobID jobId, String jobSubmitDir, org.apache.hadoop.security.Credentials ts)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

AM_CONTAINER_PRIORITY

public static final org.apache.hadoop.yarn.api.records.Priority AM_CONTAINER_PRIORITY

Constructor Detail

YARNRunner

public YARNRunner(org.apache.hadoop.conf.Configuration conf)

Yarn runner incapsulates the client interface of yarn

Parameters:

conf - the configuration object for the client

YARNRunner

public YARNRunner(org.apache.hadoop.conf.Configuration conf,
                  ResourceMgrDelegate resMgrDelegate)

Similar to YARNRunner(Configuration) but allowing injecting ResourceMgrDelegate. Enables mocking and testing.

Parameters:

conf - the configuration object for the client

resMgrDelegate - the resourcemanager client handle.

YARNRunner

public YARNRunner(org.apache.hadoop.conf.Configuration conf,
                  ResourceMgrDelegate resMgrDelegate,
                  ClientCache clientCache)

Similar to YARNRunner(Configuration, ResourceMgrDelegate) but allowing injecting ClientCache. Enable mocking and testing.

Parameters:

conf - the configuration object

resMgrDelegate - the resource manager delegate

clientCache - the client cache object.

Method Detail

setResourceMgrDelegate

@InterfaceAudience.Private
public void setResourceMgrDelegate(ResourceMgrDelegate resMgrDelegate)

cancelDelegationToken

public void cancelDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
                           throws IOException,
                                  InterruptedException

Specified by:

cancelDelegationToken in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getActiveTrackers

public org.apache.hadoop.mapreduce.TaskTrackerInfo[] getActiveTrackers()
                                                                throws IOException,
                                                                       InterruptedException

Specified by:

getActiveTrackers in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getAllJobs

public org.apache.hadoop.mapreduce.JobStatus[] getAllJobs()
                                                   throws IOException,
                                                          InterruptedException

Specified by:

getAllJobs in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getBlacklistedTrackers

public org.apache.hadoop.mapreduce.TaskTrackerInfo[] getBlacklistedTrackers()
                                                                     throws IOException,
                                                                            InterruptedException

Specified by:

getBlacklistedTrackers in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getClusterMetrics

public org.apache.hadoop.mapreduce.ClusterMetrics getClusterMetrics()
                                                             throws IOException,
                                                                    InterruptedException

Specified by:

getClusterMetrics in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getDelegationToken

public org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
                                                                                                                                           throws IOException,
                                                                                                                                                  InterruptedException

Specified by:

getDelegationToken in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getFilesystemName

public String getFilesystemName()
                         throws IOException,
                                InterruptedException

Specified by:

getFilesystemName in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getNewJobID

public org.apache.hadoop.mapreduce.JobID getNewJobID()
                                              throws IOException,
                                                     InterruptedException

Specified by:

getNewJobID in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getQueue

public org.apache.hadoop.mapreduce.QueueInfo getQueue(String queueName)
                                               throws IOException,
                                                      InterruptedException

Specified by:

getQueue in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getQueueAclsForCurrentUser

public org.apache.hadoop.mapreduce.QueueAclsInfo[] getQueueAclsForCurrentUser()
                                                                       throws IOException,
                                                                              InterruptedException

Specified by:

getQueueAclsForCurrentUser in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getQueues

public org.apache.hadoop.mapreduce.QueueInfo[] getQueues()
                                                  throws IOException,
                                                         InterruptedException

Specified by:

getQueues in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getRootQueues

public org.apache.hadoop.mapreduce.QueueInfo[] getRootQueues()
                                                      throws IOException,
                                                             InterruptedException

Specified by:

getRootQueues in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getChildQueues

public org.apache.hadoop.mapreduce.QueueInfo[] getChildQueues(String parent)
                                                       throws IOException,
                                                              InterruptedException

Specified by:

getChildQueues in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getStagingAreaDir

public String getStagingAreaDir()
                         throws IOException,
                                InterruptedException

Specified by:

getStagingAreaDir in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getSystemDir

public String getSystemDir()
                    throws IOException,
                           InterruptedException

Specified by:

getSystemDir in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getTaskTrackerExpiryInterval

public long getTaskTrackerExpiryInterval()
                                  throws IOException,
                                         InterruptedException

Specified by:

getTaskTrackerExpiryInterval in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

submitJob

public org.apache.hadoop.mapreduce.JobStatus submitJob(org.apache.hadoop.mapreduce.JobID jobId,
                                                       String jobSubmitDir,
                                                       org.apache.hadoop.security.Credentials ts)
                                                throws IOException,
                                                       InterruptedException

Specified by:

submitJob in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

createApplicationSubmissionContext

public org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext createApplicationSubmissionContext(org.apache.hadoop.conf.Configuration jobConf,
                                                                                                          String jobSubmitDir,
                                                                                                          org.apache.hadoop.security.Credentials ts)
                                                                                                   throws IOException

Constructs all the necessary information to start the MR AM.

Parameters:

jobConf - the configuration for the MR job

jobSubmitDir - the directory path for the job

ts - the security credentials for the job

Returns:

ApplicationSubmissionContext

Throws:

IOException - on IO error (e.g. path resolution)

setJobPriority

public void setJobPriority(org.apache.hadoop.mapreduce.JobID arg0,
                           String arg1)
                    throws IOException,
                           InterruptedException

Specified by:

setJobPriority in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getProtocolVersion

public long getProtocolVersion(String arg0,
                               long arg1)
                        throws IOException

Specified by:

getProtocolVersion in interface org.apache.hadoop.ipc.VersionedProtocol

Throws:

IOException

renewDelegationToken

public long renewDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
                          throws IOException,
                                 InterruptedException

Specified by:

renewDelegationToken in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getJobCounters

public org.apache.hadoop.mapreduce.Counters getJobCounters(org.apache.hadoop.mapreduce.JobID arg0)
                                                    throws IOException,
                                                           InterruptedException

Specified by:

getJobCounters in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getJobHistoryDir

public String getJobHistoryDir()
                        throws IOException,
                               InterruptedException

Specified by:

getJobHistoryDir in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getJobStatus

public org.apache.hadoop.mapreduce.JobStatus getJobStatus(org.apache.hadoop.mapreduce.JobID jobID)
                                                   throws IOException,
                                                          InterruptedException

Specified by:

getJobStatus in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getTaskCompletionEvents

public org.apache.hadoop.mapreduce.TaskCompletionEvent[] getTaskCompletionEvents(org.apache.hadoop.mapreduce.JobID arg0,
                                                                                 int arg1,
                                                                                 int arg2)
                                                                          throws IOException,
                                                                                 InterruptedException

Specified by:

getTaskCompletionEvents in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getTaskDiagnostics

public String[] getTaskDiagnostics(org.apache.hadoop.mapreduce.TaskAttemptID arg0)
                            throws IOException,
                                   InterruptedException

Specified by:

getTaskDiagnostics in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getTaskReports

public org.apache.hadoop.mapreduce.TaskReport[] getTaskReports(org.apache.hadoop.mapreduce.JobID jobID,
                                                               org.apache.hadoop.mapreduce.TaskType taskType)
                                                        throws IOException,
                                                               InterruptedException

Specified by:

getTaskReports in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

killJob

public void killJob(org.apache.hadoop.mapreduce.JobID arg0)
             throws IOException,
                    InterruptedException

Specified by:

killJob in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

killTask

public boolean killTask(org.apache.hadoop.mapreduce.TaskAttemptID arg0,
                        boolean arg1)
                 throws IOException,
                        InterruptedException

Specified by:

killTask in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getQueueAdmins

public org.apache.hadoop.security.authorize.AccessControlList getQueueAdmins(String arg0)
                                                                      throws IOException

Specified by:

getQueueAdmins in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

getJobTrackerStatus

public org.apache.hadoop.mapreduce.Cluster.JobTrackerStatus getJobTrackerStatus()
                                                                         throws IOException,
                                                                                InterruptedException

Specified by:

getJobTrackerStatus in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

InterruptedException

getProtocolSignature

public org.apache.hadoop.ipc.ProtocolSignature getProtocolSignature(String protocol,
                                                                    long clientVersion,
                                                                    int clientMethodsHash)
                                                             throws IOException

Specified by:

getProtocolSignature in interface org.apache.hadoop.ipc.VersionedProtocol

Throws:

IOException

getLogFileParams

public org.apache.hadoop.mapreduce.v2.LogParams getLogFileParams(org.apache.hadoop.mapreduce.JobID jobID,
                                                                 org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID)
                                                          throws IOException

Specified by:

getLogFileParams in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException