org.apache.hadoop.mapred
Class LocalJobRunner

java.lang.Object
  extended by org.apache.hadoop.mapred.LocalJobRunner

public class LocalJobRunner
extends Object

Implements MapReduce locally, in-process, for debugging.


Field Summary
static String LOCAL_MAX_MAPS
           
static org.apache.commons.logging.Log LOG
           
static long versionID
           
 
Constructor Summary
LocalJobRunner(JobConf conf)
           
 
Method Summary
 void cancelDelegationToken(Token<DelegationTokenIdentifier> token)
          Cancel a delegation token.
 JobStatus[] getAllJobs()
          Get all the jobs submitted.
 TaskReport[] getCleanupTaskReports(JobID id)
          Grab a bunch of info on the cleanup tasks that make up the job
 ClusterStatus getClusterStatus(boolean detailed)
          Get the current status of the cluster
 Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
          Get a new delegation token.
 String getFilesystemName()
          A MapReduce system always operates on a single filesystem.
 Counters getJobCounters(JobID id)
          Grab the current job counters
 JobProfile getJobProfile(JobID id)
          Grab a handle to a job that is already known to the JobTracker.
 JobStatus[] getJobsFromQueue(String queue)
          Gets all the jobs submitted to the particular Queue
 JobStatus getJobStatus(JobID id)
          Grab a handle to a job that is already known to the JobTracker.
static int getLocalMaxRunningMaps(JobContext job)
           
 TaskReport[] getMapTaskReports(JobID id)
          Grab a bunch of info on the map tasks that make up the job
 JobID getNewJobId()
          Allocate a name for the job.
 long getProtocolVersion(String protocol, long clientVersion)
          Return protocol version corresponding to protocol interface.
 QueueAclsInfo[] getQueueAclsForCurrentUser()
          Gets the Queue ACLs for current user
 AccessControlList getQueueAdmins(String queueName)
          Get the administrators of the given job-queue.
 JobQueueInfo getQueueInfo(String queue)
          Gets scheduling information associated with the particular Job queue
 JobQueueInfo[] getQueues()
          Gets set of Job Queues associated with the Job Tracker
 TaskReport[] getReduceTaskReports(JobID id)
          Grab a bunch of info on the reduce tasks that make up the job
 TaskReport[] getSetupTaskReports(JobID id)
          Grab a bunch of info on the setup tasks that make up the job
 String getStagingAreaDir()
          Get a hint from the JobTracker where job-specific files are to be placed.
 String getSystemDir()
          Grab the jobtracker system directory path where job-specific files are to be placed.
 TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents)
          Get task completion events for the jobid, starting from fromEventId.
 String[] getTaskDiagnostics(TaskAttemptID taskid)
          Returns the diagnostic information for a particular task in the given job.
 JobStatus[] jobsToComplete()
          Get the jobs that are not completed and not failed
 void killJob(JobID id)
          Kill the indicated job
 boolean killTask(TaskAttemptID taskId, boolean shouldFail)
          Throws UnsupportedOperationException
 long renewDelegationToken(Token<DelegationTokenIdentifier> token)
          Renew an existing delegation token
 void setJobPriority(JobID id, String jp)
          Set the priority of the specified job
static void setLocalMaxRunningMaps(JobContext job, int maxMaps)
          Set the max number of map tasks to run concurrently in the LocalJobRunner.
 JobStatus submitJob(JobID jobid, String jobSubmitDir, Credentials credentials)
          Submit a Job for execution.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

LOCAL_MAX_MAPS

public static final String LOCAL_MAX_MAPS
See Also:
Constant Field Values

versionID

public static final long versionID
See Also:
Constant Field Values
Constructor Detail

LocalJobRunner

public LocalJobRunner(JobConf conf)
               throws IOException
Throws:
IOException
Method Detail

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
Description copied from interface: VersionedProtocol
Return protocol version corresponding to protocol interface.

Parameters:
protocol - The classname of the protocol interface
clientVersion - The version of the protocol that the client speaks
Returns:
the version that the server will speak

getNewJobId

public JobID getNewJobId()
Allocate a name for the job.

Returns:
a unique job name for submitting jobs.

submitJob

public JobStatus submitJob(JobID jobid,
                           String jobSubmitDir,
                           Credentials credentials)
                    throws IOException
Submit a Job for execution. Returns the latest profile for that job. The job files should be submitted in jobSubmitDir.

Throws:
IOException

killJob

public void killJob(JobID id)
Kill the indicated job


setJobPriority

public void setJobPriority(JobID id,
                           String jp)
                    throws IOException
Set the priority of the specified job

Parameters:
id - ID of the job
jp - Priority to be set for the job
Throws:
IOException

killTask

public boolean killTask(TaskAttemptID taskId,
                        boolean shouldFail)
                 throws IOException
Throws UnsupportedOperationException

Parameters:
taskId - the id of the task to kill.
shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.
Throws:
IOException

getJobProfile

public JobProfile getJobProfile(JobID id)
Grab a handle to a job that is already known to the JobTracker.

Returns:
Profile of the job, or null if not found.

getMapTaskReports

public TaskReport[] getMapTaskReports(JobID id)
Grab a bunch of info on the map tasks that make up the job


getReduceTaskReports

public TaskReport[] getReduceTaskReports(JobID id)
Grab a bunch of info on the reduce tasks that make up the job


getCleanupTaskReports

public TaskReport[] getCleanupTaskReports(JobID id)
Grab a bunch of info on the cleanup tasks that make up the job


getSetupTaskReports

public TaskReport[] getSetupTaskReports(JobID id)
Grab a bunch of info on the setup tasks that make up the job


getJobStatus

public JobStatus getJobStatus(JobID id)
Grab a handle to a job that is already known to the JobTracker.

Returns:
Status of the job, or null if not found.

getJobCounters

public Counters getJobCounters(JobID id)
Grab the current job counters


getFilesystemName

public String getFilesystemName()
                         throws IOException
A MapReduce system always operates on a single filesystem. This function returns the fs name. ('local' if the localfs; 'addr:port' if dfs). The client can then copy files into the right locations prior to submitting the job.

Throws:
IOException

getClusterStatus

public ClusterStatus getClusterStatus(boolean detailed)
Get the current status of the cluster

Parameters:
detailed - if true then report tracker names and memory usage
Returns:
summary of the state of the cluster

jobsToComplete

public JobStatus[] jobsToComplete()
Get the jobs that are not completed and not failed

Returns:
array of JobStatus for the running/to-be-run jobs.

getTaskCompletionEvents

public TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                                     int fromEventId,
                                                     int maxEvents)
                                              throws IOException
Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Parameters:
jobid - job id
fromEventId - event id to start from.
maxEvents - the max number of events we want to look at
Returns:
array of task completion events.
Throws:
IOException

getAllJobs

public JobStatus[] getAllJobs()
Get all the jobs submitted.

Returns:
array of JobStatus for the submitted jobs

getTaskDiagnostics

public String[] getTaskDiagnostics(TaskAttemptID taskid)
                            throws IOException
Returns the diagnostic information for a particular task in the given job. To be implemented

Parameters:
taskid - the id of the task
Returns:
an array of the diagnostic messages
Throws:
IOException

getSystemDir

public String getSystemDir()
Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:
the system directory where job-specific files are to be placed.
See Also:
JobSubmissionProtocol.getSystemDir()

getQueueAdmins

public AccessControlList getQueueAdmins(String queueName)
                                 throws IOException
Get the administrators of the given job-queue. This method is for hadoop internal use only.

Returns:
Queue administrators ACL for the queue to which job is submitted to
Throws:
IOException
See Also:
org.apache.hadoop.mapred.JobSubmissionProtocol#getQueueAdmins()

getStagingAreaDir

public String getStagingAreaDir()
                         throws IOException
Get a hint from the JobTracker where job-specific files are to be placed.

Returns:
the directory where job-specific files are to be placed.
Throws:
IOException
See Also:
JobSubmissionProtocol.getStagingAreaDir()

getJobsFromQueue

public JobStatus[] getJobsFromQueue(String queue)
                             throws IOException
Gets all the jobs submitted to the particular Queue

Parameters:
queue - Queue name
Returns:
array of JobStatus for the submitted jobs
Throws:
IOException

getQueues

public JobQueueInfo[] getQueues()
                         throws IOException
Gets set of Job Queues associated with the Job Tracker

Returns:
Array of the Job Queue Information Object
Throws:
IOException

getQueueInfo

public JobQueueInfo getQueueInfo(String queue)
                          throws IOException
Gets scheduling information associated with the particular Job queue

Parameters:
queue - Queue Name
Returns:
Scheduling Information of the Queue
Throws:
IOException

getQueueAclsForCurrentUser

public QueueAclsInfo[] getQueueAclsForCurrentUser()
                                           throws IOException
Gets the Queue ACLs for current user

Returns:
array of QueueAclsInfo object for current user.
Throws:
IOException

setLocalMaxRunningMaps

public static void setLocalMaxRunningMaps(JobContext job,
                                          int maxMaps)
Set the max number of map tasks to run concurrently in the LocalJobRunner.

Parameters:
job - the job to configure
maxMaps - the maximum number of map tasks to allow.

getLocalMaxRunningMaps

public static int getLocalMaxRunningMaps(JobContext job)
Returns:
the max number of map tasks to run concurrently in the LocalJobRunner.

cancelDelegationToken

public void cancelDelegationToken(Token<DelegationTokenIdentifier> token)
                           throws IOException,
                                  InterruptedException
Cancel a delegation token.

Parameters:
token - the token to cancel
Throws:
IOException
InterruptedException

getDelegationToken

public Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
                                                    throws IOException,
                                                           InterruptedException
Get a new delegation token.

Parameters:
renewer - the user other than the creator (if any) that can renew the token
Returns:
the new delegation token
Throws:
IOException
InterruptedException

renewDelegationToken

public long renewDelegationToken(Token<DelegationTokenIdentifier> token)
                          throws IOException,
                                 InterruptedException
Renew an existing delegation token

Parameters:
token - the token to renew
Returns:
the new expiration time
Throws:
IOException
InterruptedException


Copyright © 2009 The Apache Software Foundation