org.apache.hadoop.mapreduce.jobhistory

Class JobHistoryEventHandler

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>

public class JobHistoryEventHandler
extends org.apache.hadoop.service.AbstractService
implements org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>

The job history events get routed to this class. This class writes the Job history events to the DFS directly into a staging dir and then moved to a done-dir. JobHistory implementation is in this package to access package private classes.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	JobHistoryEventHandler.MetaInfo

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.yarn.event.AsyncDispatcher	atsEventDispatcher
protected Thread	eventHandlingThread
protected BlockingQueue<JobHistoryEvent>	eventQueue
protected static Map<org.apache.hadoop.mapreduce.v2.api.records.JobId,JobHistoryEventHandler.MetaInfo>	fileMap
protected boolean	forceJobCompletion
protected boolean	handleTimelineEvent
protected org.apache.hadoop.yarn.client.api.TimelineClient	timelineClient
protected org.apache.hadoop.yarn.client.api.TimelineV2Client	timelineV2Client

Constructor Summary

Constructors

Constructor and Description
JobHistoryEventHandler(AppContext context, int startCount)

Method Summary

Modifier and Type	Method and Description
protected void	closeEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
void	closeWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId id) Close the event writer for this id
protected org.apache.hadoop.yarn.event.AsyncDispatcher	createDispatcher()
protected org.apache.hadoop.mapreduce.jobhistory.EventWriter	createEventWriter(org.apache.hadoop.fs.Path historyFilePath)
void	handle(JobHistoryEvent event)
void	handleEvent(JobHistoryEvent event)
protected void	moveTmpToDone(org.apache.hadoop.fs.Path tmpPath)
protected boolean	moveToDoneNow(org.apache.hadoop.fs.Path fromPath, org.apache.hadoop.fs.Path toPath)
protected void	processDoneFiles(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
void	processEventForJobSummary(org.apache.hadoop.mapreduce.jobhistory.HistoryEvent event, JobSummary summary, org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
protected void	serviceStop()
void	setForcejobCompletion(boolean forceJobCompletion)
protected void	setupEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId, org.apache.hadoop.mapreduce.jobhistory.AMStartedEvent amStartedEvent) Create an event writer for the Job represented by the jobID.

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

eventQueue

protected BlockingQueue<JobHistoryEvent> eventQueue

handleTimelineEvent

protected boolean handleTimelineEvent

atsEventDispatcher

protected org.apache.hadoop.yarn.event.AsyncDispatcher atsEventDispatcher

eventHandlingThread

protected Thread eventHandlingThread

fileMap

protected static final Map<org.apache.hadoop.mapreduce.v2.api.records.JobId,JobHistoryEventHandler.MetaInfo> fileMap

forceJobCompletion

protected volatile boolean forceJobCompletion

timelineClient

@VisibleForTesting
protected org.apache.hadoop.yarn.client.api.TimelineClient timelineClient

timelineV2Client

@VisibleForTesting
protected org.apache.hadoop.yarn.client.api.TimelineV2Client timelineV2Client

Constructor Detail

JobHistoryEventHandler

public JobHistoryEventHandler(AppContext context,
                              int startCount)

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

createDispatcher

protected org.apache.hadoop.yarn.event.AsyncDispatcher createDispatcher()

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

createEventWriter

protected org.apache.hadoop.mapreduce.jobhistory.EventWriter createEventWriter(org.apache.hadoop.fs.Path historyFilePath)
                                                                        throws IOException

Throws:

IOException

setupEventWriter

protected void setupEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId,
                                org.apache.hadoop.mapreduce.jobhistory.AMStartedEvent amStartedEvent)
                         throws IOException

Create an event writer for the Job represented by the jobID. Writes out the job configuration to the log directory. This should be the first call to history for a job

Parameters:

jobId - the jobId.

amStartedEvent -

Throws:

IOException

closeWriter

public void closeWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId id)
                 throws IOException

Close the event writer for this id

Throws:

IOException

handle

public void handle(JobHistoryEvent event)

Specified by:

handle in interface org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>

handleEvent

@InterfaceAudience.Private
public void handleEvent(JobHistoryEvent event)

processEventForJobSummary

public void processEventForJobSummary(org.apache.hadoop.mapreduce.jobhistory.HistoryEvent event,
                                      JobSummary summary,
                                      org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)

closeEventWriter

protected void closeEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
                         throws IOException

Throws:

IOException

processDoneFiles

protected void processDoneFiles(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
                         throws IOException

Throws:

IOException

moveTmpToDone

protected void moveTmpToDone(org.apache.hadoop.fs.Path tmpPath)
                      throws IOException

Throws:

IOException

moveToDoneNow

protected boolean moveToDoneNow(org.apache.hadoop.fs.Path fromPath,
                                org.apache.hadoop.fs.Path toPath)
                         throws IOException

Throws:

IOException

setForcejobCompletion

public void setForcejobCompletion(boolean forceJobCompletion)