@InterfaceAudience.Public @InterfaceStability.Stable public abstract class OutputCommitter extends OutputCommitter
OutputCommitter
describes the commit of task output for a
Map-Reduce job.
The Map-Reduce framework relies on the OutputCommitter
of
the job to:
FileOutputCommitter
,
JobContext
,
TaskAttemptContext
Constructor and Description |
---|
OutputCommitter() |
Modifier and Type | Method and Description |
---|---|
void |
abortJob(JobContext jobContext,
int status)
For aborting an unsuccessful job's output.
|
void |
abortJob(JobContext context,
org.apache.hadoop.mapreduce.JobStatus.State runState)
This method implements the new interface by calling the old method.
|
abstract void |
abortTask(TaskAttemptContext taskContext)
Discard the task output.
|
void |
abortTask(TaskAttemptContext taskContext)
This method implements the new interface by calling the old method.
|
void |
cleanupJob(JobContext jobContext)
Deprecated.
Use
commitJob(JobContext) or
abortJob(JobContext, int) instead. |
void |
cleanupJob(JobContext context)
|
void |
commitJob(JobContext jobContext)
For committing job's output after successful job completion.
|
void |
commitJob(JobContext context)
This method implements the new interface by calling the old method.
|
abstract void |
commitTask(TaskAttemptContext taskContext)
To promote the task's temporary output to final output location.
|
void |
commitTask(TaskAttemptContext taskContext)
This method implements the new interface by calling the old method.
|
boolean |
isCommitJobRepeatable(JobContext jobContext)
Returns true if an in-progress job commit can be retried.
|
boolean |
isCommitJobRepeatable(JobContext jobContext)
Returns true if an in-progress job commit can be retried.
|
boolean |
isRecoverySupported()
Deprecated.
Use
isRecoverySupported(JobContext) instead. |
boolean |
isRecoverySupported(JobContext jobContext)
Is task output recovery supported for restarting jobs?
If task output recovery is supported, job restart can be done more
efficiently.
|
boolean |
isRecoverySupported(JobContext context)
This method implements the new interface by calling the old method.
|
abstract boolean |
needsTaskCommit(TaskAttemptContext taskContext)
Check whether task needs a commit.
|
boolean |
needsTaskCommit(TaskAttemptContext taskContext)
This method implements the new interface by calling the old method.
|
void |
recoverTask(TaskAttemptContext taskContext)
Recover the task output.
|
void |
recoverTask(TaskAttemptContext taskContext)
This method implements the new interface by calling the old method.
|
abstract void |
setupJob(JobContext jobContext)
For the framework to setup the job output during initialization.
|
void |
setupJob(JobContext jobContext)
This method implements the new interface by calling the old method.
|
abstract void |
setupTask(TaskAttemptContext taskContext)
Sets up output for the task.
|
void |
setupTask(TaskAttemptContext taskContext)
This method implements the new interface by calling the old method.
|
public abstract void setupJob(JobContext jobContext) throws IOException
jobContext
- Context of the job whose output is being written.IOException
- if temporary output could not be created@Deprecated public void cleanupJob(JobContext jobContext) throws IOException
commitJob(JobContext)
or
abortJob(JobContext, int)
instead.jobContext
- Context of the job whose output is being written.IOException
public void commitJob(JobContext jobContext) throws IOException
jobContext
- Context of the job whose output is being written.IOException
public void abortJob(JobContext jobContext, int status) throws IOException
JobStatus.FAILED
or
JobStatus.KILLED
. This is called from the application
master process for the entire job. This may be called multiple times.jobContext
- Context of the job whose output is being written.status
- final runstate of the jobIOException
public abstract void setupTask(TaskAttemptContext taskContext) throws IOException
taskContext
- Context of the task whose output is being written.IOException
public abstract boolean needsTaskCommit(TaskAttemptContext taskContext) throws IOException
taskContext
- IOException
public abstract void commitTask(TaskAttemptContext taskContext) throws IOException
needsTaskCommit(TaskAttemptContext)
returns true and this
task is the task that the AM determines finished first, this method
is called to commit an individual task's output. This is to mark
that tasks output as complete, as commitJob(JobContext)
will
also be called later on if the entire job finished successfully. This
is called from a task's process. This may be called multiple times for the
same task, but different task attempts. It should be very rare for this to
be called multiple times and requires odd networking failures to make this
happen. In the future the Hadoop framework may eliminate this race.taskContext
- Context of the task whose output is being written.IOException
- if commit is notpublic abstract void abortTask(TaskAttemptContext taskContext) throws IOException
taskContext
- IOException
@Deprecated public boolean isRecoverySupported()
isRecoverySupported(JobContext)
instead.isRecoverySupported
in class OutputCommitter
true
if task output recovery is supported,
false
otherwiseOutputCommitter.recoverTask(TaskAttemptContext)
public boolean isRecoverySupported(JobContext jobContext) throws IOException
jobContext
- Context of the job whose output is being written.true
if task output recovery is supported,
false
otherwiseIOException
recoverTask(TaskAttemptContext)
public boolean isCommitJobRepeatable(JobContext jobContext) throws IOException
jobContext
- Context of the job whose output is being written.true
repeatable job commit is supported,
false
otherwiseIOException
public boolean isCommitJobRepeatable(JobContext jobContext) throws IOException
OutputCommitter
isCommitJobRepeatable
in class OutputCommitter
jobContext
- Context of the job whose output is being written.true
repeatable job commit is supported,
false
otherwiseIOException
public void recoverTask(TaskAttemptContext taskContext) throws IOException
MRConstants.APPLICATION_ATTEMPT_ID
key in
JobContext.getConfiguration()
for the
OutputCommitter
. This is called from the application master
process, but it is called individually for each task.
If an exception is thrown the task will be attempted again.taskContext
- Context of the task whose output is being recoveredIOException
public final void setupJob(JobContext jobContext) throws IOException
setupJob
in class OutputCommitter
jobContext
- Context of the job whose output is being written.IOException
- if temporary output could not be created@Deprecated public final void cleanupJob(JobContext context) throws IOException
commitJob(org.apache.hadoop.mapreduce.JobContext)
or abortJob(org.apache.hadoop.mapreduce.JobContext, org.apache.hadoop.mapreduce.JobStatus.State)
instead.cleanupJob
in class OutputCommitter
context
- Context of the job whose output is being written.IOException
public final void commitJob(JobContext context) throws IOException
commitJob
in class OutputCommitter
context
- Context of the job whose output is being written.IOException
public final void abortJob(JobContext context, org.apache.hadoop.mapreduce.JobStatus.State runState) throws IOException
abortJob
in class OutputCommitter
context
- Context of the job whose output is being written.runState
- final runstate of the jobIOException
public final void setupTask(TaskAttemptContext taskContext) throws IOException
setupTask
in class OutputCommitter
taskContext
- Context of the task whose output is being written.IOException
public final boolean needsTaskCommit(TaskAttemptContext taskContext) throws IOException
needsTaskCommit
in class OutputCommitter
IOException
public final void commitTask(TaskAttemptContext taskContext) throws IOException
commitTask
in class OutputCommitter
taskContext
- Context of the task whose output is being written.IOException
- if commit is not successful.public final void abortTask(TaskAttemptContext taskContext) throws IOException
abortTask
in class OutputCommitter
IOException
public final void recoverTask(TaskAttemptContext taskContext) throws IOException
recoverTask
in class OutputCommitter
taskContext
- Context of the task whose output is being recoveredIOException
public final boolean isRecoverySupported(JobContext context) throws IOException
isRecoverySupported
in class OutputCommitter
context
- Context of the job whose output is being written.true
if task output recovery is supported,
false
otherwiseIOException
OutputCommitter.recoverTask(TaskAttemptContext)
Copyright © 2021 Apache Software Foundation. All rights reserved.