org.apache.hadoop.mapreduce.lib.output

Class PartialFileOutputCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

org.apache.hadoop.mapreduce.lib.output.PartialFileOutputCommitter

All Implemented Interfaces:

PartialOutputCommitter

@Checkpointable
@InterfaceAudience.Public
@InterfaceStability.Evolving
public class PartialFileOutputCommitter
extends FileOutputCommitter
implements PartialOutputCommitter

An OutputCommitter that commits files specified in job output directory i.e. ${mapreduce.output.fileoutputformat.outputdir}.

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT, FILEOUTPUTCOMMITTER_CLEANUP_FAILURES_IGNORED, FILEOUTPUTCOMMITTER_CLEANUP_FAILURES_IGNORED_DEFAULT, FILEOUTPUTCOMMITTER_CLEANUP_SKIPPED, FILEOUTPUTCOMMITTER_CLEANUP_SKIPPED_DEFAULT, FILEOUTPUTCOMMITTER_FAILURE_ATTEMPTS, FILEOUTPUTCOMMITTER_FAILURE_ATTEMPTS_DEFAULT, FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED, FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED_DEFAULT, PENDING_DIR_NAME, SUCCEEDED_FILE_NAME, SUCCESSFUL_JOB_OUTPUT_DIR_MARKER, TEMP_DIR_NAME

Constructor Summary

Constructors

Constructor and Description
PartialFileOutputCommitter(Path outputPath, JobContext context)
PartialFileOutputCommitter(Path outputPath, TaskAttemptContext context)

Method Summary

Methods

Modifier and Type	Method and Description
void	cleanUpPartialOutputForTask(TaskAttemptContext context) Remove all previously committed outputs from prior executions of this task.
Path	getCommittedTaskPath(int appAttemptId, TaskAttemptContext context) Compute the path where the output of a committed task is stored until the entire job is committed for a specific application attempt.

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

abortJob, abortTask, cleanupJob, commitJob, commitJobInternal, commitTask, getCommittedTaskPath, getCommittedTaskPath, getJobAttemptPath, getJobAttemptPath, getJobAttemptPath, getTaskAttemptPath, getTaskAttemptPath, getWorkPath, isCommitJobRepeatable, isRecoverySupported, needsTaskCommit, recoverTask, setupJob, setupTask

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PartialFileOutputCommitter

public PartialFileOutputCommitter(Path outputPath,
                          TaskAttemptContext context)
                           throws IOException

Throws:

IOException

PartialFileOutputCommitter

public PartialFileOutputCommitter(Path outputPath,
                          JobContext context)
                           throws IOException

Throws:

IOException

Method Detail

getCommittedTaskPath

public Path getCommittedTaskPath(int appAttemptId,
                        TaskAttemptContext context)

Description copied from class: FileOutputCommitter

Compute the path where the output of a committed task is stored until the entire job is committed for a specific application attempt.

Overrides:

getCommittedTaskPath in class FileOutputCommitter

Parameters:

appAttemptId - the id of the application attempt to use

context - the context of any task.

Returns:

the path where the output of a committed task is stored.

cleanUpPartialOutputForTask

public void cleanUpPartialOutputForTask(TaskAttemptContext context)
                                 throws IOException

Description copied from interface: PartialOutputCommitter

Remove all previously committed outputs from prior executions of this task.

Specified by:

cleanUpPartialOutputForTask in interface PartialOutputCommitter

Parameters:

context - Context for cleaning up previously promoted output.

Throws:

IOException - If cleanup fails, then the state of the task my not be well defined.