Package org.apache.hadoop.mapred

Class FileOutputCommitter

java.lang.Object
org.apache.hadoop.mapreduce.OutputCommitter
org.apache.hadoop.mapred.OutputCommitter
org.apache.hadoop.mapred.FileOutputCommitter
@Public @Stable public class FileOutputCommitter extends OutputCommitter
An OutputCommitter that commits files specified in job output directory i.e. ${mapreduce.output.fileoutputformat.outputdir}.

  • Field Details

  • Constructor Details

    • FileOutputCommitter

      public FileOutputCommitter()

  • Method Details

    • getTaskAttemptPath

      @Private public Path getTaskAttemptPath(TaskAttemptContext context) throws IOException
      Throws:
      IOException

    • getWorkPath

      public Path getWorkPath(TaskAttemptContext context, Path outputPath) throws IOException
      Throws:
      IOException

    • setupJob

      public void setupJob(JobContext context) throws IOException
      Description copied from class: OutputCommitter
      For the framework to setup the job output during initialization. This is called from the application master process for the entire job. This will be called multiple times, once per job attempt.
      Specified by:
      setupJob in class OutputCommitter
      Parameters:
      context - Context of the job whose output is being written.
      Throws:
      IOException - if temporary output could not be created

    • commitJob

      public void commitJob(JobContext context) throws IOException
      Description copied from class: OutputCommitter
      For committing job's output after successful job completion. Note that this is invoked for jobs with final runstate as SUCCESSFUL. This is called from the application master process for the entire job. This is guaranteed to only be called once. If it throws an exception the entire job will fail.
      Overrides:
      commitJob in class OutputCommitter
      Parameters:
      context - Context of the job whose output is being written.
      Throws:
      IOException

    • cleanupJob

      @Deprecated public void cleanupJob(JobContext context) throws IOException
      Deprecated.
      Description copied from class: OutputCommitter
      For cleaning up the job's output after job completion. This is called from the application master process for the entire job. This may be called multiple times.
      Overrides:
      cleanupJob in class OutputCommitter
      Parameters:
      context - Context of the job whose output is being written.
      Throws:
      IOException

    • abortJob

      public void abortJob(JobContext context, int runState) throws IOException
      Description copied from class: OutputCommitter
      For aborting an unsuccessful job's output. Note that this is invoked for jobs with final runstate as JobStatus.FAILED or JobStatus.KILLED. This is called from the application master process for the entire job. This may be called multiple times.
      Overrides:
      abortJob in class OutputCommitter
      Parameters:
      context - Context of the job whose output is being written.
      runState - final runstate of the job
      Throws:
      IOException

    • setupTask

      public void setupTask(TaskAttemptContext context) throws IOException
      Description copied from class: OutputCommitter
      Sets up output for the task. This is called from each individual task's process that will output to HDFS, and it is called just for that task. This may be called multiple times for the same task, but for different task attempts.
      Specified by:
      setupTask in class OutputCommitter
      Parameters:
      context - Context of the task whose output is being written.
      Throws:
      IOException

    • commitTask

      public void commitTask(TaskAttemptContext context) throws IOException
      Description copied from class: OutputCommitter
      To promote the task's temporary output to final output location. If OutputCommitter.needsTaskCommit(TaskAttemptContext) returns true and this task is the task that the AM determines finished first, this method is called to commit an individual task's output. This is to mark that tasks output as complete, as OutputCommitter.commitJob(JobContext) will also be called later on if the entire job finished successfully. This is called from a task's process. This may be called multiple times for the same task, but different task attempts. It should be very rare for this to be called multiple times and requires odd networking failures to make this happen. In the future the Hadoop framework may eliminate this race.
      Specified by:
      commitTask in class OutputCommitter
      Parameters:
      context - Context of the task whose output is being written.
      Throws:
      IOException - if commit is not

    • abortTask

      public void abortTask(TaskAttemptContext context) throws IOException
      Description copied from class: OutputCommitter
      Discard the task output. This is called from a task's process to clean up a single task's output that can not yet been committed. This may be called multiple times for the same task, but for different task attempts.
      Specified by:
      abortTask in class OutputCommitter
      Throws:
      IOException

    • needsTaskCommit

      public boolean needsTaskCommit(TaskAttemptContext context) throws IOException
      Description copied from class: OutputCommitter
      Check whether task needs a commit. This is called from each individual task's process that will output to HDFS, and it is called just for that task.
      Specified by:
      needsTaskCommit in class OutputCommitter
      Returns:
      true/false
      Throws:
      IOException

    • isRecoverySupported

      @Deprecated public boolean isRecoverySupported()
      Deprecated.
      Description copied from class: OutputCommitter
      This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.
      Overrides:
      isRecoverySupported in class OutputCommitter
      Returns:
      true if task output recovery is supported, false otherwise
      See Also:

    • isCommitJobRepeatable

      public boolean isCommitJobRepeatable(JobContext context) throws IOException
      Description copied from class: OutputCommitter
      Returns true if an in-progress job commit can be retried. If the MR AM is re-run then it will check this value to determine if it can retry an in-progress commit that was started by a previous version. Note that in rare scenarios, the previous AM version might still be running at that time, due to system anomalies. Hence if this method returns true then the retry commit operation should be able to run concurrently with the previous operation. If repeatable job commit is supported, job restart can tolerate previous AM failures during job commit. By default, it is not supported. Extended classes (like: FileOutputCommitter) should explicitly override it if provide support.
      Overrides:
      isCommitJobRepeatable in class OutputCommitter
      Parameters:
      context - Context of the job whose output is being written.
      Returns:
      true repeatable job commit is supported, false otherwise
      Throws:
      IOException

    • isRecoverySupported

      public boolean isRecoverySupported(JobContext context) throws IOException
      Description copied from class: OutputCommitter
      Is task output recovery supported for restarting jobs? If task output recovery is supported, job restart can be done more efficiently.
      Overrides:
      isRecoverySupported in class OutputCommitter
      Parameters:
      context - Context of the job whose output is being written.
      Returns:
      true if task output recovery is supported, false otherwise
      Throws:
      IOException
      See Also:

    • recoverTask

      public void recoverTask(TaskAttemptContext context) throws IOException
      Description copied from class: OutputCommitter
      Recover the task output. The retry-count for the job will be passed via the MRConstants.APPLICATION_ATTEMPT_ID key in JobContext.getConfiguration() for the OutputCommitter. This is called from the application master process, but it is called individually for each task. If an exception is thrown the task will be attempted again.
      Overrides:
      recoverTask in class OutputCommitter
      Parameters:
      context - Context of the task whose output is being recovered
      Throws:
      IOException