FileOutputFormat (Apache Hadoop Main 3.3.0 API)

java.lang.Object
- org.apache.hadoop.mapred.FileOutputFormat<K,V>

All Implemented Interfaces:

OutputFormat<K,V>

Direct Known Subclasses:

MapFileOutputFormat, MultipleOutputFormat, SequenceFileOutputFormat, TextOutputFormat
```
@InterfaceAudience.Public
 @InterfaceStability.Stable
public abstract class FileOutputFormat<K,V>
extends Object
implements OutputFormat<K,V>
```
A base class for OutputFormat.

Constructor Summary

Constructors
Constructor and Description

FileOutputFormat()

Constructors
Constructor and Description
`FileOutputFormat()`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`checkOutputSpecs(FileSystem ignored, JobConf job)` Check for validity of the output-specification for the job.
`static boolean`	`getCompressOutput(JobConf conf)` Is the job output compressed?
`static Class<? extends CompressionCodec>`	`getOutputCompressorClass(JobConf conf, Class<? extends CompressionCodec> defaultValue)` Get the `CompressionCodec` for compressing the job outputs.
`static Path`	`getOutputPath(JobConf conf)` Get the `Path` to the output directory for the map-reduce job.
`static Path`	`getPathForCustomFile(JobConf conf, String name)` Helper function to generate a `Path` for a file that is unique for the task within the job output directory.
`abstract RecordWriter<K,V>`	`getRecordWriter(FileSystem ignored, JobConf job, String name, Progressable progress)` Get the `RecordWriter` for the given job.
`static Path`	`getTaskOutputPath(JobConf conf, String name)` Helper function to create the task's temporary output directory and return the path to the task's output file.
`static String`	`getUniqueName(JobConf conf, String name)` Helper function to generate a name that is unique for the task.
`static Path`	`getWorkOutputPath(JobConf conf)` Get the `Path` to the task's temporary output directory for the map-reduce job Tasks' Side-Effect Files
`static void`	`setCompressOutput(JobConf conf, boolean compress)` Set whether the output of the job is compressed.
`static void`	`setOutputCompressorClass(JobConf conf, Class<? extends CompressionCodec> codecClass)` Set the `CompressionCodec` to be used to compress job outputs.
`static void`	`setOutputPath(JobConf conf, Path outputDir)` Set the `Path` of the output directory for the map-reduce job.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - FileOutputFormat
```
public FileOutputFormat()
```
- Method Detail
  - setCompressOutput
```
public static void setCompressOutput(JobConf conf,
                                     boolean compress)
```
    Set whether the output of the job is compressed.
    
    Parameters:
    
    conf - the JobConf to modify
    
    compress - should the output of the job be compressed?
  - getCompressOutput
```
public static boolean getCompressOutput(JobConf conf)
```
    Is the job output compressed?
    
    Parameters:
    
    conf - the JobConf to look in
    
    Returns:
    
    true if the job output should be compressed, false otherwise
  - setOutputCompressorClass
```
public static void setOutputCompressorClass(JobConf conf,
                                            Class<? extends CompressionCodec> codecClass)
```
    Set the CompressionCodec to be used to compress job outputs.
    
    Parameters:
    
    conf - the JobConf to modify
    
    codecClass - the CompressionCodec to be used to compress the job outputs
  - getOutputCompressorClass
```
public static Class<? extends CompressionCodec> getOutputCompressorClass(JobConf conf,
                                                                         Class<? extends CompressionCodec> defaultValue)
```
    Get the CompressionCodec for compressing the job outputs.
    
    Parameters:
    
    conf - the JobConf to look in
    
    defaultValue - the CompressionCodec to return if not set
    
    Returns:
    
    the CompressionCodec to be used to compress the job outputs
    
    Throws:
    
    IllegalArgumentException - if the class was specified, but not found
  - getRecordWriter
```
public abstract RecordWriter<K,V> getRecordWriter(FileSystem ignored,
                                                  JobConf job,
                                                  String name,
                                                  Progressable progress)
                                           throws IOException
```
    Description copied from interface: OutputFormat
    
    Get the RecordWriter for the given job.
    
    Specified by:
    
    getRecordWriter in interface OutputFormat<K,V>
    
    job - configuration for the job whose output is being written.
    
    name - the unique name for this part of the output.
    
    progress - mechanism for reporting progress while writing to file.
    
    Returns:
    
    a RecordWriter to write the output for the job.
    
    Throws:
    
    IOException
  - checkOutputSpecs
```
public void checkOutputSpecs(FileSystem ignored,
                             JobConf job)
                      throws FileAlreadyExistsException,
                             InvalidJobConfException,
                             IOException
```
    Description copied from interface: OutputFormat
    
    Check for validity of the output-specification for the job.
    This is to validate the output specification for the job when it is a job is submitted. Typically checks that it does not already exist, throwing an exception when it already exists, so that output is not overwritten.
    Implementations which write to filesystems which support delegation tokens usually collect the tokens for the destination path(s) and attach them to the job configuration.
    
    Specified by:
    
    checkOutputSpecs in interface OutputFormat<K,V>
    
    job - job configuration.
    
    Throws:
    
    IOException - when output should not be attempted
    
    FileAlreadyExistsException
    
    InvalidJobConfException
  - setOutputPath
```
public static void setOutputPath(JobConf conf,
                                 Path outputDir)
```
    Set the Path of the output directory for the map-reduce job.
    
    Parameters:
    
    conf - The configuration of the job.
    
    outputDir - the Path of the output directory for the map-reduce job.
  - getOutputPath
```
public static Path getOutputPath(JobConf conf)
```
    Get the Path to the output directory for the map-reduce job.
    
    Returns:
    
    the Path to the output directory for the map-reduce job.
    
    See Also:
    
    getWorkOutputPath(JobConf)
  - getWorkOutputPath
```
public static Path getWorkOutputPath(JobConf conf)
```
    Get the Path to the task's temporary output directory for the map-reduce job Tasks' Side-Effect Files
    Note: The following is valid only if the OutputCommitter is FileOutputCommitter. If OutputCommitter is not a FileOutputCommitter, the task's temporary output directory is same as getOutputPath(JobConf) i.e. ${mapreduce.output.fileoutputformat.outputdir}$
    
    Some applications need to create/write-to side-files, which differ from the actual job-outputs.
    In such cases there could be issues with 2 instances of the same TIP (running simultaneously e.g. speculative tasks) trying to open/write-to the same file (path) on HDFS. Hence the application-writer will have to pick unique names per task-attempt (e.g. using the attemptid, say attempt_200709221812_0001_m_000000_0), not just per TIP.
    
    To get around this the Map-Reduce framework helps the application-writer out by maintaining a special ${mapreduce.output.fileoutputformat.outputdir}/_temporary/_${taskid} sub-directory for each task-attempt on HDFS where the output of the task-attempt goes. On successful completion of the task-attempt the files in the ${mapreduce.output.fileoutputformat.outputdir}/_temporary/_${taskid} (only) are promoted to ${mapreduce.output.fileoutputformat.outputdir}. Of course, the framework discards the sub-directory of unsuccessful task-attempts. This is completely transparent to the application.
    
    The application-writer can take advantage of this by creating any side-files required in ${mapreduce.task.output.dir} during execution of his reduce-task i.e. via getWorkOutputPath(JobConf), and the framework will move them out similarly - thus she doesn't have to pick unique paths per task-attempt.
    
    Note: the value of ${mapreduce.task.output.dir} during execution of a particular task-attempt is actually ${mapreduce.output.fileoutputformat.outputdir}/_temporary/_{$taskid}, and this value is set by the map-reduce framework. So, just create any side-files in the path returned by getWorkOutputPath(JobConf) from map/reduce task to take advantage of this feature.
    
    The entire discussion holds true for maps of jobs with reducer=NONE (i.e. 0 reduces) since output of the map, in that case, goes directly to HDFS.
    
    Returns:
    
    the Path to the task's temporary output directory for the map-reduce job.
  - getTaskOutputPath
```
public static Path getTaskOutputPath(JobConf conf,
                                     String name)
                              throws IOException
```
    Helper function to create the task's temporary output directory and return the path to the task's output file.
    
    Parameters:
    
    conf - job-configuration
    
    name - temporary task-output filename
    
    Returns:
    
    path to the task's temporary output file
    
    Throws:
    
    IOException
  - getUniqueName
```
public static String getUniqueName(JobConf conf,
                                   String name)
```
    Helper function to generate a name that is unique for the task.
    The generated name can be used to create custom files from within the different tasks for the job, the names for different tasks will not collide with each other.
    
    The given name is postfixed with the task type, 'm' for maps, 'r' for reduces and the task partition number. For example, give a name 'test' running on the first map o the job the generated name will be 'test-m-00000'.
    
    Parameters:
    
    conf - the configuration for the job.
    
    name - the name to make unique.
    
    Returns:
    
    a unique name accross all tasks of the job.
  - getPathForCustomFile
```
public static Path getPathForCustomFile(JobConf conf,
                                        String name)
```
    Helper function to generate a Path for a file that is unique for the task within the job output directory.
    The path can be used to create custom files from within the map and reduce tasks. The path name will be unique for each task. The path parent will be the job output directory.
    ls
    This method uses the getUniqueName(org.apache.hadoop.mapred.JobConf, java.lang.String) method to make the file name unique for the task.
    
    Parameters:
    
    conf - the configuration for the job.
    
    name - the name for the file.
    
    Returns:
    
    a unique path accross all tasks of the job.

Class FileOutputFormat<K,V>

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

FileOutputFormat

Method Detail

setCompressOutput

getCompressOutput

setOutputCompressorClass

getOutputCompressorClass

getRecordWriter

checkOutputSpecs

setOutputPath

getOutputPath

getWorkOutputPath

getTaskOutputPath

getUniqueName

getPathForCustomFile