@InterfaceAudience.Public @InterfaceStability.Stable public class NLineInputFormat extends FileInputFormat<LongWritable,Text>
Modifier and Type | Field and Description |
---|---|
static String |
LINES_PER_MAP |
DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE
Constructor and Description |
---|
NLineInputFormat() |
Modifier and Type | Method and Description |
---|---|
protected static FileSplit |
createFileSplit(Path fileName,
long begin,
long length)
NLineInputFormat uses LineRecordReader, which always reads
(and consumes) at least one character out of its upper split
boundary.
|
RecordReader<LongWritable,Text> |
createRecordReader(InputSplit genericSplit,
TaskAttemptContext context)
Create a record reader for a given split.
|
static int |
getNumLinesPerSplit(JobContext job)
Get the number of lines per split
|
List<InputSplit> |
getSplits(JobContext job)
Logically splits the set of input files for the job, splits N lines
of the input as one split.
|
static List<FileSplit> |
getSplitsForFile(FileStatus status,
Configuration conf,
int numLinesPerSplit) |
static void |
setNumLinesPerSplit(Job job,
int numLines)
Set the number of lines per split
|
addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, isSplitable, listStatus, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize
public static final String LINES_PER_MAP
public NLineInputFormat()
public RecordReader<LongWritable,Text> createRecordReader(InputSplit genericSplit, TaskAttemptContext context) throws IOException
InputFormat
RecordReader.initialize(InputSplit, TaskAttemptContext)
before
the split is used.createRecordReader
in class InputFormat<LongWritable,Text>
genericSplit
- the split to be readcontext
- the information about the taskIOException
public List<InputSplit> getSplits(JobContext job) throws IOException
getSplits
in class FileInputFormat<LongWritable,Text>
job
- the job contextInputSplit
s for the job.IOException
FileInputFormat.getSplits(JobContext)
public static List<FileSplit> getSplitsForFile(FileStatus status, Configuration conf, int numLinesPerSplit) throws IOException
IOException
protected static FileSplit createFileSplit(Path fileName, long begin, long length)
fileName
- Path of filebegin
- the position of the first byte in the file to processlength
- number of bytes in InputSplitpublic static void setNumLinesPerSplit(Job job, int numLines)
job
- the job to modifynumLines
- the number of lines per splitpublic static int getNumLinesPerSplit(JobContext job)
job
- the jobCopyright © 2016 Apache Software Foundation. All rights reserved.