org.apache.hadoop.mapred.lib

Class CombineFileRecordReader<K,V>

java.lang.Object

org.apache.hadoop.mapred.lib.CombineFileRecordReader<K,V>

All Implemented Interfaces:

RecordReader<K,V>

@InterfaceAudience.Public
@InterfaceStability.Stable
public class CombineFileRecordReader<K,V>
extends Object
implements RecordReader<K,V>

A generic RecordReader that can hand out different recordReaders for each chunk in a CombineFileSplit. A CombineFileSplit can combine data chunks from multiple files. This class allows using different RecordReaders for processing these data chunks from different files.

See Also:

CombineFileSplit

Field Summary

Fields

Modifier and Type	Field and Description
protected RecordReader<K,V>	curReader
protected FileSystem	fs
protected int	idx
protected JobConf	jc
protected long	progress
protected Reporter	reporter
protected Class<RecordReader<K,V>>	rrClass
protected Constructor<RecordReader<K,V>>	rrConstructor
protected CombineFileSplit	split

Constructor Summary

Constructors

Constructor and Description
CombineFileRecordReader(JobConf job, CombineFileSplit split, Reporter reporter, Class<RecordReader<K,V>> rrClass) A generic RecordReader that can hand out different recordReaders for each chunk in the CombineFileSplit.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Close this InputSplit to future operations.
K	createKey() Create an object of the appropriate type to be used as a key.
V	createValue() Create an object of the appropriate type to be used as a value.
long	getPos() return the amount of data processed
float	getProgress() return progress based on the amount of data processed so far.
protected boolean	initNextRecordReader() Get the record reader for the next chunk in this CombineFileSplit.
boolean	next(K key, V value) Reads the next key/value pair from the input for processing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

split

protected CombineFileSplit split

jc

protected JobConf jc

reporter

protected Reporter reporter

rrClass

protected Class<RecordReader<K,V>> rrClass

rrConstructor

protected Constructor<RecordReader<K,V>> rrConstructor

fs

protected FileSystem fs

idx

protected int idx

progress

protected long progress

curReader

protected RecordReader<K,V> curReader

Constructor Detail

CombineFileRecordReader

public CombineFileRecordReader(JobConf job,
                       CombineFileSplit split,
                       Reporter reporter,
                       Class<RecordReader<K,V>> rrClass)
                        throws IOException

A generic RecordReader that can hand out different recordReaders for each chunk in the CombineFileSplit.

Throws:

IOException

Method Detail

next

public boolean next(K key,
           V value)
             throws IOException

Description copied from interface: RecordReader

Reads the next key/value pair from the input for processing.

Specified by:

next in interface RecordReader<K,V>

Parameters:

key - the key to read data into

value - the value to read data into

Returns:

true iff a key/value was read, false if at EOF

Throws:

IOException

createKey

public K createKey()

Description copied from interface: RecordReader

Create an object of the appropriate type to be used as a key.

Specified by:

createKey in interface RecordReader<K,V>

Returns:

a new key object.

createValue

public V createValue()

Description copied from interface: RecordReader

Create an object of the appropriate type to be used as a value.

Specified by:

createValue in interface RecordReader<K,V>

Returns:

a new value object.

getPos

public long getPos()
            throws IOException

return the amount of data processed

Specified by:

getPos in interface RecordReader<K,V>

Returns:

the current position in the input.

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: RecordReader

Close this InputSplit to future operations.

Specified by:

close in interface RecordReader<K,V>

Throws:

IOException

getProgress

public float getProgress()
                  throws IOException

return progress based on the amount of data processed so far.

Specified by:

getProgress in interface RecordReader<K,V>

Returns:

progress from 0.0 to 1.0.

Throws:

IOException

initNextRecordReader

protected boolean initNextRecordReader()
                                throws IOException

Get the record reader for the next chunk in this CombineFileSplit.

Throws:

IOException