org.apache.hadoop.mapred.lib
Class CombineFileRecordReader<K,V>

java.lang.Object
  extended by org.apache.hadoop.mapred.lib.CombineFileRecordReader<K,V>
All Implemented Interfaces:
RecordReader<K,V>

public class CombineFileRecordReader<K,V>
extends Object
implements RecordReader<K,V>

A generic RecordReader that can hand out different recordReaders for each chunk in a CombineFileSplit. A CombineFileSplit can combine data chunks from multiple files. This class allows using different RecordReaders for processing these data chunks from different files.

See Also:
CombineFileSplit

Field Summary
protected  RecordReader<K,V> curReader
           
protected  FileSystem fs
           
protected  int idx
           
protected  JobConf jc
           
protected  long progress
           
protected  Reporter reporter
           
protected  Class<RecordReader<K,V>> rrClass
           
protected  Constructor<RecordReader<K,V>> rrConstructor
           
protected  CombineFileSplit split
           
 
Constructor Summary
CombineFileRecordReader(JobConf job, CombineFileSplit split, Reporter reporter, Class<RecordReader<K,V>> rrClass)
          A generic RecordReader that can hand out different recordReaders for each chunk in the CombineFileSplit.
 
Method Summary
 void close()
          Close this InputSplit to future operations.
 K createKey()
          Create an object of the appropriate type to be used as a key.
 V createValue()
          Create an object of the appropriate type to be used as a value.
 long getPos()
          return the amount of data processed
 float getProgress()
          return progress based on the amount of data processed so far.
protected  boolean initNextRecordReader()
          Get the record reader for the next chunk in this CombineFileSplit.
 boolean next(K key, V value)
          Reads the next key/value pair from the input for processing.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

split

protected CombineFileSplit split

jc

protected JobConf jc

reporter

protected Reporter reporter

rrClass

protected Class<RecordReader<K,V>> rrClass

rrConstructor

protected Constructor<RecordReader<K,V>> rrConstructor

fs

protected FileSystem fs

idx

protected int idx

progress

protected long progress

curReader

protected RecordReader<K,V> curReader
Constructor Detail

CombineFileRecordReader

public CombineFileRecordReader(JobConf job,
                               CombineFileSplit split,
                               Reporter reporter,
                               Class<RecordReader<K,V>> rrClass)
                        throws IOException
A generic RecordReader that can hand out different recordReaders for each chunk in the CombineFileSplit.

Throws:
IOException
Method Detail

next

public boolean next(K key,
                    V value)
             throws IOException
Description copied from interface: RecordReader
Reads the next key/value pair from the input for processing.

Specified by:
next in interface RecordReader<K,V>
Parameters:
key - the key to read data into
value - the value to read data into
Returns:
true iff a key/value was read, false if at EOF
Throws:
IOException

createKey

public K createKey()
Description copied from interface: RecordReader
Create an object of the appropriate type to be used as a key.

Specified by:
createKey in interface RecordReader<K,V>
Returns:
a new key object.

createValue

public V createValue()
Description copied from interface: RecordReader
Create an object of the appropriate type to be used as a value.

Specified by:
createValue in interface RecordReader<K,V>
Returns:
a new value object.

getPos

public long getPos()
            throws IOException
return the amount of data processed

Specified by:
getPos in interface RecordReader<K,V>
Returns:
the current position in the input.
Throws:
IOException

close

public void close()
           throws IOException
Description copied from interface: RecordReader
Close this InputSplit to future operations.

Specified by:
close in interface RecordReader<K,V>
Throws:
IOException

getProgress

public float getProgress()
                  throws IOException
return progress based on the amount of data processed so far.

Specified by:
getProgress in interface RecordReader<K,V>
Returns:
progress from 0.0 to 1.0.
Throws:
IOException

initNextRecordReader

protected boolean initNextRecordReader()
                                throws IOException
Get the record reader for the next chunk in this CombineFileSplit.

Throws:
IOException


Copyright © 2009 The Apache Software Foundation