org.apache.hadoop.mapreduce.lib.join
Class CompositeRecordReader<K extends WritableComparable<?>,V extends Writable,X extends Writable>

java.lang.Object
  extended by org.apache.hadoop.mapreduce.RecordReader<K,V>
      extended by org.apache.hadoop.mapreduce.lib.join.ComposableRecordReader<K,X>
          extended by org.apache.hadoop.mapreduce.lib.join.CompositeRecordReader<K,V,X>
All Implemented Interfaces:
Closeable, Comparable<ComposableRecordReader<K,?>>, Configurable
Direct Known Subclasses:
JoinRecordReader, MultiFilterRecordReader

@InterfaceAudience.Public
@InterfaceStability.Stable
public abstract class CompositeRecordReader<K extends WritableComparable<?>,V extends Writable,X extends Writable>
extends ComposableRecordReader<K,X>
implements Configurable

A RecordReader that can effect joins of RecordReaders sharing a common key type and partitioning.


Field Summary
protected  Configuration conf
           
protected  org.apache.hadoop.mapreduce.lib.join.CompositeRecordReader.JoinCollector jc
           
protected  K key
           
protected  Class<? extends WritableComparable> keyclass
           
protected  ComposableRecordReader<K,? extends V>[] kids
           
protected  X value
           
 
Constructor Summary
CompositeRecordReader(int id, int capacity, Class<? extends WritableComparator> cmpcl)
          Create a RecordReader with capacity children to position id in the parent reader.
 
Method Summary
 void accept(org.apache.hadoop.mapreduce.lib.join.CompositeRecordReader.JoinCollector jc, K key)
          If key provided matches that of this Composite, give JoinCollector iterator over values it may emit.
 void add(ComposableRecordReader<K,? extends V> rr)
          Add a RecordReader to this collection.
 void close()
          Close all child RRs.
protected abstract  boolean combine(Object[] srcs, TupleWritable value)
           
 int compareTo(ComposableRecordReader<K,?> other)
          Implement Comparable contract (compare key of join or head of heap with that of another).
protected  K createKey()
          Create a new key common to all child RRs.
protected  TupleWritable createTupleWritable()
          Create a value to be used internally for joins.
protected  void fillJoinCollector(K iterkey)
          For all child RRs offering the key provided, obtain an iterator at that position in the JoinCollector.
protected  WritableComparator getComparator()
          Return comparator defining the ordering for RecordReaders in this composite.
 Configuration getConf()
          Return the configuration used by this object.
 K getCurrentKey()
          Get the current key
 X getCurrentValue()
          Get the current value.
protected abstract  ResetableIterator<X> getDelegate()
          Obtain an iterator over the child RRs apropos of the value type ultimately emitted from this join.
 float getProgress()
          Report progress as the minimum of all child RR progress.
protected  PriorityQueue<ComposableRecordReader<K,?>> getRecordReaderQueue()
          Return sorted list of RecordReaders for this composite.
 boolean hasNext()
          Return true if it is possible that this could emit more values.
 int id()
          Return the position in the collector this class occupies.
 void initialize(InputSplit split, TaskAttemptContext context)
          Called once at initialization.
 K key()
          Return the key for the current join or the value at the top of the RecordReader heap.
 void key(K key)
          Clone the key at the top of this RR into the given object.
 void setConf(Configuration conf)
          Set the configuration to be used by this object.
 void skip(K key)
          Pass skip key to child RRs.
 
Methods inherited from class org.apache.hadoop.mapreduce.RecordReader
nextKeyValue
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

conf

protected Configuration conf

keyclass

protected Class<? extends WritableComparable> keyclass

jc

protected final org.apache.hadoop.mapreduce.lib.join.CompositeRecordReader.JoinCollector jc

kids

protected final ComposableRecordReader<K extends WritableComparable<?>,? extends V extends Writable>[] kids

key

protected K extends WritableComparable<?> key

value

protected X extends Writable value
Constructor Detail

CompositeRecordReader

public CompositeRecordReader(int id,
                             int capacity,
                             Class<? extends WritableComparator> cmpcl)
                      throws IOException
Create a RecordReader with capacity children to position id in the parent reader. The id of a root CompositeRecordReader is -1 by convention, but relying on this is not recommended.

Throws:
IOException
Method Detail

combine

protected abstract boolean combine(Object[] srcs,
                                   TupleWritable value)

initialize

public void initialize(InputSplit split,
                       TaskAttemptContext context)
                throws IOException,
                       InterruptedException
Description copied from class: RecordReader
Called once at initialization.

Specified by:
initialize in class RecordReader<K extends WritableComparable<?>,X extends Writable>
Parameters:
split - the split that defines the range of records to read
context - the information about the task
Throws:
IOException
InterruptedException

id

public int id()
Return the position in the collector this class occupies.


setConf

public void setConf(Configuration conf)
Set the configuration to be used by this object.

Specified by:
setConf in interface Configurable

getConf

public Configuration getConf()
Return the configuration used by this object.

Specified by:
getConf in interface Configurable

getRecordReaderQueue

protected PriorityQueue<ComposableRecordReader<K,?>> getRecordReaderQueue()
Return sorted list of RecordReaders for this composite.


getComparator

protected WritableComparator getComparator()
Return comparator defining the ordering for RecordReaders in this composite.


add

public void add(ComposableRecordReader<K,? extends V> rr)
         throws IOException,
                InterruptedException
Add a RecordReader to this collection. The id() of a RecordReader determines where in the Tuple its entry will appear. Adding RecordReaders with the same id has undefined behavior.

Throws:
IOException
InterruptedException

key

public K key()
Return the key for the current join or the value at the top of the RecordReader heap.


key

public void key(K key)
         throws IOException
Clone the key at the top of this RR into the given object.

Throws:
IOException

getCurrentKey

public K getCurrentKey()
Description copied from class: RecordReader
Get the current key

Specified by:
getCurrentKey in class RecordReader<K extends WritableComparable<?>,X extends Writable>
Returns:
the current key or null if there is no current key

hasNext

public boolean hasNext()
Return true if it is possible that this could emit more values.


skip

public void skip(K key)
          throws IOException,
                 InterruptedException
Pass skip key to child RRs.

Throws:
IOException
InterruptedException

getDelegate

protected abstract ResetableIterator<X> getDelegate()
Obtain an iterator over the child RRs apropos of the value type ultimately emitted from this join.


accept

public void accept(org.apache.hadoop.mapreduce.lib.join.CompositeRecordReader.JoinCollector jc,
                   K key)
            throws IOException,
                   InterruptedException
If key provided matches that of this Composite, give JoinCollector iterator over values it may emit.

Throws:
IOException
InterruptedException

fillJoinCollector

protected void fillJoinCollector(K iterkey)
                          throws IOException,
                                 InterruptedException
For all child RRs offering the key provided, obtain an iterator at that position in the JoinCollector.

Throws:
IOException
InterruptedException

compareTo

public int compareTo(ComposableRecordReader<K,?> other)
Implement Comparable contract (compare key of join or head of heap with that of another).

Specified by:
compareTo in interface Comparable<ComposableRecordReader<K extends WritableComparable<?>,?>>

createKey

protected K createKey()
Create a new key common to all child RRs.

Throws:
ClassCastException - if key classes differ.

createTupleWritable

protected TupleWritable createTupleWritable()
Create a value to be used internally for joins.


getCurrentValue

public X getCurrentValue()
                                   throws IOException,
                                          InterruptedException
Get the current value.

Specified by:
getCurrentValue in class RecordReader<K extends WritableComparable<?>,X extends Writable>
Returns:
the object that was read
Throws:
IOException
InterruptedException

close

public void close()
           throws IOException
Close all child RRs.

Specified by:
close in interface Closeable
Specified by:
close in class RecordReader<K extends WritableComparable<?>,X extends Writable>
Throws:
IOException

getProgress

public float getProgress()
                  throws IOException,
                         InterruptedException
Report progress as the minimum of all child RR progress.

Specified by:
getProgress in class RecordReader<K extends WritableComparable<?>,X extends Writable>
Returns:
a number between 0.0 and 1.0 that is the fraction of the data read
Throws:
IOException
InterruptedException


Copyright © 2012 Apache Software Foundation. All Rights Reserved.