org.apache.hadoop.mapred.lib

Class MultithreadedMapRunner<K1,V1,K2,V2>

java.lang.Object

org.apache.hadoop.mapred.lib.MultithreadedMapRunner<K1,V1,K2,V2>

All Implemented Interfaces:

JobConfigurable, MapRunnable<K1,V1,K2,V2>

@InterfaceAudience.Public
@InterfaceStability.Stable
public class MultithreadedMapRunner<K1,V1,K2,V2>
extends Object
implements MapRunnable<K1,V1,K2,V2>

Multithreaded implementation for MapRunnable.

It can be used instead of the default implementation, of MapRunner, when the Map operation is not CPU bound in order to improve throughput.

Map implementations using this MapRunnable must be thread-safe.

The Map-Reduce job has to be configured to use this MapRunnable class (using the JobConf.setMapRunnerClass method) and the number of threads the thread-pool can use with the mapred.map.multithreadedrunner.threads property, its default value is 10 threads.

Constructor Summary

Constructors

Constructor and Description
MultithreadedMapRunner()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(JobConf jobConf) Initializes a new instance from a JobConf.
void	run(RecordReader<K1,V1> input, OutputCollector<K2,V2> output, Reporter reporter) Start mapping input <key, value> pairs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultithreadedMapRunner

public MultithreadedMapRunner()

Method Detail

configure

public void configure(JobConf jobConf)

Description copied from interface: JobConfigurable

Initializes a new instance from a JobConf.

Specified by:

configure in interface JobConfigurable

Parameters:

jobConf - the configuration

run

public void run(RecordReader<K1,V1> input,
       OutputCollector<K2,V2> output,
       Reporter reporter)
         throws IOException

Description copied from interface: MapRunnable

Start mapping input <key, value> pairs.

Mapping of input records to output records is complete when this method returns.

Specified by:

run in interface MapRunnable<K1,V1,K2,V2>

Parameters:

input - the RecordReader to read the input records.

output - the OutputCollector to collect the outputrecords.

reporter - Reporter to report progress, status-updates etc.

Throws:

IOException