org.apache.hadoop.mapred.pipes
Class Submitter

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.mapred.pipes.Submitter
All Implemented Interfaces:
Configurable, Tool

@InterfaceAudience.Public
@InterfaceStability.Stable
public class Submitter
extends Configured
implements Tool

The main entry point and job submitter. It may either be used as a command line-based or API-based method to launch Pipes jobs.


Field Summary
static String EXECUTABLE
           
static String INPUT_FORMAT
           
static String INTERPRETOR
           
static String IS_JAVA_MAP
           
static String IS_JAVA_REDUCE
           
static String IS_JAVA_RR
           
static String IS_JAVA_RW
           
protected static org.apache.commons.logging.Log LOG
           
static String PARTITIONER
           
static String PORT
           
static String PRESERVE_COMMANDFILE
           
 
Constructor Summary
Submitter()
           
Submitter(Configuration conf)
           
 
Method Summary
static String getExecutable(JobConf conf)
          Get the URI of the application's executable.
static boolean getIsJavaMapper(JobConf conf)
          Check whether the job is using a Java Mapper.
static boolean getIsJavaRecordReader(JobConf conf)
          Check whether the job is using a Java RecordReader
static boolean getIsJavaRecordWriter(JobConf conf)
          Will the reduce use a Java RecordWriter?
static boolean getIsJavaReducer(JobConf conf)
          Check whether the job is using a Java Reducer.
static boolean getKeepCommandFile(JobConf conf)
          Does the user want to keep the command file for debugging? If this is true, pipes will write a copy of the command data to a file in the task directory named "downlink.data", which may be used to run the C++ program under the debugger.
static RunningJob jobSubmit(JobConf conf)
          Submit a job to the Map-Reduce framework.
static void main(String[] args)
          Submit a pipes job based on the command line arguments.
 int run(String[] args)
          Execute the command with the given arguments.
static RunningJob runJob(JobConf conf)
          Submit a job to the map/reduce cluster.
static void setExecutable(JobConf conf, String executable)
          Set the URI for the application's executable.
static void setIsJavaMapper(JobConf conf, boolean value)
          Set whether the Mapper is written in Java.
static void setIsJavaRecordReader(JobConf conf, boolean value)
          Set whether the job is using a Java RecordReader.
static void setIsJavaRecordWriter(JobConf conf, boolean value)
          Set whether the job will use a Java RecordWriter.
static void setIsJavaReducer(JobConf conf, boolean value)
          Set whether the Reducer is written in Java.
static void setKeepCommandFile(JobConf conf, boolean keep)
          Set whether to keep the command file for debugging
static RunningJob submitJob(JobConf conf)
          Deprecated. Use runJob(JobConf)
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface org.apache.hadoop.conf.Configurable
getConf, setConf
 

Field Detail

LOG

protected static final org.apache.commons.logging.Log LOG

PRESERVE_COMMANDFILE

public static final String PRESERVE_COMMANDFILE
See Also:
Constant Field Values

EXECUTABLE

public static final String EXECUTABLE
See Also:
Constant Field Values

INTERPRETOR

public static final String INTERPRETOR
See Also:
Constant Field Values

IS_JAVA_MAP

public static final String IS_JAVA_MAP
See Also:
Constant Field Values

IS_JAVA_RR

public static final String IS_JAVA_RR
See Also:
Constant Field Values

IS_JAVA_RW

public static final String IS_JAVA_RW
See Also:
Constant Field Values

IS_JAVA_REDUCE

public static final String IS_JAVA_REDUCE
See Also:
Constant Field Values

PARTITIONER

public static final String PARTITIONER
See Also:
Constant Field Values

INPUT_FORMAT

public static final String INPUT_FORMAT
See Also:
Constant Field Values

PORT

public static final String PORT
See Also:
Constant Field Values
Constructor Detail

Submitter

public Submitter()

Submitter

public Submitter(Configuration conf)
Method Detail

getExecutable

public static String getExecutable(JobConf conf)
Get the URI of the application's executable.

Parameters:
conf -
Returns:
the URI where the application's executable is located

setExecutable

public static void setExecutable(JobConf conf,
                                 String executable)
Set the URI for the application's executable. Normally this is a hdfs: location.

Parameters:
conf -
executable - The URI of the application's executable.

setIsJavaRecordReader

public static void setIsJavaRecordReader(JobConf conf,
                                         boolean value)
Set whether the job is using a Java RecordReader.

Parameters:
conf - the configuration to modify
value - the new value

getIsJavaRecordReader

public static boolean getIsJavaRecordReader(JobConf conf)
Check whether the job is using a Java RecordReader

Parameters:
conf - the configuration to check
Returns:
is it a Java RecordReader?

setIsJavaMapper

public static void setIsJavaMapper(JobConf conf,
                                   boolean value)
Set whether the Mapper is written in Java.

Parameters:
conf - the configuration to modify
value - the new value

getIsJavaMapper

public static boolean getIsJavaMapper(JobConf conf)
Check whether the job is using a Java Mapper.

Parameters:
conf - the configuration to check
Returns:
is it a Java Mapper?

setIsJavaReducer

public static void setIsJavaReducer(JobConf conf,
                                    boolean value)
Set whether the Reducer is written in Java.

Parameters:
conf - the configuration to modify
value - the new value

getIsJavaReducer

public static boolean getIsJavaReducer(JobConf conf)
Check whether the job is using a Java Reducer.

Parameters:
conf - the configuration to check
Returns:
is it a Java Reducer?

setIsJavaRecordWriter

public static void setIsJavaRecordWriter(JobConf conf,
                                         boolean value)
Set whether the job will use a Java RecordWriter.

Parameters:
conf - the configuration to modify
value - the new value to set

getIsJavaRecordWriter

public static boolean getIsJavaRecordWriter(JobConf conf)
Will the reduce use a Java RecordWriter?

Parameters:
conf - the configuration to check
Returns:
true, if the output of the job will be written by Java

getKeepCommandFile

public static boolean getKeepCommandFile(JobConf conf)
Does the user want to keep the command file for debugging? If this is true, pipes will write a copy of the command data to a file in the task directory named "downlink.data", which may be used to run the C++ program under the debugger. You probably also want to set JobConf.setKeepFailedTaskFiles(true) to keep the entire directory from being deleted. To run using the data file, set the environment variable "mapreduce.pipes.commandfile" to point to the file.

Parameters:
conf - the configuration to check
Returns:
will the framework save the command file?

setKeepCommandFile

public static void setKeepCommandFile(JobConf conf,
                                      boolean keep)
Set whether to keep the command file for debugging

Parameters:
conf - the configuration to modify
keep - the new value

submitJob

@Deprecated
public static RunningJob submitJob(JobConf conf)
                            throws IOException
Deprecated. Use runJob(JobConf)

Submit a job to the map/reduce cluster. All of the necessary modifications to the job to run under pipes are made to the configuration.

Parameters:
conf - the job to submit to the cluster (MODIFIED)
Throws:
IOException

runJob

public static RunningJob runJob(JobConf conf)
                         throws IOException
Submit a job to the map/reduce cluster. All of the necessary modifications to the job to run under pipes are made to the configuration.

Parameters:
conf - the job to submit to the cluster (MODIFIED)
Throws:
IOException

jobSubmit

public static RunningJob jobSubmit(JobConf conf)
                            throws IOException
Submit a job to the Map-Reduce framework. This returns a handle to the RunningJob which can be used to track the running-job.

Parameters:
conf - the job configuration.
Returns:
a handle to the RunningJob which can be used to track the running-job.
Throws:
IOException

run

public int run(String[] args)
        throws Exception
Description copied from interface: Tool
Execute the command with the given arguments.

Specified by:
run in interface Tool
Parameters:
args - command specific arguments.
Returns:
exit code.
Throws:
Exception

main

public static void main(String[] args)
                 throws Exception
Submit a pipes job based on the command line arguments.

Parameters:
args -
Throws:
Exception


Copyright © 2014 Apache Software Foundation. All Rights Reserved.