Modifier and Type | Method and Description |
---|---|
Configuration |
Configured.getConf() |
Configuration |
Configurable.getConf()
Return the configuration used by this object.
|
Modifier and Type | Method and Description |
---|---|
void |
Configuration.addResource(Configuration conf)
Add a configuration resource.
|
static void |
Configuration.dumpConfiguration(Configuration config,
String propertyName,
Writer out)
Writes properties and their attributes (final and resource)
to the given
Writer . |
static void |
Configuration.dumpConfiguration(Configuration config,
Writer out)
Writes out all properties and their attributes (final and resource) to
the given
Writer , the format of the output would be, |
void |
Configured.setConf(Configuration conf) |
void |
Configurable.setConf(Configuration conf)
Set the configuration to be used by this object.
|
Constructor and Description |
---|
Configuration(Configuration other)
A new configuration with the same settings cloned from another.
|
Configured(Configuration conf)
Construct a Configured.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
KeyProvider.getConf()
Return the provider configuration.
|
Modifier and Type | Method and Description |
---|---|
abstract KeyProvider |
KeyProviderFactory.createProvider(URI providerName,
Configuration conf) |
static KeyProvider |
KeyProviderFactory.get(URI uri,
Configuration conf)
Create a KeyProvider based on a provided URI.
|
static List<KeyProvider> |
KeyProviderFactory.getProviders(Configuration conf) |
static org.apache.hadoop.crypto.key.KeyProvider.Options |
KeyProvider.options(Configuration conf)
A helper function to create an options object.
|
Constructor and Description |
---|
KeyProvider(Configuration conf)
Constructor.
|
Modifier and Type | Method and Description |
---|---|
static void |
DistributedCache.addLocalArchives(Configuration conf,
String str)
Deprecated.
|
static void |
DistributedCache.addLocalFiles(Configuration conf,
String str)
Deprecated.
|
static void |
DistributedCache.createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
Deprecated.
Internal to MapReduce framework. Use DistributedCacheManager
instead.
|
static FileStatus |
DistributedCache.getFileStatus(Configuration conf,
URI cache)
Deprecated.
|
static long |
DistributedCache.getTimestamp(Configuration conf,
URI cache)
Deprecated.
|
static void |
DistributedCache.setArchiveTimestamps(Configuration conf,
String timestamps)
Deprecated.
|
static void |
DistributedCache.setFileTimestamps(Configuration conf,
String timestamps)
Deprecated.
|
static void |
DistributedCache.setLocalArchives(Configuration conf,
String str)
Deprecated.
|
static void |
DistributedCache.setLocalFiles(Configuration conf,
String str)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
FilterFileSystem.getConf() |
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
FileStatus srcStatus,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copyMerge(FileSystem srcFS,
Path srcDir,
FileSystem dstFS,
Path dstFile,
boolean deleteSource,
Configuration conf,
String addString)
Deprecated.
|
static AbstractFileSystem |
AbstractFileSystem.createFileSystem(URI uri,
Configuration conf)
Create a file system instance for the specified uri using the conf.
|
static FileSystem |
FileSystem.get(Configuration conf)
Returns the configured filesystem implementation.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static AbstractFileSystem |
AbstractFileSystem.get(URI uri,
Configuration conf)
The main factory method for creating a file system.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf,
String user)
Get a filesystem instance based on the uri, the passed
configuration and the user
|
static URI |
FileSystem.getDefaultUri(Configuration conf)
Get the default filesystem URI from a configuration.
|
static FileContext |
FileContext.getFileContext(AbstractFileSystem defFS,
Configuration aConf)
Create a FileContext with specified FS as default using the specified
config.
|
static FileContext |
FileContext.getFileContext(Configuration aConf)
Create a FileContext using the passed config.
|
static FileContext |
FileContext.getFileContext(URI defaultFsUri,
Configuration aConf)
Create a FileContext for specified default URI using the specified config.
|
FileSystem |
Path.getFileSystem(Configuration conf)
Return the FileSystem that owns this Path.
|
static Class<? extends FileSystem> |
FileSystem.getFileSystemClass(String scheme,
Configuration conf) |
protected static FileSystem |
FileSystem.getFSofPath(Path absOrFqPath,
Configuration conf) |
static TrashPolicy |
TrashPolicy.getInstance(Configuration conf,
FileSystem fs)
Get an instance of the configured TrashPolicy based on the value
of the configuration parameter fs.trash.classname.
|
static TrashPolicy |
TrashPolicy.getInstance(Configuration conf,
FileSystem fs,
Path home)
Deprecated.
Use
TrashPolicy.getInstance(Configuration, FileSystem) instead. |
static LocalFileSystem |
FileSystem.getLocal(Configuration conf)
Get the local file system.
|
static FileContext |
FileContext.getLocalFSFileContext(Configuration aConf) |
static FileSystem |
FileSystem.getNamed(String name,
Configuration conf)
Deprecated.
call #get(URI,Configuration) instead.
|
void |
TrashPolicy.initialize(Configuration conf,
FileSystem fs)
Used to setup the trash policy.
|
abstract void |
TrashPolicy.initialize(Configuration conf,
FileSystem fs,
Path home)
Deprecated.
Use
TrashPolicy.initialize(Configuration, FileSystem) instead. |
void |
RawLocalFileSystem.initialize(URI uri,
Configuration conf) |
void |
LocalFileSystem.initialize(URI name,
Configuration conf) |
void |
FilterFileSystem.initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
void |
FileSystem.initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
static boolean |
Trash.moveToAppropriateTrash(FileSystem fs,
Path p,
Configuration conf)
In case of the symlinks or mount points, one has to move the appropriate
trashbin in the actual volume of the path p being deleted.
|
static FileSystem |
FileSystem.newInstance(Configuration conf)
Returns a unique configured filesystem implementation.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration conf,
String user)
Returns the FileSystem for this URI's scheme and authority and the
passed user.
|
static LocalFileSystem |
FileSystem.newInstanceLocal(Configuration conf)
Get a unique local file system object
|
void |
ChecksumFileSystem.setConf(Configuration conf) |
static void |
FileSystem.setDefaultUri(Configuration conf,
String uri)
Set the default filesystem URI in a configuration.
|
static void |
FileSystem.setDefaultUri(Configuration conf,
URI uri)
Set the default filesystem URI in a configuration.
|
Constructor and Description |
---|
Trash(Configuration conf)
Construct a trash can accessor.
|
Trash(FileSystem fs,
Configuration conf)
Construct a trash can accessor for the FileSystem provided.
|
Modifier and Type | Method and Description |
---|---|
protected AzureADTokenProvider |
AdlFileSystem.getCustomAccessTokenProvider(Configuration conf)
This method is provided for convenience for derived classes to define
custom
AzureADTokenProvider instance. |
void |
AdlFileSystem.initialize(URI storeUri,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
Modifier and Type | Method and Description |
---|---|
abstract void |
AzureADTokenProvider.initialize(Configuration configuration)
Initialize with supported configuration.
|
Modifier and Type | Method and Description |
---|---|
void |
NativeAzureFileSystem.initialize(URI uri,
Configuration conf) |
Constructor and Description |
---|
WasbFsck(Configuration conf) |
Constructor and Description |
---|
AzureFileSystemInstrumentation(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
FTPFileSystem.initialize(URI uri,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static FsPermission |
FsPermission.getUMask(Configuration conf)
Get the user file creation mask (umask)
UMASK_LABEL config param has umask value that is either symbolic
or octal. |
static void |
FsPermission.setUMask(Configuration conf,
FsPermission umask)
Set the user file creation mask (umask)
|
Modifier and Type | Method and Description |
---|---|
void |
S3FileSystem.initialize(URI uri,
Configuration conf)
Deprecated.
|
Constructor and Description |
---|
S3A(URI theUri,
Configuration conf) |
SimpleAWSCredentialsProvider(Configuration conf) |
TemporaryAWSCredentialsProvider(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
NativeS3FileSystem.initialize(URI uri,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
ViewFileSystem.initialize(URI theUri,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
Constructor and Description |
---|
ViewFileSystem(Configuration conf)
Convenience Constructor for apps to call directly
|
ViewFs(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
HAServiceProtocol |
HAServiceTarget.getHealthMonitorProxy(Configuration conf,
int timeoutMs)
Returns a proxy to connect to the target HA service for health monitoring.
|
HAServiceProtocol |
HAServiceTarget.getProxy(Configuration conf,
int timeoutMs) |
org.apache.hadoop.ha.ZKFCProtocol |
HAServiceTarget.getZKFCProxy(Configuration conf,
int timeoutMs) |
Modifier and Type | Method and Description |
---|---|
Configuration |
WritableComparator.getConf() |
Configuration |
ObjectWritable.getConf() |
Configuration |
GenericWritable.getConf() |
Configuration |
EnumSetWritable.getConf() |
Configuration |
AbstractMapWritable.getConf() |
Modifier and Type | Method and Description |
---|---|
static <T extends Writable> |
WritableUtils.clone(T orig,
Configuration conf)
Make a copy of a writable object using serialization to a buffer.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf)
Copies from one stream to another.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf,
boolean close)
Copies from one stream to another.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
org.apache.hadoop.io.SequenceFile.Writer.Option... opts)
Create a new Writer with the given options.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileContext fc,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
org.apache.hadoop.io.SequenceFile.Metadata metadata,
EnumSet<CreateFlag> createFlag,
org.apache.hadoop.fs.Options.CreateOpts... opts)
Construct the preferred type of SequenceFile Writer.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
boolean createParent,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
Progressable progress)
Deprecated.
|
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
static WritableComparator |
WritableComparator.get(Class<? extends WritableComparable> c,
Configuration conf)
Get a comparator for a
WritableComparable implementation. |
static org.apache.hadoop.io.SequenceFile.CompressionType |
SequenceFile.getDefaultCompressionType(Configuration job)
Get the compression type for the reduce outputs
|
static <K> K |
DefaultStringifier.load(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the object from the configuration.
|
static <K> K[] |
DefaultStringifier.loadArray(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the array of objects from the configuration.
|
static Class<?> |
ObjectWritable.loadClass(Configuration conf,
String className)
Find and load the class with given name className by first finding
it in the specified conf.
|
static Writable |
WritableFactories.newInstance(Class<? extends Writable> c,
Configuration conf)
Create a new instance of a class with a defined factory.
|
static Object |
ObjectWritable.readObject(DataInput in,
Configuration conf)
|
static Object |
ObjectWritable.readObject(DataInput in,
ObjectWritable objectWritable,
Configuration conf)
|
void |
WritableComparator.setConf(Configuration conf) |
void |
ObjectWritable.setConf(Configuration conf) |
void |
GenericWritable.setConf(Configuration conf) |
void |
EnumSetWritable.setConf(Configuration conf) |
void |
AbstractMapWritable.setConf(Configuration conf) |
static void |
SequenceFile.setDefaultCompressionType(Configuration job,
org.apache.hadoop.io.SequenceFile.CompressionType val)
Set the default compression type for sequence files.
|
static <K> void |
DefaultStringifier.store(Configuration conf,
K item,
String keyName)
Stores the item in the configuration with the given keyName.
|
static <K> void |
DefaultStringifier.storeArray(Configuration conf,
K[] items,
String keyName)
Stores the array of items in the configuration with the given keyName.
|
static void |
ObjectWritable.writeObject(DataOutput out,
Object instance,
Class declaredClass,
Configuration conf)
|
static void |
ObjectWritable.writeObject(DataOutput out,
Object instance,
Class declaredClass,
Configuration conf,
boolean allowCompactArrays)
|
Constructor and Description |
---|
DefaultStringifier(Configuration conf,
Class<T> c) |
WritableComparator(Class<? extends WritableComparable> keyClass,
Configuration conf,
boolean createInstances) |
Modifier and Type | Method and Description |
---|---|
Configuration |
DefaultCodec.getConf() |
Configuration |
BZip2Codec.getConf()
Return the configuration used by this object.
|
Modifier and Type | Method and Description |
---|---|
static List<Class<? extends CompressionCodec>> |
CompressionCodecFactory.getCodecClasses(Configuration conf)
Get the list of codecs discovered via a Java ServiceLoader, or
listed in the configuration.
|
static Compressor |
CodecPool.getCompressor(CompressionCodec codec,
Configuration conf)
Get a
Compressor for the given CompressionCodec from the
pool or a new one. |
void |
Compressor.reinit(Configuration conf)
Prepare the compressor to be used in a new stream with settings defined in
the given Configuration
|
static void |
CompressionCodecFactory.setCodecClasses(Configuration conf,
List<Class> classes)
Sets a list of codec classes in the configuration.
|
void |
DefaultCodec.setConf(Configuration conf) |
void |
BZip2Codec.setConf(Configuration conf)
Set the configuration to be used by this object.
|
Constructor and Description |
---|
CompressionCodecFactory(Configuration conf)
Find the codecs specified in the config value io.compression.codecs
and register them.
|
Modifier and Type | Method and Description |
---|---|
static void |
Server.bind(ServerSocket socket,
InetSocketAddress address,
int backlog,
Configuration conf,
String rangeConf) |
static int |
Client.getPingInterval(Configuration conf)
Get the ping interval from configuration;
If not set in the configuration, return the default value.
|
static int |
Client.getRpcTimeout(Configuration conf)
The time after which a RPC will timeout.
|
static int |
Client.getTimeout(Configuration conf)
Deprecated.
use
Client.getRpcTimeout(Configuration) instead |
void |
Server.refreshCallQueue(Configuration conf) |
void |
Server.refreshServiceAcl(Configuration conf,
org.apache.hadoop.security.authorize.PolicyProvider provider)
Refresh the service authorization ACL for the service handled by this server.
|
static void |
Client.setConnectTimeout(Configuration conf,
int timeout)
set the connection timeout value in configuration
|
static void |
Client.setPingInterval(Configuration conf,
int pingInterval)
set the ping interval value in configuration
|
Constructor and Description |
---|
Client(Class<? extends Writable> valueClass,
Configuration conf)
Construct an IPC client with the default SocketFactory
|
Client(Class<? extends Writable> valueClass,
Configuration conf,
SocketFactory factory)
Construct an IPC client whose values are of the given
Writable
class. |
Server(String bindAddress,
int port,
Class<? extends Writable> paramClass,
int handlerCount,
Configuration conf) |
Server(String bindAddress,
int port,
Class<? extends Writable> rpcRequestClass,
int handlerCount,
int numReaders,
int queueSizePerHandler,
Configuration conf,
String serverName,
SecretManager<? extends TokenIdentifier> secretManager) |
Server(String bindAddress,
int port,
Class<? extends Writable> rpcRequestClass,
int handlerCount,
int numReaders,
int queueSizePerHandler,
Configuration conf,
String serverName,
SecretManager<? extends TokenIdentifier> secretManager,
String portRangeConfig)
Constructs a server listening on the named port and address.
|
Modifier and Type | Class and Description |
---|---|
class |
JobConf
A map/reduce job configuration.
|
Modifier and Type | Field and Description |
---|---|
protected Configuration |
SequenceFileRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
Configuration |
RunningJob.getConfiguration()
Get the underlying job configuration
|
Modifier and Type | Method and Description |
---|---|
static int |
SkipBadRecords.getAttemptsToStartSkipping(Configuration conf)
Get the number of Task attempts AFTER which skip mode
will be kicked off.
|
static boolean |
SkipBadRecords.getAutoIncrMapperProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static boolean |
SkipBadRecords.getAutoIncrReducerProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
static long |
SkipBadRecords.getMapperMaxSkipRecords(Configuration conf)
Get the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static org.apache.hadoop.io.SequenceFile.Reader[] |
SequenceFileOutputFormat.getReaders(Configuration conf,
Path dir)
Open the output generated by this format.
|
static org.apache.hadoop.io.MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
static int |
FixedLengthInputFormat.getRecordLength(Configuration conf)
Get record length value
|
static long |
SkipBadRecords.getReducerMaxSkipGroups(Configuration conf)
Get the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
static Path |
SkipBadRecords.getSkipOutputPath(Configuration conf)
Get the directory to which skipped records are written.
|
static void |
SkipBadRecords.setAttemptsToStartSkipping(Configuration conf,
int attemptsToStartSkipping)
Set the number of Task attempts AFTER which skip mode
will be kicked off.
|
static void |
SkipBadRecords.setAutoIncrMapperProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static void |
SkipBadRecords.setAutoIncrReducerProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
static void |
SequenceFileInputFilter.setFilterClass(Configuration conf,
Class filterClass)
set the filter class
|
static void |
SkipBadRecords.setMapperMaxSkipRecords(Configuration conf,
long maxSkipRecs)
Set the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static void |
FixedLengthInputFormat.setRecordLength(Configuration conf,
int recordLength)
Set the length of each record
|
static void |
SkipBadRecords.setReducerMaxSkipGroups(Configuration conf,
long maxSkipGrps)
Set the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
Constructor and Description |
---|
JobClient(Configuration conf)
Build a job client with the given
Configuration ,
and connect to the default cluster |
JobClient(InetSocketAddress jobTrackAddr,
Configuration conf)
Build a job client, connect to the indicated job tracker.
|
JobConf(Configuration conf)
Construct a map/reduce job configuration.
|
JobConf(Configuration conf,
Class exampleClass)
Construct a map/reduce job configuration.
|
KeyValueLineRecordReader(Configuration job,
FileSplit split) |
SequenceFileAsTextRecordReader(Configuration conf,
FileSplit split) |
SequenceFileRecordReader(Configuration conf,
FileSplit split) |
Modifier and Type | Method and Description |
---|---|
Configuration |
WrappedRecordReader.getConf() |
Configuration |
CompositeRecordReader.getConf()
Return the configuration used by this object.
|
Modifier and Type | Method and Description |
---|---|
void |
WrappedRecordReader.setConf(Configuration conf) |
void |
CompositeRecordReader.setConf(Configuration conf)
Set the configuration to be used by this object.
|
Constructor and Description |
---|
CombineFileRecordReaderWrapper(FileInputFormat<K,V> inputFormat,
CombineFileSplit split,
Configuration conf,
Reporter reporter,
Integer idx) |
Constructor and Description |
---|
Submitter(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
JobContext.getConfiguration()
Return the configuration for the job.
|
Modifier and Type | Method and Description |
---|---|
static int |
Job.getCompletionPollInterval(Configuration conf)
The interval at which waitForCompletion() should check.
|
static Job |
Job.getInstance(Cluster ignored,
Configuration conf)
Deprecated.
|
static Job |
Job.getInstance(Configuration conf)
|
static Job |
Job.getInstance(Configuration conf,
String jobName)
|
static Job |
Job.getInstance(JobStatus status,
Configuration conf)
|
static int |
Job.getProgressPollInterval(Configuration conf)
The interval at which monitorAndPrintJob() prints status
|
static org.apache.hadoop.mapreduce.Job.TaskStatusFilter |
Job.getTaskOutputFilter(Configuration conf)
Get the task output filter.
|
static void |
Job.setTaskOutputFilter(Configuration conf,
org.apache.hadoop.mapreduce.Job.TaskStatusFilter newValue)
Modify the Configuration to set the task output filter.
|
Constructor and Description |
---|
Cluster(Configuration conf) |
Cluster(InetSocketAddress jobTrackAddr,
Configuration conf) |
Job(Configuration conf)
Deprecated.
|
Job(Configuration conf,
String jobName)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
static Configuration |
ValueAggregatorJob.setAggregatorDescriptors(Class<? extends ValueAggregatorDescriptor>[] descriptors) |
Modifier and Type | Method and Description |
---|---|
void |
ValueAggregatorDescriptor.configure(Configuration conf)
Configure the object
|
void |
ValueAggregatorBaseDescriptor.configure(Configuration conf)
get the input file name.
|
void |
UserDefinedValueAggregatorDescriptor.configure(Configuration conf)
Do nothing.
|
static Job |
ValueAggregatorJob.createValueAggregatorJob(Configuration conf,
String[] args)
Create an Aggregate based map/reduce job.
|
protected static ArrayList<ValueAggregatorDescriptor> |
ValueAggregatorJobBase.getAggregatorDescriptors(Configuration conf) |
protected static ValueAggregatorDescriptor |
ValueAggregatorJobBase.getValueAggregatorDescriptor(String spec,
Configuration conf) |
static void |
ValueAggregatorJobBase.setup(Configuration job) |
Constructor and Description |
---|
UserDefinedValueAggregatorDescriptor(String className,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static void |
ChainReducer.addMapper(Job job,
Class<? extends Mapper> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
Configuration mapperConf)
Adds a
Mapper class to the chain reducer. |
static void |
ChainMapper.addMapper(Job job,
Class<? extends Mapper> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
Configuration mapperConf)
Adds a
Mapper class to the chain mapper. |
static void |
ChainReducer.setReducer(Job job,
Class<? extends Reducer> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
Configuration reducerConf)
Sets the
Reducer class to the chain job. |
Modifier and Type | Method and Description |
---|---|
Configuration |
DBInputFormat.getConf() |
Configuration |
DBConfiguration.getConf() |
Modifier and Type | Method and Description |
---|---|
static void |
DBConfiguration.configureDB(Configuration job,
String driverClass,
String dbUrl)
Sets the DB access related fields in the JobConf.
|
static void |
DBConfiguration.configureDB(Configuration conf,
String driverClass,
String dbUrl,
String userName,
String passwd)
Sets the DB access related fields in the
Configuration . |
protected RecordReader<LongWritable,T> |
OracleDataDrivenDBInputFormat.createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Configuration conf) |
protected RecordReader<LongWritable,T> |
DBInputFormat.createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Configuration conf) |
protected RecordReader<LongWritable,T> |
DataDrivenDBInputFormat.createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Configuration conf) |
static void |
DataDrivenDBInputFormat.setBoundingQuery(Configuration conf,
String query)
Set the user-defined bounding query to use with a user-defined query.
|
void |
DBInputFormat.setConf(Configuration conf)
Set the configuration to be used by this object.
|
static void |
OracleDBRecordReader.setSessionTimeZone(Configuration conf,
Connection conn)
Set session time zone
|
List<InputSplit> |
TextSplitter.split(Configuration conf,
ResultSet results,
String colName)
This method needs to determine the splits between two user-provided strings.
|
List<InputSplit> |
IntegerSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
FloatSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
DBSplitter.split(Configuration conf,
ResultSet results,
String colName)
Given a ResultSet containing one record (and already advanced to that record)
with two columns (a low value, and a high value, both of the same type), determine
a set of splits that span the given values.
|
List<InputSplit> |
DateSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
BooleanSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
BigDecimalSplitter.split(Configuration conf,
ResultSet results,
String colName) |
Constructor and Description |
---|
DataDrivenDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table,
String dbProduct) |
DBConfiguration(Configuration job) |
DBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
MySQLDataDrivenDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
MySQLDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
OracleDataDrivenDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
OracleDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
Modifier and Type | Field and Description |
---|---|
protected Configuration |
SequenceFileRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
static int |
FixedLengthInputFormat.getRecordLength(Configuration conf)
Get record length value
|
static List<FileSplit> |
NLineInputFormat.getSplitsForFile(FileStatus status,
Configuration conf,
int numLinesPerSplit) |
static void |
FixedLengthInputFormat.setRecordLength(Configuration conf,
int recordLength)
Set the length of each record
|
Constructor and Description |
---|
KeyValueLineRecordReader(Configuration conf) |
Constructor and Description |
---|
ControlledJob(Configuration conf)
Construct a job.
|
Modifier and Type | Field and Description |
---|---|
protected Configuration |
CompositeRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
Configuration |
CompositeRecordReader.getConf()
Return the configuration used by this object.
|
Modifier and Type | Method and Description |
---|---|
void |
CompositeRecordReader.setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
CompositeInputFormat.setFormat(Configuration conf)
Interpret a given string as a composite expression.
|
Constructor and Description |
---|
JoinRecordReader(int id,
Configuration conf,
int capacity,
Class<? extends WritableComparator> cmpcl) |
MultiFilterRecordReader(int id,
Configuration conf,
int capacity,
Class<? extends WritableComparator> cmpcl) |
Modifier and Type | Method and Description |
---|---|
static org.apache.hadoop.io.MapFile.Reader[] |
MapFileOutputFormat.getReaders(Path dir,
Configuration conf)
Open the output generated by this format.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
TotalOrderPartitioner.getConf() |
Configuration |
KeyFieldBasedPartitioner.getConf() |
Configuration |
KeyFieldBasedComparator.getConf() |
Configuration |
BinaryPartitioner.getConf() |
Modifier and Type | Method and Description |
---|---|
static String |
TotalOrderPartitioner.getPartitionFile(Configuration conf)
Get the path to the SequenceFile storing the sorted partition keyset.
|
void |
TotalOrderPartitioner.setConf(Configuration conf)
Read in the partition file and build indexing data structures.
|
void |
KeyFieldBasedPartitioner.setConf(Configuration conf) |
void |
KeyFieldBasedComparator.setConf(Configuration conf) |
void |
BinaryPartitioner.setConf(Configuration conf) |
static void |
BinaryPartitioner.setLeftOffset(Configuration conf,
int offset)
Set the subarray to be used for partitioning to
bytes[offset:] in Python syntax. |
static void |
BinaryPartitioner.setOffsets(Configuration conf,
int left,
int right)
Set the subarray to be used for partitioning to
bytes[left:(right+1)] in Python syntax. |
static void |
TotalOrderPartitioner.setPartitionFile(Configuration conf,
Path p)
Set the path to the SequenceFile storing the sorted partition keyset.
|
static void |
BinaryPartitioner.setRightOffset(Configuration conf,
int offset)
Set the subarray to be used for partitioning to
bytes[:(offset+1)] in Python syntax. |
Constructor and Description |
---|
InputSampler(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static void |
TokenCache.cleanUpTokenReferral(Configuration conf)
Remove jobtoken referrals which don't make sense in the context
of the task execution.
|
static void |
TokenCache.obtainTokensForNamenodes(Credentials credentials,
Path[] ps,
Configuration conf)
Convenience method to obtain delegation tokens from namenodes
corresponding to the paths passed.
|
Constructor and Description |
---|
CLI(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected void |
HistoryFileManager.serviceInit(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
TableMapping.getConf() |
Configuration |
SocksSocketFactory.getConf() |
Configuration |
ScriptBasedMapping.getConf() |
Configuration |
AbstractDNSToSwitchMapping.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
TableMapping.setConf(Configuration conf) |
void |
SocksSocketFactory.setConf(Configuration conf) |
void |
ScriptBasedMapping.setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
AbstractDNSToSwitchMapping.setConf(Configuration conf) |
Constructor and Description |
---|
AbstractDNSToSwitchMapping(Configuration conf)
Create an instance, caching the configuration file.
|
ScriptBasedMapping(Configuration conf)
Create an instance from the given configuration
|
Modifier and Type | Method and Description |
---|---|
static UserGroupInformation.AuthenticationMethod |
SecurityUtil.getAuthenticationMethod(Configuration conf) |
static org.apache.hadoop.security.KerberosInfo |
SecurityUtil.getKerberosInfo(Class<?> protocol,
Configuration conf)
Look up the KerberosInfo for a given protocol.
|
static TokenInfo |
SecurityUtil.getTokenInfo(Class<?> protocol,
Configuration conf)
Look up the TokenInfo for a given protocol.
|
static void |
SecurityUtil.login(Configuration conf,
String keytabFileKey,
String userNameKey)
Login as a principal specified in config.
|
static void |
SecurityUtil.login(Configuration conf,
String keytabFileKey,
String userNameKey,
String hostname)
Login as a principal specified in config.
|
static Credentials |
Credentials.readTokenStorageFile(File filename,
Configuration conf)
Convenience method for reading a token storage file, and loading the Tokens
therein in the passed UGI
|
static Credentials |
Credentials.readTokenStorageFile(Path filename,
Configuration conf)
Convenience method for reading a token storage file, and loading the Tokens
therein in the passed UGI
|
static void |
SecurityUtil.setAuthenticationMethod(UserGroupInformation.AuthenticationMethod authenticationMethod,
Configuration conf) |
static void |
UserGroupInformation.setConfiguration(Configuration conf)
Set the static configuration for UGI.
|
void |
Credentials.writeTokenStorageFile(Path filename,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
abstract CredentialProvider |
CredentialProviderFactory.createProvider(URI providerName,
Configuration conf) |
static List<CredentialProvider> |
CredentialProviderFactory.getProviders(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
DefaultImpersonationProvider.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
DefaultImpersonationProvider.setConf(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static Map<String,String> |
XFrameOptionsFilter.getFilterParams(Configuration conf,
String confPrefix)
Constructs a mapping of configuration properties to be used for filter
initialization.
|
static Map<String,String> |
RestCsrfPreventionFilter.getFilterParams(Configuration conf,
String confPrefix)
Constructs a mapping of configuration properties to be used for filter
initialization.
|
Modifier and Type | Method and Description |
---|---|
void |
Token.cancel(Configuration conf)
Cancel this delegation token
|
abstract void |
TokenRenewer.cancel(Token<?> token,
Configuration conf)
Cancel the given token
|
void |
Token.TrivialRenewer.cancel(Token<?> token,
Configuration conf) |
long |
Token.renew(Configuration conf)
Renew this delegation token
|
abstract long |
TokenRenewer.renew(Token<?> token,
Configuration conf)
Renew the given token.
|
long |
Token.TrivialRenewer.renew(Token<?> token,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
Service.getConfig()
Get the configuration of this service.
|
Configuration |
AbstractService.getConfig() |
Modifier and Type | Method and Description |
---|---|
void |
Service.init(Configuration config)
Initialize the service.
|
void |
AbstractService.init(Configuration conf)
Initialize the service.
|
protected void |
CompositeService.serviceInit(Configuration conf) |
protected void |
AbstractService.serviceInit(Configuration conf)
All initialization code needed by a service.
|
protected void |
AbstractService.setConfig(Configuration conf)
Set the configuration for this service.
|
Constructor and Description |
---|
DistCp(Configuration configuration,
org.apache.hadoop.tools.DistCpOptions inputOptions)
Public Constructor.
|
Modifier and Type | Method and Description |
---|---|
static <T> T |
ReflectionUtils.copy(Configuration conf,
T src,
T dst)
Make a copy of the writable object using serialization to a buffer
|
static <T> T |
ReflectionUtils.newInstance(Class<T> theClass,
Configuration conf)
Create an object for the given class and initialize it from conf
|
static int |
ToolRunner.run(Configuration conf,
Tool tool,
String[] args)
Runs the given
Tool by Tool.run(String[]) , after
parsing with the given generic arguments. |
static void |
ReflectionUtils.setConf(Object theObject,
Configuration conf)
Check and set 'configuration' if necessary.
|
Constructor and Description |
---|
Client(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static <T> T |
AHSProxy.createAHSProxy(Configuration conf,
Class<T> protocol,
InetSocketAddress ahsAddress) |
static <T> T |
NMProxy.createNMProxy(Configuration conf,
Class<T> protocol,
UserGroupInformation ugi,
org.apache.hadoop.yarn.ipc.YarnRPC rpc,
InetSocketAddress serverAddress) |
protected static <T> T |
ServerProxy.createRetriableProxy(Configuration conf,
Class<T> protocol,
UserGroupInformation user,
org.apache.hadoop.yarn.ipc.YarnRPC rpc,
InetSocketAddress serverAddress,
org.apache.hadoop.io.retry.RetryPolicy retryPolicy) |
protected static org.apache.hadoop.io.retry.RetryPolicy |
ServerProxy.createRetryPolicy(Configuration conf,
String maxWaitTimeStr,
long defMaxWaitTime,
String connectRetryIntervalStr,
long defRetryInterval) |
static <T> T |
ClientRMProxy.createRMProxy(Configuration configuration,
Class<T> protocol)
Create a proxy to the ResourceManager for the specified protocol.
|
static <T> T |
RMProxy.createRMProxy(Configuration conf,
Class<T> protocol,
InetSocketAddress rmAddress)
Deprecated.
This method is deprecated and is not used by YARN internally any more.
To create a proxy to the RM, use ClientRMProxy#createRMProxy or
ServerRMProxy#createRMProxy.
Create a proxy to the ResourceManager at the specified address.
|
static Text |
ClientRMProxy.getAMRMTokenService(Configuration conf) |
protected static <T> T |
AHSProxy.getProxy(Configuration conf,
Class<T> protocol,
InetSocketAddress rmAddress) |
static Text |
ClientRMProxy.getRMDelegationTokenService(Configuration conf)
Get the token service name to be used for RMDelegationToken.
|
static Text |
ClientRMProxy.getTokenService(Configuration conf,
String address,
String defaultAddr,
int defaultPort) |
Modifier and Type | Class and Description |
---|---|
class |
YarnConfiguration |
Modifier and Type | Method and Description |
---|---|
static boolean |
YarnConfiguration.shouldRMFailFast(Configuration conf) |
static boolean |
YarnConfiguration.useHttps(Configuration conf) |
Constructor and Description |
---|
YarnConfiguration(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected void |
AsyncDispatcher.serviceInit(Configuration conf) |
Constructor and Description |
---|
AggregatedLogFormat.LogReader(Configuration conf,
Path remoteAppLogFile) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.security.KerberosInfo |
SchedulerSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
org.apache.hadoop.security.KerberosInfo |
ContainerManagerSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
SchedulerSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
ContainerManagerSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.security.KerberosInfo |
AdminSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
AdminSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.security.KerberosInfo |
ClientTimelineSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
org.apache.hadoop.security.KerberosInfo |
ClientRMSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
ClientTimelineSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
ClientRMSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static SharedCacheChecksum |
SharedCacheChecksumFactory.getChecksum(Configuration conf)
Get a new
SharedCacheChecksum object based on the configurable
algorithm implementation
(see yarn.sharedcache.checksum.algo.impl ) |
Modifier and Type | Method and Description |
---|---|
static ResourceCalculatorProcessTree |
ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(String pid,
Class<? extends ResourceCalculatorProcessTree> clazz,
Configuration conf)
Create the ResourceCalculatorProcessTree rooted to specified process
from the class name and configure it.
|
Modifier and Type | Method and Description |
---|---|
static Text |
TimelineUtils.buildTimelineTokenService(Configuration conf) |
static float |
TimelineUtils.getTimelineServiceVersion(Configuration conf)
Returns the timeline service version.
|
static InetSocketAddress |
TimelineUtils.getTimelineTokenServiceAddress(Configuration conf) |
static boolean |
TimelineUtils.timelineServiceEnabled(Configuration conf)
Returns whether the timeline service is enabled via configuration.
|
static boolean |
TimelineUtils.timelineServiceV1_5Enabled(Configuration conf)
Returns whether the timeline service v.1.5 is enabled via configuration.
|
Copyright © 2017 Apache Software Foundation. All rights reserved.