| Package | Description |
|---|---|
| org.apache.hadoop.fs |
Implementations of
AbstractFileSystem for hdfs
over rpc and hdfs over web. |
| org.apache.hadoop.fs.adl |
Supporting classes for metrics instrumentation.
|
| org.apache.hadoop.fs.azure |
A distributed implementation of
FileSystem for reading and writing files on
Azure Block Storage. |
| org.apache.hadoop.fs.ftp | |
| org.apache.hadoop.fs.s3 |
A distributed, block-based implementation of
FileSystem that uses Amazon S3
as a backing store. |
| org.apache.hadoop.fs.s3native |
A distributed implementation of
FileSystem for reading and writing files on
Amazon S3. |
| org.apache.hadoop.fs.viewfs | |
| org.apache.hadoop.io |
Generic i/o code for use when reading and writing data to the network,
to databases, and to files.
|
| org.apache.hadoop.mapred | |
| org.apache.hadoop.mapred.lib | |
| org.apache.hadoop.mapred.lib.db | |
| org.apache.hadoop.mapreduce | |
| org.apache.hadoop.mapreduce.lib.input | |
| org.apache.hadoop.metrics2.sink |
Builtin metrics sinks
|
| Modifier and Type | Class and Description |
|---|---|
class |
ChecksumFileSystem
Abstract Checksumed FileSystem.
|
class |
FilterFileSystem
A
FilterFileSystem contains
some other file system, which it uses as
its basic file system, possibly transforming
the data along the way or providing additional
functionality. |
class |
LocalFileSystem
Implement the FileSystem API for the checksumed local filesystem.
|
class |
RawLocalFileSystem
Implement the FileSystem API for the raw local filesystem.
|
| Modifier and Type | Field and Description |
|---|---|
protected FileSystem |
FilterFileSystem.fs |
protected FileSystem |
TrashPolicy.fs |
| Modifier and Type | Method and Description |
|---|---|
static FileSystem |
FileSystem.get(Configuration conf)
Returns the configured FileSystem implementation.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf)
Get a FileSystem for this URI's scheme and authority.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf,
String user)
Get a FileSystem instance based on the uri, the passed in
configuration and the user.
|
FileSystem[] |
FilterFileSystem.getChildFileSystems() |
FileSystem |
Path.getFileSystem(Configuration conf)
Return the FileSystem that owns this Path.
|
protected FileSystem |
FSDataOutputStreamBuilder.getFS() |
protected static FileSystem |
FileSystem.getFSofPath(Path absOrFqPath,
Configuration conf) |
static FileSystem |
FileSystem.getNamed(String name,
Configuration conf)
Deprecated.
call
get(URI, Configuration) instead. |
FileSystem |
LocalFileSystem.getRaw() |
FileSystem |
ChecksumFileSystem.getRawFileSystem()
get the raw file system
|
FileSystem |
FilterFileSystem.getRawFileSystem()
Get the raw file system
|
static FileSystem |
FileSystem.newInstance(Configuration conf)
Returns a unique configured FileSystem implementation for the default
filesystem of the supplied configuration.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration config)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration conf,
String user)
Returns the FileSystem for this URI's scheme and authority and the
given user.
|
| Modifier and Type | Method and Description |
|---|---|
static Class<? extends FileSystem> |
FileSystem.getFileSystemClass(String scheme,
Configuration conf)
Get the FileSystem implementation class of a filesystem.
|
| Modifier and Type | Method and Description |
|---|---|
static boolean |
FileUtil.compareFs(FileSystem srcFs,
FileSystem destFs) |
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
FileStatus srcStatus,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copyMerge(FileSystem srcFS,
Path srcDir,
FileSystem dstFS,
Path dstFile,
boolean deleteSource,
Configuration conf,
String addString)
Deprecated.
|
static FSDataOutputStream |
FileSystem.create(FileSystem fs,
Path file,
FsPermission permission)
Create a file with the provided permission.
|
static void |
FileUtil.fullyDelete(FileSystem fs,
Path dir)
Deprecated.
|
static TrashPolicy |
TrashPolicy.getInstance(Configuration conf,
FileSystem fs)
Get an instance of the configured TrashPolicy based on the value
of the configuration parameter fs.trash.classname.
|
static TrashPolicy |
TrashPolicy.getInstance(Configuration conf,
FileSystem fs,
Path home)
Deprecated.
Use
TrashPolicy.getInstance(Configuration, FileSystem) instead. |
void |
TrashPolicy.initialize(Configuration conf,
FileSystem fs)
Used to setup the trash policy.
|
abstract void |
TrashPolicy.initialize(Configuration conf,
FileSystem fs,
Path home)
Deprecated.
Use
TrashPolicy.initialize(Configuration, FileSystem) instead. |
Path |
Path.makeQualified(FileSystem fs)
Deprecated.
|
static boolean |
FileSystem.mkdirs(FileSystem fs,
Path dir,
FsPermission permission)
Create a directory with the provided permission.
|
static boolean |
Trash.moveToAppropriateTrash(FileSystem fs,
Path p,
Configuration conf)
In case of the symlinks or mount points, one has to move the appropriate
trashbin in the actual volume of the path p being deleted.
|
| Modifier and Type | Method and Description |
|---|---|
static org.apache.hadoop.fs.FileSystem.Statistics |
FileSystem.getStatistics(String scheme,
Class<? extends FileSystem> cls)
Deprecated.
|
| Constructor and Description |
|---|
ChecksumFileSystem(FileSystem fs) |
FilterFileSystem(FileSystem fs) |
FSDataOutputStreamBuilder(FileSystem fileSystem,
Path p)
Constructor.
|
LocalFileSystem(FileSystem rawLocalFileSystem) |
Trash(FileSystem fs,
Configuration conf)
Construct a trash can accessor for the FileSystem provided.
|
| Modifier and Type | Class and Description |
|---|---|
class |
AdlFileSystem
A FileSystem to access Azure Data Lake Store.
|
| Modifier and Type | Class and Description |
|---|---|
class |
NativeAzureFileSystem
A
FileSystem for reading and writing files stored on Windows Azure. |
| Modifier and Type | Method and Description |
|---|---|
void |
WasbFsck.setMockFileSystemForTesting(FileSystem fileSystem)
For testing purposes, set the file system to use here instead of relying on
getting it from the FileSystem class based on the URI.
|
| Modifier and Type | Class and Description |
|---|---|
class |
FTPFileSystem
A
FileSystem backed by an FTP client provided by Apache Commons Net. |
| Modifier and Type | Class and Description |
|---|---|
class |
S3FileSystem
Deprecated.
Use
NativeS3FileSystem and S3AFileSystem instead. |
| Modifier and Type | Class and Description |
|---|---|
class |
NativeS3FileSystem
A
FileSystem for reading and writing files stored on
Amazon S3. |
| Modifier and Type | Class and Description |
|---|---|
class |
ViewFileSystem
ViewFileSystem (extends the FileSystem interface) implements a client-side
mount table.
|
| Modifier and Type | Method and Description |
|---|---|
FileSystem[] |
ViewFileSystem.getChildFileSystems() |
| Modifier and Type | Method and Description |
|---|---|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
boolean createParent,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
Progressable progress)
Deprecated.
|
static void |
MapFile.delete(FileSystem fs,
String name)
Deletes the named map file.
|
static void |
BloomMapFile.delete(FileSystem fs,
String name) |
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
static void |
MapFile.rename(FileSystem fs,
String oldName,
String newName)
Renames an existing map directory.
|
| Modifier and Type | Method and Description |
|---|---|
FileSystem |
JobClient.getFs()
Get a filesystem handle.
|
| Modifier and Type | Method and Description |
|---|---|
protected void |
FileInputFormat.addInputPathRecursively(List<FileStatus> result,
FileSystem fs,
Path path,
PathFilter inputFilter)
Add files in the input path recursively into the results.
|
void |
OutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job)
Check for validity of the output-specification for the job.
|
void |
SequenceFileAsBinaryOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
FileOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
static org.apache.hadoop.io.MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
RecordWriter<K,V> |
OutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress)
Get the
RecordWriter for the given job. |
RecordWriter<BytesWritable,BytesWritable> |
SequenceFileAsBinaryOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
SequenceFileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
abstract RecordWriter<K,V> |
FileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
TextOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<WritableComparable,Writable> |
MapFileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
static boolean |
JobClient.isJobDirValid(Path jobDirPath,
FileSystem fs)
Checks if the job directory is clean and has all the required components
for (re) starting the job
|
protected boolean |
FileInputFormat.isSplitable(FileSystem fs,
Path filename)
Is the given filename splittable? Usually, true, but if the file is
stream compressed, it will not be.
|
protected boolean |
TextInputFormat.isSplitable(FileSystem fs,
Path file) |
protected boolean |
FixedLengthInputFormat.isSplitable(FileSystem fs,
Path file) |
protected boolean |
KeyValueTextInputFormat.isSplitable(FileSystem fs,
Path file) |
| Modifier and Type | Method and Description |
|---|---|
void |
LazyOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
FilterOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
NullOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
protected RecordWriter<K,V> |
MultipleSequenceFileOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
protected abstract RecordWriter<K,V> |
MultipleOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
protected RecordWriter<K,V> |
MultipleTextOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
RecordWriter<K,V> |
LazyOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
FilterOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
NullOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
MultipleOutputFormat.getRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3)
Create a composite record writer that can write key/value data to different
output files
|
protected boolean |
CombineFileInputFormat.isSplitable(FileSystem fs,
Path file) |
| Modifier and Type | Method and Description |
|---|---|
void |
DBOutputFormat.checkOutputSpecs(FileSystem filesystem,
JobConf job)
Check for validity of the output-specification for the job.
|
RecordWriter<K,V> |
DBOutputFormat.getRecordWriter(FileSystem filesystem,
JobConf job,
String name,
Progressable progress)
Get the
RecordWriter for the given job. |
| Modifier and Type | Method and Description |
|---|---|
FileSystem |
Cluster.getFileSystem()
Get the file system where job-specific files are stored
|
| Modifier and Type | Method and Description |
|---|---|
protected void |
FileInputFormat.addInputPathRecursively(List<FileStatus> result,
FileSystem fs,
Path path,
PathFilter inputFilter)
Add files in the input path recursively into the results.
|
protected BlockLocation[] |
CombineFileInputFormat.getFileBlockLocations(FileSystem fs,
FileStatus stat) |
| Modifier and Type | Field and Description |
|---|---|
protected static FileSystem |
RollingFileSystemSink.suppliedFilesystem |
Copyright © 2018 Apache Software Foundation. All rights reserved.