@InterfaceAudience.Public @InterfaceStability.Stable public abstract class FileSystem extends Configured implements Closeable
All user code that may potentially use the Hadoop Distributed File System should be written to use a FileSystem object. The Hadoop DFS is a multi-machine system that appears as a single disk. It's useful because of its fault tolerance and potentially very large capacity.
The local implementation is LocalFileSystem and distributed
implementation is DistributedFileSystem.
| Modifier and Type | Field and Description |
|---|---|
static String |
DEFAULT_FS |
static String |
FS_DEFAULT_NAME_KEY |
static org.apache.commons.logging.Log |
LOG |
static int |
SHUTDOWN_HOOK_PRIORITY
Priority of the FileSystem shutdown hook.
|
protected org.apache.hadoop.fs.FileSystem.Statistics |
statistics
The statistics for this file system.
|
static String |
TRASH_PREFIX |
| Modifier | Constructor and Description |
|---|---|
protected |
FileSystem() |
| Modifier and Type | Method and Description |
|---|---|
Token<?>[] |
addDelegationTokens(String renewer,
Credentials credentials)
Obtain all delegation tokens used by this FileSystem that are not
already present in the given Credentials.
|
FSDataOutputStream |
append(Path f)
Append to an existing file (optional operation).
|
FSDataOutputStream |
append(Path f,
int bufferSize)
Append to an existing file (optional operation).
|
abstract FSDataOutputStream |
append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation).
|
static boolean |
areSymlinksEnabled() |
boolean |
cancelDeleteOnExit(Path f)
Cancel the deletion of the path when the FileSystem is closed
|
protected URI |
canonicalizeUri(URI uri)
Canonicalize the given URI.
|
protected void |
checkPath(Path path)
Check that a Path belongs to this FileSystem.
|
static void |
clearStatistics()
Reset all statistics for all file systems
|
void |
close()
No more filesystem operations are needed.
|
static void |
closeAll()
Close all cached filesystems.
|
static void |
closeAllForUGI(UserGroupInformation ugi)
Close all cached filesystems for a given UGI.
|
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target.
|
void |
concat(Path trg,
Path[] psrcs)
Concat existing files together.
|
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path[] srcs,
Path dst)
The src files are on the local disk.
|
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path src,
Path dst)
The src file is on the local disk.
|
void |
copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is on the local disk.
|
void |
copyFromLocalFile(Path src,
Path dst)
The src file is on the local disk.
|
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst,
boolean useRawLocalFileSystem)
The src file is under FS, and the dst is on the local disk.
|
void |
copyToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
static FSDataOutputStream |
create(FileSystem fs,
Path file,
FsPermission permission)
create a file with the provided permission
The permission of the file is set to be the provided permission as in
setPermission, not permission&~umask
It is implemented using two RPCs.
|
FSDataOutputStream |
create(Path f)
Create an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite)
Create an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize)
Create an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Create an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
abstract FSDataOutputStream |
create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
FsPermission permission,
EnumSet<CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
FsPermission permission,
EnumSet<CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
Create an FSDataOutputStream at the indicated Path with a custom
checksum option
|
FSDataOutputStream |
create(Path f,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
short replication)
Create an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
short replication,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
boolean |
createNewFile(Path f)
Creates the given Path as a brand-new zero-length file.
|
FSDataOutputStream |
createNonRecursive(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
createNonRecursive(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
createNonRecursive(Path f,
FsPermission permission,
EnumSet<CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
Path |
createSnapshot(Path path)
Create a snapshot with a default name.
|
Path |
createSnapshot(Path path,
String snapshotName)
Create a snapshot
|
void |
createSymlink(Path target,
Path link,
boolean createParent)
|
boolean |
delete(Path f)
Deprecated.
Use
delete(Path, boolean) instead. |
abstract boolean |
delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
deleteOnExit(Path f)
Mark a path to be deleted when FileSystem is closed.
|
void |
deleteSnapshot(Path path,
String snapshotName)
Delete a snapshot of a directory
|
static void |
enableSymlinks() |
boolean |
exists(Path f)
Check if exists.
|
protected Path |
fixRelativePart(Path p)
|
static FileSystem |
get(Configuration conf)
Returns the configured filesystem implementation.
|
static FileSystem |
get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
get(URI uri,
Configuration conf,
String user)
Get a filesystem instance based on the uri, the passed
configuration and the user
|
AclStatus |
getAclStatus(Path path)
Gets the ACL of a file or directory.
|
static List<org.apache.hadoop.fs.FileSystem.Statistics> |
getAllStatistics()
Deprecated.
|
Collection<? extends BlockStoragePolicySpi> |
getAllStoragePolicies()
Retrieve all the storage policies supported by this file system.
|
long |
getBlockSize(Path f)
Deprecated.
Use getFileStatus() instead
|
String |
getCanonicalServiceName()
Get a canonical service name for this file system.
|
protected URI |
getCanonicalUri()
Return a canonicalized form of this FileSystem's URI.
|
ContentSummary |
getContentSummary(Path f)
Return the
ContentSummary of a given Path. |
long |
getDefaultBlockSize()
Deprecated.
use
getDefaultBlockSize(Path) instead |
long |
getDefaultBlockSize(Path f)
Return the number of bytes that large input files should be optimally
be split into to minimize i/o time.
|
protected int |
getDefaultPort()
Get the default port for this file system.
|
short |
getDefaultReplication()
Deprecated.
use
getDefaultReplication(Path) instead |
short |
getDefaultReplication(Path path)
Get the default replication for a path.
|
static URI |
getDefaultUri(Configuration conf)
Get the default filesystem URI from a configuration.
|
BlockLocation[] |
getFileBlockLocations(FileStatus file,
long start,
long len)
Return an array containing hostnames, offset and size of
portions of the given file.
|
BlockLocation[] |
getFileBlockLocations(Path p,
long start,
long len)
Return an array containing hostnames, offset and size of
portions of the given file.
|
FileChecksum |
getFileChecksum(Path f)
Get the checksum of a file.
|
FileChecksum |
getFileChecksum(Path f,
long length)
Get the checksum of a file, from the beginning of the file till the
specific length.
|
FileStatus |
getFileLinkStatus(Path f)
|
abstract FileStatus |
getFileStatus(Path f)
Return a file status object that represents the path.
|
static Class<? extends FileSystem> |
getFileSystemClass(String scheme,
Configuration conf) |
protected static FileSystem |
getFSofPath(Path absOrFqPath,
Configuration conf) |
static GlobalStorageStatistics |
getGlobalStorageStatistics()
Get the global storage statistics.
|
Path |
getHomeDirectory()
Return the current user's home directory in this filesystem.
|
protected Path |
getInitialWorkingDirectory()
Note: with the new FilesContext class, getWorkingDirectory()
will be removed.
|
long |
getLength(Path f)
Deprecated.
Use getFileStatus() instead
|
Path |
getLinkTarget(Path f)
|
static LocalFileSystem |
getLocal(Configuration conf)
Get the local file system.
|
String |
getName()
Deprecated.
call #getUri() instead.
|
static FileSystem |
getNamed(String name,
Configuration conf)
Deprecated.
call #get(URI,Configuration) instead.
|
QuotaUsage |
getQuotaUsage(Path f)
Return the
QuotaUsage of a given Path. |
short |
getReplication(Path src)
Deprecated.
Use getFileStatus() instead
|
String |
getScheme()
Return the protocol scheme for the FileSystem.
|
FsServerDefaults |
getServerDefaults()
Deprecated.
use
getServerDefaults(Path) instead |
FsServerDefaults |
getServerDefaults(Path p)
Return a set of server default configuration values
|
static Map<String,org.apache.hadoop.fs.FileSystem.Statistics> |
getStatistics()
Deprecated.
|
static org.apache.hadoop.fs.FileSystem.Statistics |
getStatistics(String scheme,
Class<? extends FileSystem> cls)
Deprecated.
|
FsStatus |
getStatus()
Returns a status object describing the use and capacity of the
file system.
|
FsStatus |
getStatus(Path p)
Returns a status object describing the use and capacity of the
file system.
|
BlockStoragePolicySpi |
getStoragePolicy(Path src)
Query the effective storage policy ID for the given file or directory.
|
StorageStatistics |
getStorageStatistics()
Get the StorageStatistics for this FileSystem object.
|
Path |
getTrashRoot(Path path)
Get the root directory of Trash for current user when the path specified
is deleted.
|
Collection<FileStatus> |
getTrashRoots(boolean allUsers)
Get all the trash roots for current user or all users.
|
abstract URI |
getUri()
Returns a URI whose scheme and authority identify this FileSystem.
|
long |
getUsed()
Return the total size of all files in the filesystem.
|
long |
getUsed(Path path)
Return the total size of all files from a specified path.
|
abstract Path |
getWorkingDirectory()
Get the current working directory for the given file system
|
byte[] |
getXAttr(Path path,
String name)
Get an xattr name and value for a file or directory.
|
Map<String,byte[]> |
getXAttrs(Path path)
Get all of the xattr name/value pairs for a file or directory.
|
Map<String,byte[]> |
getXAttrs(Path path,
List<String> names)
Get all of the xattrs name/value pairs for a file or directory.
|
FileStatus[] |
globStatus(Path pathPattern)
Return all the files that match filePattern and are not checksum
files.
|
FileStatus[] |
globStatus(Path pathPattern,
PathFilter filter)
Return an array of FileStatus objects whose path names match pathPattern
and is accepted by the user-supplied path filter.
|
void |
initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
boolean |
isDirectory(Path f)
True iff the named path is a directory.
|
boolean |
isFile(Path f)
True iff the named path is a regular file.
|
org.apache.hadoop.fs.RemoteIterator<Path> |
listCorruptFileBlocks(Path path) |
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> |
listFiles(Path f,
boolean recursive)
List the statuses and block locations of the files in the given path.
|
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> |
listLocatedStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
protected org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> |
listLocatedStatus(Path f,
PathFilter filter)
Listing a directory
The returned results include its block location if it is a file
The results are filtered by the given path filter
|
abstract FileStatus[] |
listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
FileStatus[] |
listStatus(Path[] files)
Filter files/directories in the given list of paths using default
path filter.
|
FileStatus[] |
listStatus(Path[] files,
PathFilter filter)
Filter files/directories in the given list of paths using user-supplied
path filter.
|
FileStatus[] |
listStatus(Path f,
PathFilter filter)
Filter files/directories in the given path using the user-supplied path
filter.
|
org.apache.hadoop.fs.RemoteIterator<FileStatus> |
listStatusIterator(Path p)
Returns a remote iterator so that followup calls are made on demand
while consuming the entries.
|
List<String> |
listXAttrs(Path path)
Get all of the xattr names for a file or directory.
|
Path |
makeQualified(Path path)
Make sure that a path specifies a FileSystem.
|
static boolean |
mkdirs(FileSystem fs,
Path dir,
FsPermission permission)
create a directory with the provided permission
The permission of the directory is set to be the provided permission as in
setPermission, not permission&~umask
|
boolean |
mkdirs(Path f)
Call
mkdirs(Path, FsPermission) with default permission. |
abstract boolean |
mkdirs(Path f,
FsPermission permission)
Make the given file and all non-existent parents into
directories.
|
void |
modifyAclEntries(Path path,
List<AclEntry> aclSpec)
Modifies ACL entries of files and directories.
|
void |
moveFromLocalFile(Path[] srcs,
Path dst)
The src files is on the local disk.
|
void |
moveFromLocalFile(Path src,
Path dst)
The src file is on the local disk.
|
void |
moveToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
static FileSystem |
newInstance(Configuration conf)
Returns a unique configured filesystem implementation.
|
static FileSystem |
newInstance(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
newInstance(URI uri,
Configuration conf,
String user)
Returns the FileSystem for this URI's scheme and authority and the
passed user.
|
static LocalFileSystem |
newInstanceLocal(Configuration conf)
Get a unique local file system object
|
FSDataInputStream |
open(Path f)
Opens an FSDataInputStream at the indicated Path.
|
abstract FSDataInputStream |
open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
protected FSDataOutputStream |
primitiveCreate(Path f,
FsPermission absolutePermission,
EnumSet<CreateFlag> flag,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
Deprecated.
|
protected boolean |
primitiveMkdir(Path f,
FsPermission absolutePermission)
Deprecated.
|
protected void |
primitiveMkdir(Path f,
FsPermission absolutePermission,
boolean createParent)
Deprecated.
|
static void |
printStatistics()
Print all statistics for all file systems
|
protected void |
processDeleteOnExit()
Delete all files that were marked as delete-on-exit.
|
void |
removeAcl(Path path)
Removes all but the base ACL entries of files and directories.
|
void |
removeAclEntries(Path path,
List<AclEntry> aclSpec)
Removes ACL entries from files and directories.
|
void |
removeDefaultAcl(Path path)
Removes all default ACL entries from files and directories.
|
void |
removeXAttr(Path path,
String name)
Remove an xattr of a file or directory.
|
abstract boolean |
rename(Path src,
Path dst)
Renames Path src to Path dst.
|
protected void |
rename(Path src,
Path dst,
org.apache.hadoop.fs.Options.Rename... options)
Deprecated.
|
void |
renameSnapshot(Path path,
String snapshotOldName,
String snapshotNewName)
Rename a snapshot
|
protected Path |
resolveLink(Path f)
|
Path |
resolvePath(Path p)
Return the fully-qualified path of path f resolving the path
through any symlinks or mount point
|
void |
setAcl(Path path,
List<AclEntry> aclSpec)
Fully replaces ACL of files and directories, discarding all existing
entries.
|
static void |
setDefaultUri(Configuration conf,
String uri)
Set the default filesystem URI in a configuration.
|
static void |
setDefaultUri(Configuration conf,
URI uri)
Set the default filesystem URI in a configuration.
|
void |
setOwner(Path p,
String username,
String groupname)
Set owner of a path (i.e.
|
void |
setPermission(Path p,
FsPermission permission)
Set permission of a path.
|
boolean |
setReplication(Path src,
short replication)
Set replication for an existing file.
|
void |
setStoragePolicy(Path src,
String policyName)
Set the storage policy for a given file or directory.
|
void |
setTimes(Path p,
long mtime,
long atime)
Set access time of a file
|
void |
setVerifyChecksum(boolean verifyChecksum)
Set the verify checksum flag.
|
abstract void |
setWorkingDirectory(Path new_dir)
Set the current working directory for the given file system.
|
void |
setWriteChecksum(boolean writeChecksum)
Set the write checksum flag.
|
void |
setXAttr(Path path,
String name,
byte[] value)
Set an xattr of a file or directory.
|
void |
setXAttr(Path path,
String name,
byte[] value,
EnumSet<XAttrSetFlag> flag)
Set an xattr of a file or directory.
|
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to.
|
boolean |
supportsSymlinks()
|
boolean |
truncate(Path f,
long newLength)
Truncate the file in the indicated path to the indicated size.
|
void |
unsetStoragePolicy(Path src)
Unset the storage policy set for a given file or directory.
|
getConf, setConfpublic static final String FS_DEFAULT_NAME_KEY
public static final String DEFAULT_FS
public static final org.apache.commons.logging.Log LOG
public static final int SHUTDOWN_HOOK_PRIORITY
public static final String TRASH_PREFIX
protected org.apache.hadoop.fs.FileSystem.Statistics statistics
public static FileSystem get(URI uri, Configuration conf, String user) throws IOException, InterruptedException
uri - of the filesystemconf - the configuration to useuser - to perform the get asIOExceptionInterruptedExceptionpublic static FileSystem get(Configuration conf) throws IOException
conf - the configuration to useIOExceptionpublic static URI getDefaultUri(Configuration conf)
conf - the configuration to usepublic static void setDefaultUri(Configuration conf, URI uri)
conf - the configuration to alteruri - the new default filesystem uripublic static void setDefaultUri(Configuration conf, String uri)
conf - the configuration to alteruri - the new default filesystem uripublic void initialize(URI name, Configuration conf) throws IOException
name - a uri whose authority section names the host, port, etc.
for this FileSystemconf - the configurationIOExceptionpublic String getScheme()
UnsupportedOperationException.public abstract URI getUri()
protected URI getCanonicalUri()
canonicalizeUri(URI)
on the filesystem's own URI, so subclasses typically only need to
implement that method.canonicalizeUri(URI)protected URI canonicalizeUri(URI uri)
NetUtils.getCanonicalUri(URI, int)protected int getDefaultPort()
protected static FileSystem getFSofPath(Path absOrFqPath, Configuration conf) throws UnsupportedFileSystemException, IOException
@InterfaceAudience.Public @InterfaceStability.Evolving public String getCanonicalServiceName()
SecurityUtil.buildDTServiceName(URI, int)@Deprecated public String getName()
@Deprecated public static FileSystem getNamed(String name, Configuration conf) throws IOException
IOExceptionpublic static LocalFileSystem getLocal(Configuration conf) throws IOException
conf - the configuration to configure the file system withIOExceptionpublic static FileSystem get(URI uri, Configuration conf) throws IOException
IOExceptionpublic static FileSystem newInstance(URI uri, Configuration conf, String user) throws IOException, InterruptedException
newInstance(URI, Configuration)uri - of the filesystemconf - the configuration to useuser - to perform the get asIOExceptionInterruptedExceptionpublic static FileSystem newInstance(URI uri, Configuration conf) throws IOException
IOExceptionpublic static FileSystem newInstance(Configuration conf) throws IOException
conf - the configuration to useIOExceptionpublic static LocalFileSystem newInstanceLocal(Configuration conf) throws IOException
conf - the configuration to configure the file system withIOExceptionpublic static void closeAll()
throws IOException
IOExceptionpublic static void closeAllForUGI(UserGroupInformation ugi) throws IOException
ugi - user group info to closeIOExceptionpublic Path makeQualified(Path path)
path - to use@InterfaceAudience.Public @InterfaceStability.Evolving public Token<?>[] addDelegationTokens(String renewer, Credentials credentials) throws IOException
renewer - the user allowed to renew the delegation tokenscredentials - cache in which to add new delegation tokensIOExceptionpublic static FSDataOutputStream create(FileSystem fs, Path file, FsPermission permission) throws IOException
fs - file system handlefile - the name of the file to be createdpermission - the permission of the fileIOExceptionpublic static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission) throws IOException
fs - file system handledir - the name of the directory to be createdpermission - the permission of the directoryIOExceptioncreate(FileSystem, Path, FsPermission)protected void checkPath(Path path)
path - to checkpublic BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws IOException
file - FilesStatus to get data fromstart - offset into the given filelen - length for which to get locations forIOExceptionpublic BlockLocation[] getFileBlockLocations(Path p, long start, long len) throws IOException
p - path is used to identify an FS since an FS could have
another FS that it could be delegating the call tostart - offset into the given filelen - length for which to get locations forIOException@Deprecated public FsServerDefaults getServerDefaults() throws IOException
getServerDefaults(Path) insteadIOExceptionpublic FsServerDefaults getServerDefaults(Path p) throws IOException
p - path is used to identify an FS since an FS could have
another FS that it could be delegating the call toIOExceptionpublic Path resolvePath(Path p) throws IOException
p - path to be resolvedFileNotFoundExceptionIOExceptionpublic abstract FSDataInputStream open(Path f, int bufferSize) throws IOException
f - the file name to openbufferSize - the size of the buffer to be used.IOExceptionpublic FSDataInputStream open(Path f) throws IOException
f - the file to openIOExceptionpublic FSDataOutputStream create(Path f) throws IOException
f - the file to createIOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite) throws IOException
f - the file to createoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an exception will be thrown.IOExceptionpublic FSDataOutputStream create(Path f, Progressable progress) throws IOException
f - the file to createprogress - to report progressIOExceptionpublic FSDataOutputStream create(Path f, short replication) throws IOException
f - the file to createreplication - the replication factorIOExceptionpublic FSDataOutputStream create(Path f, short replication, Progressable progress) throws IOException
f - the file to createreplication - the replication factorprogress - to report progressIOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize) throws IOException
f - the file name to createoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, Progressable progress) throws IOException
f - the path of the file to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.IOExceptionpublic abstract FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openpermission - overwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openpermission - flags - CreateFlags to use for this stream.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) throws IOException
f - the file name to openpermission - flags - CreateFlags to use for this stream.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - checksumOpt - checksum parameter. If null, the values
found in conf will be used.IOExceptionsetPermission(Path, FsPermission)@Deprecated protected FSDataOutputStream primitiveCreate(Path f, FsPermission absolutePermission, EnumSet<CreateFlag> flag, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) throws IOException
IOException@Deprecated protected boolean primitiveMkdir(Path f, FsPermission absolutePermission) throws IOException
IOException@Deprecated protected void primitiveMkdir(Path f, FsPermission absolutePermission, boolean createParent) throws IOException
IOExceptionpublic FSDataOutputStream createNonRecursive(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openpermission - overwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openpermission - flags - CreateFlags to use for this stream.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)public boolean createNewFile(Path f) throws IOException
f - path to use for createIOExceptionpublic FSDataOutputStream append(Path f) throws IOException
f - the existing file to be appended.IOExceptionpublic FSDataOutputStream append(Path f, int bufferSize) throws IOException
f - the existing file to be appended.bufferSize - the size of the buffer to be used.IOExceptionpublic abstract FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException
f - the existing file to be appended.bufferSize - the size of the buffer to be used.progress - for reporting progress if it is not null.IOExceptionpublic void concat(Path trg, Path[] psrcs) throws IOException
trg - the path to the target destination.psrcs - the paths to the sources to use for the concatenation.IOException@Deprecated public short getReplication(Path src) throws IOException
src - file nameIOExceptionpublic boolean setReplication(Path src, short replication) throws IOException
src - file namereplication - new replicationIOExceptionpublic abstract boolean rename(Path src, Path dst) throws IOException
src - path to be renameddst - new path after renameIOException - on failure@Deprecated protected void rename(Path src, Path dst, org.apache.hadoop.fs.Options.Rename... options) throws IOException
If OVERWRITE option is not passed as an argument, rename fails if the dst already exists.
If OVERWRITE option is passed as an argument, rename overwrites the dst if it is a file or an empty directory. Rename fails if dst is a non-empty directory.
Note that atomicity of rename is dependent on the file system implementation. Please refer to the file system documentation for details. This default implementation is non atomic.
This method is deprecated since it is a temporary method added to support the transition from FileSystem to FileContext for user applications.
src - path to be renameddst - new path after renameIOException - on failurepublic boolean truncate(Path f, long newLength) throws IOException
f - The path to the file to be truncatednewLength - The size the file is to be truncated totrue if the file has been truncated to the desired
newLength and is immediately available to be reused for
write operations such as append, or
false if a background process of adjusting the length of
the last block has been started, and clients should wait for it to
complete before proceeding with further file updates.IOException@Deprecated public boolean delete(Path f) throws IOException
delete(Path, boolean) instead.IOExceptionpublic abstract boolean delete(Path f, boolean recursive) throws IOException
f - the path to delete.recursive - if path is a directory and set to
true, the directory is deleted else throws an exception. In
case of a file the recursive can be set to either true or false.IOExceptionpublic boolean deleteOnExit(Path f) throws IOException
f - the path to delete.IOExceptionpublic boolean cancelDeleteOnExit(Path f)
f - the path to cancel deletionprotected void processDeleteOnExit()
public boolean exists(Path f) throws IOException
f - source fileIOExceptionpublic boolean isDirectory(Path f) throws IOException
f - path to checkIOExceptionpublic boolean isFile(Path f) throws IOException
f - path to checkIOException@Deprecated public long getLength(Path f) throws IOException
IOExceptionpublic ContentSummary getContentSummary(Path f) throws IOException
ContentSummary of a given Path.f - path to useIOExceptionpublic QuotaUsage getQuotaUsage(Path f) throws IOException
QuotaUsage of a given Path.f - path to useIOExceptionpublic abstract FileStatus[] listStatus(Path f) throws FileNotFoundException, IOException
Does not guarantee to return the List of files/directories status in a sorted order.
f - given pathFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic org.apache.hadoop.fs.RemoteIterator<Path> listCorruptFileBlocks(Path path) throws IOException
IOExceptionpublic FileStatus[] listStatus(Path f, PathFilter filter) throws FileNotFoundException, IOException
Does not guarantee to return the List of files/directories status in a sorted order.
f - a path namefilter - the user-supplied path filterFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic FileStatus[] listStatus(Path[] files) throws FileNotFoundException, IOException
Does not guarantee to return the List of files/directories status in a sorted order.
files - a list of pathsFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic FileStatus[] listStatus(Path[] files, PathFilter filter) throws FileNotFoundException, IOException
Does not guarantee to return the List of files/directories status in a sorted order.
files - a list of pathsfilter - the user-supplied path filterFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic FileStatus[] globStatus(Path pathPattern) throws IOException
Return all the files that match filePattern and are not checksum files. Results are sorted by their names.
A filename pattern is composed of regular characters and special pattern matching characters, which are:
pathPattern - a regular expression specifying a pth patternIOExceptionpublic FileStatus[] globStatus(Path pathPattern, PathFilter filter) throws IOException
pathPattern - a regular expression specifying the path patternfilter - a user-supplied path filterIOException - if any I/O error occurs when fetching file statuspublic org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f) throws FileNotFoundException, IOException
f - is the pathFileNotFoundException - If f does not existIOException - If an I/O error occurredprotected org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f, PathFilter filter) throws FileNotFoundException, IOException
f - a pathfilter - a path filterFileNotFoundException - if f does not existIOException - if any I/O error occurredpublic org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatusIterator(Path p) throws FileNotFoundException, IOException
p - target pathFileNotFoundExceptionIOExceptionpublic org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listFiles(Path f, boolean recursive) throws FileNotFoundException, IOException
f - is the pathrecursive - if the subdirectories need to be traversed recursivelyFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic Path getHomeDirectory()
public abstract void setWorkingDirectory(Path new_dir)
new_dir - public abstract Path getWorkingDirectory()
protected Path getInitialWorkingDirectory()
public boolean mkdirs(Path f) throws IOException
mkdirs(Path, FsPermission) with default permission.IOExceptionpublic abstract boolean mkdirs(Path f, FsPermission permission) throws IOException
f - path to createpermission - to apply to fIOExceptionpublic void copyFromLocalFile(Path src, Path dst) throws IOException
src - pathdst - pathIOExceptionpublic void moveFromLocalFile(Path[] srcs, Path dst) throws IOException
srcs - pathdst - pathIOExceptionpublic void moveFromLocalFile(Path src, Path dst) throws IOException
src - pathdst - pathIOExceptionpublic void copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
delSrc - whether to delete the srcsrc - pathdst - pathIOExceptionpublic void copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path[] srcs,
Path dst)
throws IOException
delSrc - whether to delete the srcoverwrite - whether to overwrite an existing filesrcs - array of paths which are sourcedst - pathIOExceptionpublic void copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path src,
Path dst)
throws IOException
delSrc - whether to delete the srcoverwrite - whether to overwrite an existing filesrc - pathdst - pathIOExceptionpublic void copyToLocalFile(Path src, Path dst) throws IOException
src - pathdst - pathIOExceptionpublic void moveToLocalFile(Path src, Path dst) throws IOException
src - pathdst - pathIOExceptionpublic void copyToLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
delSrc - whether to delete the srcsrc - pathdst - pathIOExceptionpublic void copyToLocalFile(boolean delSrc,
Path src,
Path dst,
boolean useRawLocalFileSystem)
throws IOException
delSrc - whether to delete the srcsrc - pathdst - pathuseRawLocalFileSystem - whether to use RawLocalFileSystem as local file system or not.IOException - - if any IO errorpublic Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
fsOutputFile - path of output filetmpLocalFile - path of local tmp fileIOExceptionpublic void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
fsOutputFile - path of output filetmpLocalFile - path to local tmp fileIOExceptionpublic void close()
throws IOException
close in interface Closeableclose in interface AutoCloseableIOExceptionpublic long getUsed()
throws IOException
IOExceptionpublic long getUsed(Path path) throws IOException
IOException@Deprecated public long getBlockSize(Path f) throws IOException
IOException@Deprecated public long getDefaultBlockSize()
getDefaultBlockSize(Path) insteadpublic long getDefaultBlockSize(Path f)
f - path of file@Deprecated public short getDefaultReplication()
getDefaultReplication(Path) insteadpublic short getDefaultReplication(Path path)
path - of the filepublic abstract FileStatus getFileStatus(Path f) throws IOException
f - The path we want information fromFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic void createSymlink(Path target, Path link, boolean createParent) throws AccessControlException, FileAlreadyExistsException, FileNotFoundException, ParentNotDirectoryException, UnsupportedFileSystemException, IOException
public FileStatus getFileLinkStatus(Path f) throws AccessControlException, FileNotFoundException, UnsupportedFileSystemException, IOException
public boolean supportsSymlinks()
public Path getLinkTarget(Path f) throws IOException
IOExceptionprotected Path resolveLink(Path f) throws IOException
IOExceptionpublic FileChecksum getFileChecksum(Path f) throws IOException
f - The file pathIOExceptionpublic FileChecksum getFileChecksum(Path f, long length) throws IOException
f - The file pathlength - The length of the file range for checksum calculationIOExceptionpublic void setVerifyChecksum(boolean verifyChecksum)
verifyChecksum - public void setWriteChecksum(boolean writeChecksum)
writeChecksum - public FsStatus getStatus() throws IOException
IOException - see specific implementationpublic FsStatus getStatus(Path p) throws IOException
p - Path for which status should be obtained. null means
the default partition.IOException - see specific implementationpublic void setPermission(Path p, FsPermission permission) throws IOException
p - permission - IOExceptionpublic void setOwner(Path p, String username, String groupname) throws IOException
p - The pathusername - If it is null, the original username remains unchanged.groupname - If it is null, the original groupname remains unchanged.IOExceptionpublic void setTimes(Path p, long mtime, long atime) throws IOException
p - The pathmtime - Set the modification time of this file.
The number of milliseconds since Jan 1, 1970.
A value of -1 means that this call should not set modification time.atime - Set the access time of this file.
The number of milliseconds since Jan 1, 1970.
A value of -1 means that this call should not set access time.IOExceptionpublic final Path createSnapshot(Path path) throws IOException
path - The directory where snapshots will be taken.IOExceptionpublic Path createSnapshot(Path path, String snapshotName) throws IOException
path - The directory where snapshots will be taken.snapshotName - The name of the snapshotIOExceptionpublic void renameSnapshot(Path path, String snapshotOldName, String snapshotNewName) throws IOException
path - The directory path where the snapshot was takensnapshotOldName - Old name of the snapshotsnapshotNewName - New name of the snapshotIOExceptionpublic void deleteSnapshot(Path path, String snapshotName) throws IOException
path - The directory that the to-be-deleted snapshot belongs tosnapshotName - The name of the snapshotIOExceptionpublic void modifyAclEntries(Path path, List<AclEntry> aclSpec) throws IOException
path - Path to modifyaclSpec - ListIOException - if an ACL could not be modifiedpublic void removeAclEntries(Path path, List<AclEntry> aclSpec) throws IOException
path - Path to modifyaclSpec - ListIOException - if an ACL could not be modifiedpublic void removeDefaultAcl(Path path) throws IOException
path - Path to modifyIOException - if an ACL could not be modifiedpublic void removeAcl(Path path) throws IOException
path - Path to modifyIOException - if an ACL could not be removedpublic void setAcl(Path path, List<AclEntry> aclSpec) throws IOException
path - Path to modifyaclSpec - ListIOException - if an ACL could not be modifiedpublic AclStatus getAclStatus(Path path) throws IOException
path - Path to getIOException - if an ACL could not be readpublic void setXAttr(Path path, String name, byte[] value) throws IOException
path - Path to modifyname - xattr name.value - xattr value.IOExceptionpublic void setXAttr(Path path, String name, byte[] value, EnumSet<XAttrSetFlag> flag) throws IOException
path - Path to modifyname - xattr name.value - xattr value.flag - xattr set flagIOExceptionpublic byte[] getXAttr(Path path, String name) throws IOException
path - Path to get extended attributename - xattr name.IOExceptionpublic Map<String,byte[]> getXAttrs(Path path) throws IOException
path - Path to get extended attributesIOExceptionpublic Map<String,byte[]> getXAttrs(Path path, List<String> names) throws IOException
path - Path to get extended attributesnames - XAttr names.IOExceptionpublic List<String> listXAttrs(Path path) throws IOException
path - Path to get extended attributesIOExceptionpublic void removeXAttr(Path path, String name) throws IOException
path - Path to remove extended attributename - xattr nameIOExceptionpublic void setStoragePolicy(Path src, String policyName) throws IOException
src - file or directory path.policyName - the name of the target storage policy. The list
of supported Storage policies can be retrieved
via getAllStoragePolicies().IOExceptionpublic void unsetStoragePolicy(Path src) throws IOException
src - file or directory path.IOExceptionpublic BlockStoragePolicySpi getStoragePolicy(Path src) throws IOException
src - file or directory path.IOExceptionpublic Collection<? extends BlockStoragePolicySpi> getAllStoragePolicies() throws IOException
IOExceptionpublic Path getTrashRoot(Path path)
path - the trash root of the path to be determined.public Collection<FileStatus> getTrashRoots(boolean allUsers)
allUsers - return trash roots for all users if true.public static Class<? extends FileSystem> getFileSystemClass(String scheme, Configuration conf) throws IOException
IOException@Deprecated public static Map<String,org.apache.hadoop.fs.FileSystem.Statistics> getStatistics()
getGlobalStorageStatistics()@Deprecated public static List<org.apache.hadoop.fs.FileSystem.Statistics> getAllStatistics()
getGlobalStorageStatistics()@Deprecated public static org.apache.hadoop.fs.FileSystem.Statistics getStatistics(String scheme, Class<? extends FileSystem> cls)
getGlobalStorageStatistics()cls - the class to lookuppublic static void clearStatistics()
public static void printStatistics()
throws IOException
IOExceptionpublic static boolean areSymlinksEnabled()
public static void enableSymlinks()
public StorageStatistics getStorageStatistics()
This is a default method which is intended to be overridden by subclasses. The default implementation returns an empty storage statistics object.
public static GlobalStorageStatistics getGlobalStorageStatistics()
Copyright © 2018 Apache Software Foundation. All rights reserved.