org.apache.hadoop.fs.viewfs

Class ViewFs

java.lang.Object

org.apache.hadoop.fs.AbstractFileSystem

org.apache.hadoop.fs.viewfs.ViewFs

@InterfaceAudience.Public
@InterfaceStability.Evolving
public class ViewFs
extends AbstractFileSystem

ViewFs (extends the AbstractFileSystem interface) implements a client-side mount table. The viewFs file system is implemented completely in memory on the client side. The client-side mount table allows a client to provide a customized view of a file system namespace that is composed from one or more individual file systems (a localFs or Hdfs, S3fs, etc). For example one could have a mount table that provides links such as

• /user -> hdfs://nnContainingUserDir/user
• /project/foo -> hdfs://nnProject1/projects/foo
• /project/bar -> hdfs://nnProject2/projects/bar
• /tmp -> hdfs://nnTmp/privateTmpForUserXXX

ViewFs is specified with the following URI: viewfs:///

To use viewfs one would typically set the default file system in the config (i.e. fs.default.name< = viewfs:///) along with the mount table config variables as described below.

** Config variables to specify the mount table entries **

The file system is initialized from the standard Hadoop config through config variables. See FsConstants for URI and Scheme constants; See Constants for config var constants; see ConfigUtil for convenient lib.

All the mount table config entries for view fs are prefixed by fs.viewfs.mounttable. For example the above example can be specified with the following config variables:

• fs.viewfs.mounttable.default.link./user= hdfs://nnContainingUserDir/user
• fs.viewfs.mounttable.default.link./project/foo= hdfs://nnProject1/projects/foo
• fs.viewfs.mounttable.default.link./project/bar= hdfs://nnProject2/projects/bar
• fs.viewfs.mounttable.default.link./tmp= hdfs://nnTmp/privateTmpForUserXXX

The default mount table (when no authority is specified) is from config variables prefixed by fs.viewFs.mounttable.default The authority component of a URI can be used to specify a different mount table. For example,

• viewfs://sanjayMountable/

is initialized from fs.viewFs.mounttable.sanjayMountable.* config variables.

**** Merge Mounts **** (NOTE: merge mounts are not implemented yet.)

One can also use "MergeMounts" to merge several directories (this is sometimes called union-mounts or junction-mounts in the literature. For example of the home directories are stored on say two file systems (because they do not fit on one) then one could specify a mount entry such as following merges two dirs:

• /user -> hdfs://nnUser1/user,hdfs://nnUser2/user

Such a mergeLink can be specified with the following config var where "," is used as the separator for each of links to be merged:

• fs.viewfs.mounttable.default.linkMerge./user= hdfs://nnUser1/user,hdfs://nnUser1/user

A special case of the merge mount is where mount table's root is merged with the root (slash) of another file system:

• fs.viewfs.mounttable.default.linkMergeSlash=hdfs://nn99/

In this cases the root of the mount table is merged with the root of hdfs://nn99/

Field Summary

Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

statistics

Constructor Summary

Constructors

Constructor and Description
ViewFs(Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
FSDataOutputStream	createInternal(Path f, EnumSet<CreateFlag> flag, FsPermission absolutePermission, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent) The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.
void	createSymlink(Path target, Path link, boolean createParent) The specification of this method matches that of FileContext.createSymlink(Path, Path, boolean);
boolean	delete(Path f, boolean recursive) The specification of this method matches that of FileContext.delete(Path, boolean) except that Path f must be for this file system.
List<org.apache.hadoop.security.token.Token<?>>	getDelegationTokens(String renewer) Get one or more delegation tokens associated with the filesystem.
BlockLocation[]	getFileBlockLocations(Path f, long start, long len) The specification of this method matches that of FileContext.getFileBlockLocations(Path, long, long) except that Path f must be for this file system.
FileChecksum	getFileChecksum(Path f) The specification of this method matches that of FileContext.getFileChecksum(Path) except that Path f must be for this file system.
FileStatus	getFileLinkStatus(Path f) The specification of this method matches that of FileContext.getFileLinkStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path leading up to the final path component.
FileStatus	getFileStatus(Path f) The specification of this method matches that of FileContext.getFileStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path.
FsStatus	getFsStatus() The specification of this method matches that of FileContext.getFsStatus(Path).
Path	getHomeDirectory() Return the current user's home directory in this file system.
Path	getLinkTarget(Path f) The specification of this method matches that of FileContext.getLinkTarget(Path);
org.apache.hadoop.fs.viewfs.ViewFs.MountPoint[]	getMountPoints()
FsServerDefaults	getServerDefaults() Return a set of server default configuration values.
int	getUriDefaultPort() The default port of this file system.
FileStatus[]	listStatus(Path f) The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.
org.apache.hadoop.fs.RemoteIterator<FileStatus>	listStatusIterator(Path f) The specification of this method matches that of FileContext.listStatus(Path) except that Path f must be for this file system.
void	mkdir(Path dir, FsPermission permission, boolean createParent) The specification of this method matches that of FileContext.mkdir(Path, FsPermission, boolean) except that the Path f must be fully qualified and the permission is absolute (i.e.
FSDataInputStream	open(Path f, int bufferSize) The specification of this method matches that of FileContext.open(Path, int) except that Path f must be for this file system.
void	renameInternal(Path src, Path dst) The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system and NO OVERWRITE is performed.
void	renameInternal(Path src, Path dst, boolean overwrite) The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.
Path	resolvePath(Path f) Return the fully-qualified path of path f resolving the path through any internal symlinks or mount point
void	setOwner(Path f, String username, String groupname) The specification of this method matches that of FileContext.setOwner(Path, String, String) except that Path f must be for this file system.
void	setPermission(Path f, FsPermission permission) The specification of this method matches that of FileContext.setPermission(Path, FsPermission) except that Path f must be for this file system.
boolean	setReplication(Path f, short replication) The specification of this method matches that of FileContext.setReplication(Path, short) except that Path f must be for this file system.
void	setTimes(Path f, long mtime, long atime) The specification of this method matches that of FileContext.setTimes(Path, long, long) except that Path f must be for this file system.
void	setVerifyChecksum(boolean verifyChecksum) The specification of this method matches that of FileContext.setVerifyChecksum(boolean, Path) except that Path f must be for this file system.
boolean	supportsSymlinks() Returns true if the file system supports symlinks, false otherwise.

Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

checkPath, checkScheme, clearStatistics, create, createFileSystem, equals, get, getAllStatistics, getCanonicalServiceName, getFsStatus, getInitialWorkingDirectory, getStatistics, getStatistics, getUri, getUriPath, hashCode, listCorruptFileBlocks, listLocatedStatus, makeQualified, open, printStatistics, rename

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ViewFs

public ViewFs(Configuration conf)
       throws IOException,
              URISyntaxException

Throws:

IOException

URISyntaxException

Method Detail

getServerDefaults

public FsServerDefaults getServerDefaults()
                                   throws IOException

Description copied from class: AbstractFileSystem

Return a set of server default configuration values.

Specified by:

getServerDefaults in class AbstractFileSystem

Returns:

server default configuration values

Throws:

IOException - an I/O error occurred

getUriDefaultPort

public int getUriDefaultPort()

Description copied from class: AbstractFileSystem

The default port of this file system.

Specified by:

getUriDefaultPort in class AbstractFileSystem

Returns:

default port of this file system's Uri scheme A uri with a port of -1 => default port;

getHomeDirectory

public Path getHomeDirectory()

Description copied from class: AbstractFileSystem

Return the current user's home directory in this file system. The default implementation returns "/user/$USER/".

Overrides:

getHomeDirectory in class AbstractFileSystem

Returns:

current user's home directory.

resolvePath

public Path resolvePath(Path f)
                 throws FileNotFoundException,
                        org.apache.hadoop.security.AccessControlException,
                        org.apache.hadoop.fs.UnresolvedLinkException,
                        IOException

Description copied from class: AbstractFileSystem

Return the fully-qualified path of path f resolving the path through any internal symlinks or mount point

Overrides:

resolvePath in class AbstractFileSystem

Parameters:

f - path to be resolved

Returns:

fully qualified path

Throws:

FileNotFoundException

org.apache.hadoop.security.AccessControlException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

createInternal

public FSDataOutputStream createInternal(Path f,
                                EnumSet<CreateFlag> flag,
                                FsPermission absolutePermission,
                                int bufferSize,
                                short replication,
                                long blockSize,
                                Progressable progress,
                                org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
                                boolean createParent)
                                  throws org.apache.hadoop.security.AccessControlException,
                                         FileAlreadyExistsException,
                                         FileNotFoundException,
                                         ParentNotDirectoryException,
                                         UnsupportedFileSystemException,
                                         org.apache.hadoop.fs.UnresolvedLinkException,
                                         IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.

Specified by:

createInternal in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileAlreadyExistsException

FileNotFoundException

ParentNotDirectoryException

UnsupportedFileSystemException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

delete

public boolean delete(Path f,
             boolean recursive)
               throws org.apache.hadoop.security.AccessControlException,
                      FileNotFoundException,
                      org.apache.hadoop.fs.UnresolvedLinkException,
                      IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.delete(Path, boolean) except that Path f must be for this file system.

Specified by:

delete in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(Path f,
                                    long start,
                                    long len)
                                      throws org.apache.hadoop.security.AccessControlException,
                                             FileNotFoundException,
                                             org.apache.hadoop.fs.UnresolvedLinkException,
                                             IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.getFileBlockLocations(Path, long, long) except that Path f must be for this file system.

Specified by:

getFileBlockLocations in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

getFileChecksum

public FileChecksum getFileChecksum(Path f)
                             throws org.apache.hadoop.security.AccessControlException,
                                    FileNotFoundException,
                                    org.apache.hadoop.fs.UnresolvedLinkException,
                                    IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.getFileChecksum(Path) except that Path f must be for this file system.

Specified by:

getFileChecksum in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws org.apache.hadoop.security.AccessControlException,
                                FileNotFoundException,
                                org.apache.hadoop.fs.UnresolvedLinkException,
                                IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.getFileStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path.

Specified by:

getFileStatus in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

getFileLinkStatus

public FileStatus getFileLinkStatus(Path f)
                             throws org.apache.hadoop.security.AccessControlException,
                                    FileNotFoundException,
                                    UnsupportedFileSystemException,
                                    IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.getFileLinkStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path leading up to the final path component. If the file system does not support symlinks then the behavior is equivalent to AbstractFileSystem.getFileStatus(Path).

Overrides:

getFileLinkStatus in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

UnsupportedFileSystemException

IOException

getFsStatus

public FsStatus getFsStatus()
                     throws org.apache.hadoop.security.AccessControlException,
                            FileNotFoundException,
                            IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.getFsStatus(Path).

Specified by:

getFsStatus in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

IOException

listStatusIterator

public org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatusIterator(Path f)
                                                                   throws org.apache.hadoop.security.AccessControlException,
                                                                          FileNotFoundException,
                                                                          org.apache.hadoop.fs.UnresolvedLinkException,
                                                                          IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.listStatus(Path) except that Path f must be for this file system.

Overrides:

listStatusIterator in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws org.apache.hadoop.security.AccessControlException,
                               FileNotFoundException,
                               org.apache.hadoop.fs.UnresolvedLinkException,
                               IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.

Specified by:

listStatus in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

mkdir

public void mkdir(Path dir,
         FsPermission permission,
         boolean createParent)
           throws org.apache.hadoop.security.AccessControlException,
                  FileAlreadyExistsException,
                  FileNotFoundException,
                  org.apache.hadoop.fs.UnresolvedLinkException,
                  IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.mkdir(Path, FsPermission, boolean) except that the Path f must be fully qualified and the permission is absolute (i.e. umask has been applied).

Specified by:

mkdir in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileAlreadyExistsException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

open

public FSDataInputStream open(Path f,
                     int bufferSize)
                       throws org.apache.hadoop.security.AccessControlException,
                              FileNotFoundException,
                              org.apache.hadoop.fs.UnresolvedLinkException,
                              IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.open(Path, int) except that Path f must be for this file system.

Specified by:

open in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

renameInternal

public void renameInternal(Path src,
                  Path dst,
                  boolean overwrite)
                    throws IOException,
                           org.apache.hadoop.fs.UnresolvedLinkException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.

Overrides:

renameInternal in class AbstractFileSystem

Throws:

IOException

org.apache.hadoop.fs.UnresolvedLinkException

renameInternal

public void renameInternal(Path src,
                  Path dst)
                    throws org.apache.hadoop.security.AccessControlException,
                           FileAlreadyExistsException,
                           FileNotFoundException,
                           ParentNotDirectoryException,
                           org.apache.hadoop.fs.UnresolvedLinkException,
                           IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system and NO OVERWRITE is performed. File systems that do not have a built in overwrite need implement only this method and can take advantage of the default impl of the other AbstractFileSystem.renameInternal(Path, Path, boolean)

Specified by:

renameInternal in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileAlreadyExistsException

FileNotFoundException

ParentNotDirectoryException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

supportsSymlinks

public boolean supportsSymlinks()

Description copied from class: AbstractFileSystem

Returns true if the file system supports symlinks, false otherwise.

Overrides:

supportsSymlinks in class AbstractFileSystem

createSymlink

public void createSymlink(Path target,
                 Path link,
                 boolean createParent)
                   throws IOException,
                          org.apache.hadoop.fs.UnresolvedLinkException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.createSymlink(Path, Path, boolean);

Overrides:

createSymlink in class AbstractFileSystem

Throws:

IOException

org.apache.hadoop.fs.UnresolvedLinkException

getLinkTarget

public Path getLinkTarget(Path f)
                   throws IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.getLinkTarget(Path);

Overrides:

getLinkTarget in class AbstractFileSystem

Throws:

IOException

setOwner

public void setOwner(Path f,
            String username,
            String groupname)
              throws org.apache.hadoop.security.AccessControlException,
                     FileNotFoundException,
                     org.apache.hadoop.fs.UnresolvedLinkException,
                     IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.setOwner(Path, String, String) except that Path f must be for this file system.

Specified by:

setOwner in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

setPermission

public void setPermission(Path f,
                 FsPermission permission)
                   throws org.apache.hadoop.security.AccessControlException,
                          FileNotFoundException,
                          org.apache.hadoop.fs.UnresolvedLinkException,
                          IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.setPermission(Path, FsPermission) except that Path f must be for this file system.

Specified by:

setPermission in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

setReplication

public boolean setReplication(Path f,
                     short replication)
                       throws org.apache.hadoop.security.AccessControlException,
                              FileNotFoundException,
                              org.apache.hadoop.fs.UnresolvedLinkException,
                              IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.setReplication(Path, short) except that Path f must be for this file system.

Specified by:

setReplication in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

setTimes

public void setTimes(Path f,
            long mtime,
            long atime)
              throws org.apache.hadoop.security.AccessControlException,
                     FileNotFoundException,
                     org.apache.hadoop.fs.UnresolvedLinkException,
                     IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.setTimes(Path, long, long) except that Path f must be for this file system.

Specified by:

setTimes in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

FileNotFoundException

org.apache.hadoop.fs.UnresolvedLinkException

IOException

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)
                       throws org.apache.hadoop.security.AccessControlException,
                              IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.setVerifyChecksum(boolean, Path) except that Path f must be for this file system.

Specified by:

setVerifyChecksum in class AbstractFileSystem

Throws:

org.apache.hadoop.security.AccessControlException

IOException

getMountPoints

public org.apache.hadoop.fs.viewfs.ViewFs.MountPoint[] getMountPoints()

getDelegationTokens

public List<org.apache.hadoop.security.token.Token<?>> getDelegationTokens(String renewer)
                                                                    throws IOException

Description copied from class: AbstractFileSystem

Get one or more delegation tokens associated with the filesystem. Normally a file system returns a single delegation token. A file system that manages multiple file systems underneath, could return set of delegation tokens for all the file systems it manages

Parameters:

renewer - the account name that is allowed to renew the token.

Returns:

List of delegation tokens. If delegation tokens not supported then return a list of size zero.

Throws:

IOException