org.apache.hadoop.fs.viewfs
Class ViewFs

java.lang.Object
  extended by org.apache.hadoop.fs.AbstractFileSystem
      extended by org.apache.hadoop.fs.viewfs.ViewFs

@InterfaceAudience.Public
@InterfaceStability.Evolving
public class ViewFs
extends AbstractFileSystem

ViewFs (extends the AbstractFileSystem interface) implements a client-side mount table. The viewFs file system is implemented completely in memory on the client side. The client-side mount table allows a client to provide a customized view of a file system namespace that is composed from one or more individual file systems (a localFs or Hdfs, S3fs, etc). For example one could have a mount table that provides links such as

ViewFs is specified with the following URI: viewfs:///

To use viewfs one would typically set the default file system in the config (i.e. fs.default.name< = viewfs:///) along with the mount table config variables as described below.

** Config variables to specify the mount table entries **

The file system is initialized from the standard Hadoop config through config variables. See FsConstants for URI and Scheme constants; See Constants for config var constants; see ConfigUtil for convenient lib.

All the mount table config entries for view fs are prefixed by fs.viewfs.mounttable. For example the above example can be specified with the following config variables:

The default mount table (when no authority is specified) is from config variables prefixed by fs.viewFs.mounttable.default The authority component of a URI can be used to specify a different mount table. For example, is initialized from fs.viewFs.mounttable.sanjayMountable.* config variables.

**** Merge Mounts **** (NOTE: merge mounts are not implemented yet.)

One can also use "MergeMounts" to merge several directories (this is sometimes called union-mounts or junction-mounts in the literature. For example of the home directories are stored on say two file systems (because they do not fit on one) then one could specify a mount entry such as following merges two dirs:

Such a mergeLink can be specified with the following config var where "," is used as the separator for each of links to be merged: A special case of the merge mount is where mount table's root is merged with the root (slash) of another file system: In this cases the root of the mount table is merged with the root of hdfs://nn99/


Field Summary
 
Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem
statistics
 
Constructor Summary
ViewFs(Configuration conf)
           
 
Method Summary
 FSDataOutputStream createInternal(Path f, EnumSet<CreateFlag> flag, FsPermission absolutePermission, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent)
          The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.
 void createSymlink(Path target, Path link, boolean createParent)
          The specification of this method matches that of FileContext.createSymlink(Path, Path, boolean);
 boolean delete(Path f, boolean recursive)
          The specification of this method matches that of FileContext.delete(Path, boolean) except that Path f must be for this file system.
 List<org.apache.hadoop.security.token.Token<?>> getDelegationTokens(String renewer)
          Get one or more delegation tokens associated with the filesystem.
 BlockLocation[] getFileBlockLocations(Path f, long start, long len)
          The specification of this method matches that of FileContext.getFileBlockLocations(Path, long, long) except that Path f must be for this file system.
 FileChecksum getFileChecksum(Path f)
          The specification of this method matches that of FileContext.getFileChecksum(Path) except that Path f must be for this file system.
 FileStatus getFileLinkStatus(Path f)
          The specification of this method matches that of FileContext.getFileLinkStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path leading up to the final path component.
 FileStatus getFileStatus(Path f)
          The specification of this method matches that of FileContext.getFileStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path.
 FsStatus getFsStatus()
          The specification of this method matches that of FileContext.getFsStatus(Path).
 Path getHomeDirectory()
          Return the current user's home directory in this file system.
 Path getLinkTarget(Path f)
          Partially resolves the path.
 org.apache.hadoop.fs.viewfs.ViewFs.MountPoint[] getMountPoints()
           
 FsServerDefaults getServerDefaults()
          Return a set of server default configuration values.
 int getUriDefaultPort()
          The default port of this file system.
 boolean isValidName(String src)
          Returns true if the specified string is considered valid in the path part of a URI by this file system.
 FileStatus[] listStatus(Path f)
          The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.
 org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatusIterator(Path f)
          The specification of this method matches that of FileContext.listStatus(Path) except that Path f must be for this file system.
 void mkdir(Path dir, FsPermission permission, boolean createParent)
          The specification of this method matches that of FileContext.mkdir(Path, FsPermission, boolean) except that the Path f must be fully qualified and the permission is absolute (i.e.
 FSDataInputStream open(Path f, int bufferSize)
          The specification of this method matches that of FileContext.open(Path, int) except that Path f must be for this file system.
 void renameInternal(Path src, Path dst)
          The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system and NO OVERWRITE is performed.
 void renameInternal(Path src, Path dst, boolean overwrite)
          The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.
 Path resolvePath(Path f)
          Return the fully-qualified path of path f resolving the path through any internal symlinks or mount point
 void setOwner(Path f, String username, String groupname)
          The specification of this method matches that of FileContext.setOwner(Path, String, String) except that Path f must be for this file system.
 void setPermission(Path f, FsPermission permission)
          The specification of this method matches that of FileContext.setPermission(Path, FsPermission) except that Path f must be for this file system.
 boolean setReplication(Path f, short replication)
          The specification of this method matches that of FileContext.setReplication(Path, short) except that Path f must be for this file system.
 void setTimes(Path f, long mtime, long atime)
          The specification of this method matches that of FileContext.setTimes(Path, long, long) except that Path f must be for this file system.
 void setVerifyChecksum(boolean verifyChecksum)
          The specification of this method matches that of FileContext.setVerifyChecksum(boolean, Path) except that Path f must be for this file system.
 boolean supportsSymlinks()
          Returns true if the file system supports symlinks, false otherwise.
 
Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem
checkPath, checkScheme, clearStatistics, create, createFileSystem, equals, get, getAllStatistics, getCanonicalServiceName, getFsStatus, getInitialWorkingDirectory, getStatistics, getStatistics, getUri, getUriPath, hashCode, listCorruptFileBlocks, listLocatedStatus, makeQualified, open, printStatistics, rename
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

ViewFs

public ViewFs(Configuration conf)
       throws IOException,
              URISyntaxException
Throws:
IOException
URISyntaxException
Method Detail

getServerDefaults

public FsServerDefaults getServerDefaults()
                                   throws IOException
Description copied from class: AbstractFileSystem
Return a set of server default configuration values.

Specified by:
getServerDefaults in class AbstractFileSystem
Returns:
server default configuration values
Throws:
IOException - an I/O error occurred

getUriDefaultPort

public int getUriDefaultPort()
Description copied from class: AbstractFileSystem
The default port of this file system.

Specified by:
getUriDefaultPort in class AbstractFileSystem
Returns:
default port of this file system's Uri scheme A uri with a port of -1 => default port;

getHomeDirectory

public Path getHomeDirectory()
Description copied from class: AbstractFileSystem
Return the current user's home directory in this file system. The default implementation returns "/user/$USER/".

Overrides:
getHomeDirectory in class AbstractFileSystem
Returns:
current user's home directory.

resolvePath

public Path resolvePath(Path f)
                 throws FileNotFoundException,
                        org.apache.hadoop.security.AccessControlException,
                        org.apache.hadoop.fs.UnresolvedLinkException,
                        IOException
Description copied from class: AbstractFileSystem
Return the fully-qualified path of path f resolving the path through any internal symlinks or mount point

Overrides:
resolvePath in class AbstractFileSystem
Parameters:
f - path to be resolved
Returns:
fully qualified path
Throws:
FileNotFoundException
org.apache.hadoop.security.AccessControlException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

createInternal

public FSDataOutputStream createInternal(Path f,
                                         EnumSet<CreateFlag> flag,
                                         FsPermission absolutePermission,
                                         int bufferSize,
                                         short replication,
                                         long blockSize,
                                         Progressable progress,
                                         org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
                                         boolean createParent)
                                  throws org.apache.hadoop.security.AccessControlException,
                                         FileAlreadyExistsException,
                                         FileNotFoundException,
                                         ParentNotDirectoryException,
                                         UnsupportedFileSystemException,
                                         org.apache.hadoop.fs.UnresolvedLinkException,
                                         IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.

Specified by:
createInternal in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
ParentNotDirectoryException
UnsupportedFileSystemException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

delete

public boolean delete(Path f,
                      boolean recursive)
               throws org.apache.hadoop.security.AccessControlException,
                      FileNotFoundException,
                      org.apache.hadoop.fs.UnresolvedLinkException,
                      IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.delete(Path, boolean) except that Path f must be for this file system.

Specified by:
delete in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(Path f,
                                             long start,
                                             long len)
                                      throws org.apache.hadoop.security.AccessControlException,
                                             FileNotFoundException,
                                             org.apache.hadoop.fs.UnresolvedLinkException,
                                             IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.getFileBlockLocations(Path, long, long) except that Path f must be for this file system.

Specified by:
getFileBlockLocations in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFileChecksum

public FileChecksum getFileChecksum(Path f)
                             throws org.apache.hadoop.security.AccessControlException,
                                    FileNotFoundException,
                                    org.apache.hadoop.fs.UnresolvedLinkException,
                                    IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.getFileChecksum(Path) except that Path f must be for this file system.

Specified by:
getFileChecksum in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws org.apache.hadoop.security.AccessControlException,
                                FileNotFoundException,
                                org.apache.hadoop.fs.UnresolvedLinkException,
                                IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.getFileStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path.

Specified by:
getFileStatus in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFileLinkStatus

public FileStatus getFileLinkStatus(Path f)
                             throws org.apache.hadoop.security.AccessControlException,
                                    FileNotFoundException,
                                    UnsupportedFileSystemException,
                                    IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.getFileLinkStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path leading up to the final path component. If the file system does not support symlinks then the behavior is equivalent to AbstractFileSystem.getFileStatus(Path).

Overrides:
getFileLinkStatus in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
UnsupportedFileSystemException
IOException

getFsStatus

public FsStatus getFsStatus()
                     throws org.apache.hadoop.security.AccessControlException,
                            FileNotFoundException,
                            IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.getFsStatus(Path).

Specified by:
getFsStatus in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
IOException

listStatusIterator

public org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatusIterator(Path f)
                                                                   throws org.apache.hadoop.security.AccessControlException,
                                                                          FileNotFoundException,
                                                                          org.apache.hadoop.fs.UnresolvedLinkException,
                                                                          IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.listStatus(Path) except that Path f must be for this file system.

Overrides:
listStatusIterator in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws org.apache.hadoop.security.AccessControlException,
                               FileNotFoundException,
                               org.apache.hadoop.fs.UnresolvedLinkException,
                               IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.

Specified by:
listStatus in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

mkdir

public void mkdir(Path dir,
                  FsPermission permission,
                  boolean createParent)
           throws org.apache.hadoop.security.AccessControlException,
                  FileAlreadyExistsException,
                  FileNotFoundException,
                  org.apache.hadoop.fs.UnresolvedLinkException,
                  IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.mkdir(Path, FsPermission, boolean) except that the Path f must be fully qualified and the permission is absolute (i.e. umask has been applied).

Specified by:
mkdir in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

open

public FSDataInputStream open(Path f,
                              int bufferSize)
                       throws org.apache.hadoop.security.AccessControlException,
                              FileNotFoundException,
                              org.apache.hadoop.fs.UnresolvedLinkException,
                              IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.open(Path, int) except that Path f must be for this file system.

Specified by:
open in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

renameInternal

public void renameInternal(Path src,
                           Path dst,
                           boolean overwrite)
                    throws IOException,
                           org.apache.hadoop.fs.UnresolvedLinkException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.

Overrides:
renameInternal in class AbstractFileSystem
Throws:
IOException
org.apache.hadoop.fs.UnresolvedLinkException

renameInternal

public void renameInternal(Path src,
                           Path dst)
                    throws org.apache.hadoop.security.AccessControlException,
                           FileAlreadyExistsException,
                           FileNotFoundException,
                           ParentNotDirectoryException,
                           org.apache.hadoop.fs.UnresolvedLinkException,
                           IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system and NO OVERWRITE is performed. File systems that do not have a built in overwrite need implement only this method and can take advantage of the default impl of the other AbstractFileSystem.renameInternal(Path, Path, boolean)

Specified by:
renameInternal in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
ParentNotDirectoryException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

supportsSymlinks

public boolean supportsSymlinks()
Description copied from class: AbstractFileSystem
Returns true if the file system supports symlinks, false otherwise.

Overrides:
supportsSymlinks in class AbstractFileSystem
Returns:
true if filesystem supports symlinks

createSymlink

public void createSymlink(Path target,
                          Path link,
                          boolean createParent)
                   throws IOException,
                          org.apache.hadoop.fs.UnresolvedLinkException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.createSymlink(Path, Path, boolean);

Overrides:
createSymlink in class AbstractFileSystem
Throws:
IOException
org.apache.hadoop.fs.UnresolvedLinkException

getLinkTarget

public Path getLinkTarget(Path f)
                   throws IOException
Description copied from class: AbstractFileSystem
Partially resolves the path. This is used during symlink resolution in FSLinkResolver, and differs from the similarly named method FileContext.getLinkTarget(Path).

Overrides:
getLinkTarget in class AbstractFileSystem
Throws:
IOException

setOwner

public void setOwner(Path f,
                     String username,
                     String groupname)
              throws org.apache.hadoop.security.AccessControlException,
                     FileNotFoundException,
                     org.apache.hadoop.fs.UnresolvedLinkException,
                     IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.setOwner(Path, String, String) except that Path f must be for this file system.

Specified by:
setOwner in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setPermission

public void setPermission(Path f,
                          FsPermission permission)
                   throws org.apache.hadoop.security.AccessControlException,
                          FileNotFoundException,
                          org.apache.hadoop.fs.UnresolvedLinkException,
                          IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.setPermission(Path, FsPermission) except that Path f must be for this file system.

Specified by:
setPermission in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setReplication

public boolean setReplication(Path f,
                              short replication)
                       throws org.apache.hadoop.security.AccessControlException,
                              FileNotFoundException,
                              org.apache.hadoop.fs.UnresolvedLinkException,
                              IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.setReplication(Path, short) except that Path f must be for this file system.

Specified by:
setReplication in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setTimes

public void setTimes(Path f,
                     long mtime,
                     long atime)
              throws org.apache.hadoop.security.AccessControlException,
                     FileNotFoundException,
                     org.apache.hadoop.fs.UnresolvedLinkException,
                     IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.setTimes(Path, long, long) except that Path f must be for this file system.

Specified by:
setTimes in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)
                       throws org.apache.hadoop.security.AccessControlException,
                              IOException
Description copied from class: AbstractFileSystem
The specification of this method matches that of FileContext.setVerifyChecksum(boolean, Path) except that Path f must be for this file system.

Specified by:
setVerifyChecksum in class AbstractFileSystem
Throws:
org.apache.hadoop.security.AccessControlException
IOException

getMountPoints

public org.apache.hadoop.fs.viewfs.ViewFs.MountPoint[] getMountPoints()

getDelegationTokens

public List<org.apache.hadoop.security.token.Token<?>> getDelegationTokens(String renewer)
                                                                    throws IOException
Description copied from class: AbstractFileSystem
Get one or more delegation tokens associated with the filesystem. Normally a file system returns a single delegation token. A file system that manages multiple file systems underneath, could return set of delegation tokens for all the file systems it manages

Parameters:
renewer - the account name that is allowed to renew the token.
Returns:
List of delegation tokens. If delegation tokens not supported then return a list of size zero.
Throws:
IOException

isValidName

public boolean isValidName(String src)
Description copied from class: AbstractFileSystem
Returns true if the specified string is considered valid in the path part of a URI by this file system. The default implementation enforces the rules of HDFS, but subclasses may override this method to implement specific validation rules for specific file systems.

Overrides:
isValidName in class AbstractFileSystem
Parameters:
src - String source filename to check, path part of the URI
Returns:
boolean true if the specified string is considered valid


Copyright © 2013 Apache Software Foundation. All Rights Reserved.