Package org.apache.hadoop.fs.viewfs

Class ViewFs

java.lang.Object
org.apache.hadoop.fs.AbstractFileSystem
org.apache.hadoop.fs.viewfs.ViewFs
All Implemented Interfaces:
org.apache.hadoop.fs.PathCapabilities
@Public @Evolving public class ViewFs extends AbstractFileSystem
ViewFs (extends the AbstractFileSystem interface) implements a client-side mount table. The viewFs file system is implemented completely in memory on the client side. The client-side mount table allows a client to provide a customized view of a file system namespace that is composed from one or more individual file systems (a localFs or Hdfs, S3fs, etc). For example one could have a mount table that provides links such as
  • /user -> hdfs://nnContainingUserDir/user
  • /project/foo -> hdfs://nnProject1/projects/foo
  • /project/bar -> hdfs://nnProject2/projects/bar
  • /tmp -> hdfs://nnTmp/privateTmpForUserXXX
ViewFs is specified with the following URI: viewfs:///

To use viewfs one would typically set the default file system in the config (i.e. fs.defaultFS < = viewfs:///) along with the mount table config variables as described below.

** Config variables to specify the mount table entries **

The file system is initialized from the standard Hadoop config through config variables. See FsConstants for URI and Scheme constants; See Constants for config var constants; see ConfigUtil for convenient lib.

All the mount table config entries for view fs are prefixed by fs.viewfs.mounttable. For example the above example can be specified with the following config variables:

  • fs.viewfs.mounttable.default.link./user= hdfs://nnContainingUserDir/user
  • fs.viewfs.mounttable.default.link./project/foo= hdfs://nnProject1/projects/foo
  • fs.viewfs.mounttable.default.link./project/bar= hdfs://nnProject2/projects/bar
  • fs.viewfs.mounttable.default.link./tmp= hdfs://nnTmp/privateTmpForUserXXX
The default mount table (when no authority is specified) is from config variables prefixed by fs.viewFs.mounttable.default The authority component of a URI can be used to specify a different mount table. For example,
  • viewfs://sanjayMountable/
is initialized from fs.viewFs.mounttable.sanjayMountable.* config variables.

**** Merge Mounts **** (NOTE: merge mounts are not implemented yet.)

One can also use "MergeMounts" to merge several directories (this is sometimes called union-mounts or junction-mounts in the literature. For example of the home directories are stored on say two file systems (because they do not fit on one) then one could specify a mount entry such as following merges two dirs:

  • /user -> hdfs://nnUser1/user,hdfs://nnUser2/user
Such a mergeLink can be specified with the following config var where "," is used as the separator for each of links to be merged:
  • fs.viewfs.mounttable.default.linkMerge./user= hdfs://nnUser1/user,hdfs://nnUser1/user
A special case of the merge mount is where mount table's root is merged with the root (slash) of another file system:
  • fs.viewfs.mounttable.default.linkMergeSlash=hdfs://nn99/
In this cases the root of the mount table is merged with the root of hdfs://nn99/