|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.hadoop.conf.Configured org.apache.hadoop.fs.FileSystem org.apache.hadoop.fs.FilterFileSystem org.apache.hadoop.fs.HarFileSystem
public class HarFileSystem
This is an implementation of the Hadoop Archive Filesystem. This archive Filesystem has index files of the form _index* and has contents of the form part-*. The index files store the indexes of the real files. The index files are of the form _masterindex and _index. The master index is a level of indirection in to the index file to make the look ups faster. the index file is sorted with hash code of the paths that it contains and the master index contains pointers to the positions in index for ranges of hashcodes.
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem |
---|
FileSystem.Statistics |
Field Summary | |
---|---|
static int |
VERSION
|
Fields inherited from class org.apache.hadoop.fs.FilterFileSystem |
---|
fs |
Fields inherited from class org.apache.hadoop.fs.FileSystem |
---|
FS_DEFAULT_NAME_KEY, LOG, statistics |
Constructor Summary | |
---|---|
HarFileSystem()
public construction of harfilesystem |
|
HarFileSystem(FileSystem fs)
Constructor to create a HarFileSystem with an underlying filesystem. |
Method Summary | |
---|---|
void |
close()
No more filesystem operations are needed. |
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
not implemented. |
void |
copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
not implemented. |
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst)
copies the file in the har filesystem to a local file. |
FSDataOutputStream |
create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
FSDataOutputStream |
create(Path f,
int bufferSize)
|
boolean |
delete(Path f,
boolean recursive)
Not implemented. |
String |
getCanonicalServiceName()
Get a canonical service name for this file system. |
BlockLocation[] |
getFileBlockLocations(FileStatus file,
long start,
long len)
Get block locations from the underlying fs and fix their offsets and lengths. |
FileChecksum |
getFileChecksum(Path f)
Get the checksum of a file. |
FileStatus |
getFileStatus(Path f)
return the filestatus of files in har archive. |
static int |
getHarHash(Path p)
the hash of the path p inside iniside the filesystem |
int |
getHarVersion()
|
Path |
getHomeDirectory()
return the top level archive path. |
URI |
getUri()
Returns the uri of this filesystem. |
Path |
getWorkingDirectory()
return the top level archive. |
void |
initialize(URI name,
Configuration conf)
Initialize a Har filesystem per har archive. |
FileStatus[] |
listStatus(Path f)
liststatus returns the children of a directory after looking up the index files. |
Path |
makeQualified(Path path)
Make sure that a path specifies a FileSystem. |
boolean |
mkdirs(Path f,
FsPermission permission)
not implemented. |
FSDataInputStream |
open(Path f,
int bufferSize)
Returns a har input stream which fakes end of file. |
void |
setOwner(Path p,
String username,
String groupname)
not implemented. |
void |
setPermission(Path p,
FsPermission permisssion)
Not implemented. |
boolean |
setReplication(Path src,
short replication)
Not implemented. |
void |
setWorkingDirectory(Path newDir)
Set the current working directory for the given file system. |
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
not implemented. |
Methods inherited from class org.apache.hadoop.fs.FilterFileSystem |
---|
append, checkPath, concat, delete, getConf, getDefaultBlockSize, getDefaultReplication, getName, rename, setVerifyChecksum |
Methods inherited from class org.apache.hadoop.conf.Configured |
---|
setConf |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final int VERSION
Constructor Detail |
---|
public HarFileSystem()
public HarFileSystem(FileSystem fs)
fs
- Method Detail |
---|
public void initialize(URI name, Configuration conf) throws IOException
initialize
in class FilterFileSystem
name
- a uri whose authority section names the host, port, etc.
for this FileSystemconf
- the configuration
IOException
public int getHarVersion() throws IOException
IOException
public Path getWorkingDirectory()
getWorkingDirectory
in class FilterFileSystem
public URI getUri()
getUri
in class FilterFileSystem
public String getCanonicalServiceName()
FileSystem
getCanonicalServiceName
in class FilterFileSystem
SecurityUtil.buildDTServiceName(URI, int)
public Path makeQualified(Path path)
FilterFileSystem
makeQualified
in class FilterFileSystem
public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws IOException
getFileBlockLocations
in class FilterFileSystem
file
- the input filestatus to get block locationsstart
- the start of the desired range in the contained filelen
- the length of the desired range
IOException
public static int getHarHash(Path p)
p
- the path in the harfilesystem
public FileStatus getFileStatus(Path f) throws IOException
getFileStatus
in class FilterFileSystem
f
- the path in har filesystem
IOException
FileNotFoundException
- when the path does not exist;
IOException see specific implementationpublic FileChecksum getFileChecksum(Path f)
FilterFileSystem
getFileChecksum
in class FilterFileSystem
f
- The file path
public FSDataInputStream open(Path f, int bufferSize) throws IOException
open
in class FilterFileSystem
f
- the file name to openbufferSize
- the size of the buffer to be used.
IOException
public FSDataOutputStream create(Path f, int bufferSize) throws IOException
IOException
public FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystem
create
in class FilterFileSystem
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.
IOException
FileSystem.setPermission(Path, FsPermission)
public void close() throws IOException
FileSystem
close
in interface Closeable
close
in class FilterFileSystem
IOException
public boolean setReplication(Path src, short replication) throws IOException
setReplication
in class FilterFileSystem
src
- file namereplication
- new replication
IOException
public boolean delete(Path f, boolean recursive) throws IOException
delete
in class FilterFileSystem
f
- the path to delete.recursive
- if path is a directory and set to
true, the directory is deleted else throws an exception. In
case of a file the recursive can be set to either true or false.
IOException
public FileStatus[] listStatus(Path f) throws IOException
listStatus
in class FilterFileSystem
f
- given path
IOException
public Path getHomeDirectory()
getHomeDirectory
in class FilterFileSystem
public void setWorkingDirectory(Path newDir)
FilterFileSystem
setWorkingDirectory
in class FilterFileSystem
public boolean mkdirs(Path f, FsPermission permission) throws IOException
mkdirs
in class FilterFileSystem
IOException
public void copyFromLocalFile(boolean delSrc, Path src, Path dst) throws IOException
copyFromLocalFile
in class FilterFileSystem
IOException
public void copyToLocalFile(boolean delSrc, Path src, Path dst) throws IOException
copyToLocalFile
in class FilterFileSystem
IOException
public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
startLocalOutput
in class FilterFileSystem
IOException
public void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
completeLocalOutput
in class FilterFileSystem
IOException
public void setOwner(Path p, String username, String groupname) throws IOException
setOwner
in class FilterFileSystem
p
- The pathusername
- If it is null, the original username remains unchanged.groupname
- If it is null, the original groupname remains unchanged.
IOException
public void setPermission(Path p, FsPermission permisssion) throws IOException
setPermission
in class FilterFileSystem
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |