org.apache.hadoop.fs

Class ChecksumFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FilterFileSystem

org.apache.hadoop.fs.ChecksumFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

Direct Known Subclasses:

LocalFileSystem

@InterfaceAudience.Public
 @InterfaceStability.Stable
public abstract class ChecksumFileSystem
extends FilterFileSystem

Abstract Checksumed FileSystem. It provide a basic implementation of a Checksumed FileSystem, which creates a checksum file for each raw file. It generates & verifies checksums at the client side.

Field Summary

Fields inherited from class org.apache.hadoop.fs.FilterFileSystem

fs, swapScheme

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
ChecksumFileSystem(FileSystem fs)

Method Summary

Modifier and Type	Method and Description
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress) Append to an existing file (optional operation).
void	completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) Called when we're all done writing to the target.
void	copyFromLocalFile(boolean delSrc, Path src, Path dst) The src file is on the local disk.
void	copyToLocalFile(boolean delSrc, Path src, Path dst) The src file is under FS, and the dst is on the local disk.
void	copyToLocalFile(Path src, Path dst, boolean copyCrc) The src file is under FS, and the dst is on the local disk.
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(Path f, boolean recursive) Implement the delete(Path, boolean) in checksum file system.
static double	getApproxChkSumLength(long size)
int	getBytesPerSum() Return the bytes Per Checksum
Path	getChecksumFile(Path file) Return the name of the checksum file associated with a file.
long	getChecksumFileLength(Path file, long fileSize) Return the length of the checksum file given the size of the actual file.
static long	getChecksumLength(long size, int bytesPerSum) Calculated the length of the checksum file in bytes.
FileSystem	getRawFileSystem() get the raw file system
boolean	hasPathCapability(Path path, String capability) Disable those operations which the checksummed FS blocks.
static boolean	isChecksumFile(Path file) Return true iff file is a checksum file name.
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus>	listLocatedStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
FileStatus[]	listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
org.apache.hadoop.fs.RemoteIterator<FileStatus>	listStatusIterator(Path p) Return a remote iterator for listing in a directory
boolean	mkdirs(Path f) Call FileSystem.mkdirs(Path, FsPermission) with default permission.
void	modifyAclEntries(Path src, List<AclEntry> aclSpec) Modifies ACL entries of files and directories.
FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
void	removeAcl(Path src) Removes all but the base ACL entries of files and directories.
void	removeAclEntries(Path src, List<AclEntry> aclSpec) Removes ACL entries from files and directories.
void	removeDefaultAcl(Path src) Removes all default ACL entries from files and directories.
boolean	rename(Path src, Path dst) Rename files/dirs
boolean	reportChecksumFailure(Path f, FSDataInputStream in, long inPos, FSDataInputStream sums, long sumsPos) Report a checksum error to the file system.
void	setAcl(Path src, List<AclEntry> aclSpec) Fully replaces ACL of files and directories, discarding all existing entries.
void	setConf(Configuration conf) Set the configuration to be used by this object.
void	setOwner(Path src, String username, String groupname) Set owner of a path (i.e.
void	setPermission(Path src, FsPermission permission) Set permission of a path.
boolean	setReplication(Path src, short replication) Set replication for an existing file.
void	setVerifyChecksum(boolean verifyChecksum) Set whether to verify checksum.
void	setWriteChecksum(boolean writeChecksum) Set the write checksum flag.
Path	startLocalOutput(Path fsOutputFile, Path tmpLocalFile) Returns a local File that the user can write output to.
boolean	truncate(Path f, long newLength) Truncate the file in the indicated path to the indicated size.

Methods inherited from class org.apache.hadoop.fs.FilterFileSystem

access, appendFile, canonicalizeUri, checkPath, close, concat, copyFromLocalFile, copyFromLocalFile, create, createFile, createNonRecursive, createPathHandle, createSnapshot, createSymlink, deleteSnapshot, getAclStatus, getAllStoragePolicies, getCanonicalUri, getChildFileSystems, getConf, getDefaultBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getServerDefaults, getServerDefaults, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUri, getUsed, getUsed, getWorkingDirectory, getXAttr, getXAttrs, getXAttrs, initialize, listCorruptFileBlocks, listLocatedStatus, listXAttrs, makeQualified, mkdirs, msync, open, primitiveCreate, primitiveMkdir, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setStoragePolicy, setTimes, setWorkingDirectory, setXAttr, setXAttr, supportsSymlinks, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createSnapshot, delete, deleteOnExit, enableSymlinks, exists, fixRelativePart, get, get, get, getAllStatistics, getBlockSize, getCanonicalServiceName, getContentSummary, getDefaultPort, getDefaultUri, getFileBlockLocations, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getLength, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getScheme, getStatistics, getStatistics, getStatus, getStorageStatistics, globStatus, globStatus, isDirectory, isFile, listFiles, listStatus, listStatus, listStatus, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, open, primitiveMkdir, printStatistics, processDeleteOnExit, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageType

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, getAdditionalTokenIssuers, getDelegationToken

Constructor Detail

ChecksumFileSystem

public ChecksumFileSystem(FileSystem fs)

Method Detail

getApproxChkSumLength

public static double getApproxChkSumLength(long size)

setConf

public void setConf(Configuration conf)

Description copied from interface: Configurable

Set the configuration to be used by this object.

Specified by:

setConf in interface Configurable

Overrides:

setConf in class Configured

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)

Set whether to verify checksum.

Overrides:

setVerifyChecksum in class FilterFileSystem

Parameters:

verifyChecksum - Verify checksum flag

setWriteChecksum

public void setWriteChecksum(boolean writeChecksum)

Description copied from class: FileSystem

Set the write checksum flag. This is only applicable if the corresponding filesystem supports checksums. By default doesn't do anything.

Overrides:

setWriteChecksum in class FilterFileSystem

Parameters:

writeChecksum - Write checksum flag

getRawFileSystem

public FileSystem getRawFileSystem()

get the raw file system

Overrides:

getRawFileSystem in class FilterFileSystem

Returns:

FileSystem being filtered

getChecksumFile

public Path getChecksumFile(Path file)

Return the name of the checksum file associated with a file.

isChecksumFile

public static boolean isChecksumFile(Path file)

Return true iff file is a checksum file name.

getChecksumFileLength

public long getChecksumFileLength(Path file,
                                  long fileSize)

Return the length of the checksum file given the size of the actual file.

getBytesPerSum

public int getBytesPerSum()

Return the bytes Per Checksum

open

public FSDataInputStream open(Path f,
                              int bufferSize)
                       throws IOException

Opens an FSDataInputStream at the indicated Path.

Overrides:

open in class FilterFileSystem

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

IOException - IO failure

append

public FSDataOutputStream append(Path f,
                                 int bufferSize,
                                 Progressable progress)
                          throws IOException

Description copied from class: FileSystem

Append to an existing file (optional operation).

Overrides:

append in class FilterFileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

Throws:

IOException - IO failure

truncate

public boolean truncate(Path f,
                        long newLength)
                 throws IOException

Description copied from class: FileSystem

Truncate the file in the indicated path to the indicated size.

• Fails if path is a directory.
• Fails if path does not exist.
• Fails if path is not closed.
• Fails if new size is greater than current size.

Overrides:

truncate in class FilterFileSystem

Parameters:

f - The path to the file to be truncated

newLength - The size the file is to be truncated to

Returns:

true if the file has been truncated to the desired newLength and is immediately available to be reused for write operations such as append, or false if a background process of adjusting the length of the last block has been started, and clients should wait for it to complete before proceeding with further file updates.

Throws:

IOException - IO failure

getChecksumLength

public static long getChecksumLength(long size,
                                     int bytesPerSum)

Calculated the length of the checksum file in bytes.

Parameters:

size - the length of the data file in bytes

bytesPerSum - the number of bytes in a checksum block

Returns:

the number of bytes in the checksum file

create

public FSDataOutputStream create(Path f,
                                 FsPermission permission,
                                 boolean overwrite,
                                 int bufferSize,
                                 short replication,
                                 long blockSize,
                                 Progressable progress)
                          throws IOException

Description copied from class: FileSystem

Create an FSDataOutputStream at the indicated Path with write-progress reporting.

Overrides:

create in class FilterFileSystem

Parameters:

f - the file name to open

permission - file permission

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size

progress - the progress reporter

Throws:

IOException - IO failure

See Also:

FileSystem.setPermission(Path, FsPermission)

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                             FsPermission permission,
                                             boolean overwrite,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             Progressable progress)
                                      throws IOException

Description copied from class: FileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

Overrides:

createNonRecursive in class FileSystem

Parameters:

f - the file name to open

permission - file permission

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size

progress - the progress reporter

Throws:

IOException - IO failure

See Also:

FileSystem.setPermission(Path, FsPermission)

setPermission

public void setPermission(Path src,
                          FsPermission permission)
                   throws IOException

Description copied from class: FileSystem

Set permission of a path.

Overrides:

setPermission in class FilterFileSystem

Parameters:

src - The path

permission - permission

Throws:

IOException - IO failure

setOwner

public void setOwner(Path src,
                     String username,
                     String groupname)
              throws IOException

Description copied from class: FileSystem

Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.

Overrides:

setOwner in class FilterFileSystem

Parameters:

src - The path

username - If it is null, the original username remains unchanged.

groupname - If it is null, the original groupname remains unchanged.

Throws:

IOException - IO failure

setAcl

public void setAcl(Path src,
                   List<AclEntry> aclSpec)
            throws IOException

Description copied from class: FileSystem

Fully replaces ACL of files and directories, discarding all existing entries.

Overrides:

setAcl in class FilterFileSystem

Parameters:

src - Path to modify

aclSpec - List describing modifications, which must include entries for user, group, and others for compatibility with permission bits.

Throws:

IOException - if an ACL could not be modified

modifyAclEntries

public void modifyAclEntries(Path src,
                             List<AclEntry> aclSpec)
                      throws IOException

Description copied from class: FileSystem

Modifies ACL entries of files and directories. This method can add new ACL entries or modify the permissions on existing ACL entries. All existing ACL entries that are not specified in this call are retained without changes. (Modifications are merged into the current ACL.)

Overrides:

modifyAclEntries in class FilterFileSystem

Parameters:

src - Path to modify

aclSpec - List describing modifications

Throws:

IOException - if an ACL could not be modified

removeAcl

public void removeAcl(Path src)
               throws IOException

Description copied from class: FileSystem

Removes all but the base ACL entries of files and directories. The entries for user, group, and others are retained for compatibility with permission bits.

Overrides:

removeAcl in class FilterFileSystem

Parameters:

src - Path to modify

Throws:

IOException - if an ACL could not be removed

removeAclEntries

public void removeAclEntries(Path src,
                             List<AclEntry> aclSpec)
                      throws IOException

Description copied from class: FileSystem

Removes ACL entries from files and directories. Other ACL entries are retained.

Overrides:

removeAclEntries in class FilterFileSystem

Parameters:

src - Path to modify

aclSpec - List describing entries to remove

Throws:

IOException - if an ACL could not be modified

removeDefaultAcl

public void removeDefaultAcl(Path src)
                      throws IOException

Description copied from class: FileSystem

Removes all default ACL entries from files and directories.

Overrides:

removeDefaultAcl in class FilterFileSystem

Parameters:

src - Path to modify

Throws:

IOException - if an ACL could not be modified

setReplication

public boolean setReplication(Path src,
                              short replication)
                       throws IOException

Set replication for an existing file. Implement the abstract setReplication of FileSystem

Overrides:

setReplication in class FilterFileSystem

Parameters:

src - file name

replication - new replication

Returns:

true if successful; false if file does not exist or is a directory

Throws:

IOException

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Rename files/dirs

Overrides:

rename in class FilterFileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on failure

delete

public boolean delete(Path f,
                      boolean recursive)
               throws IOException

Implement the delete(Path, boolean) in checksum file system.

Overrides:

delete in class FilterFileSystem

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

IOException - IO failure

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

List the statuses of the files/directories in the given path if the path is a directory.

Overrides:

listStatus in class FilterFileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given path

Throws:

IOException

FileNotFoundException - when the path does not exist

listStatusIterator

public org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatusIterator(Path p)
                                                                   throws IOException

Description copied from class: FilterFileSystem

Return a remote iterator for listing in a directory

Overrides:

listStatusIterator in class FilterFileSystem

Parameters:

p - target path

Returns:

remote iterator

Throws:

FileNotFoundException - if p does not exist

IOException - if any I/O error occurred

listLocatedStatus

public org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f)
                                                                         throws IOException

List the statuses of the files/directories in the given path if the path is a directory.

Overrides:

listLocatedStatus in class FilterFileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

IOException

FileNotFoundException - If f does not exist

mkdirs

public boolean mkdirs(Path f)
               throws IOException

Description copied from class: FileSystem

Call FileSystem.mkdirs(Path, FsPermission) with default permission.

Overrides:

mkdirs in class FilterFileSystem

Parameters:

f - path

Returns:

true if the directory was created

Throws:

IOException - IO failure

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                              Path src,
                              Path dst)
                       throws IOException

Description copied from class: FilterFileSystem

The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed

Overrides:

copyFromLocalFile in class FilterFileSystem

Parameters:

delSrc - whether to delete the src

src - path

dst - path

Throws:

IOException

copyToLocalFile

public void copyToLocalFile(boolean delSrc,
                            Path src,
                            Path dst)
                     throws IOException

The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name.

Overrides:

copyToLocalFile in class FilterFileSystem

Parameters:

delSrc - whether to delete the src

src - path src file in the remote filesystem

dst - path local destination

Throws:

IOException - IO failure

copyToLocalFile

public void copyToLocalFile(Path src,
                            Path dst,
                            boolean copyCrc)
                     throws IOException

The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. If src and dst are directories, the copyCrc parameter determines whether to copy CRC files.

Throws:

IOException

startLocalOutput

public Path startLocalOutput(Path fsOutputFile,
                             Path tmpLocalFile)
                      throws IOException

Description copied from class: FilterFileSystem

Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

Overrides:

startLocalOutput in class FilterFileSystem

Parameters:

fsOutputFile - path of output file

tmpLocalFile - path of local tmp file

Throws:

IOException - IO failure

completeLocalOutput

public void completeLocalOutput(Path fsOutputFile,
                                Path tmpLocalFile)
                         throws IOException

Description copied from class: FilterFileSystem

Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

Overrides:

completeLocalOutput in class FilterFileSystem

Parameters:

fsOutputFile - path of output file

tmpLocalFile - path to local tmp file

Throws:

IOException - IO failure

reportChecksumFailure

public boolean reportChecksumFailure(Path f,
                                     FSDataInputStream in,
                                     long inPos,
                                     FSDataInputStream sums,
                                     long sumsPos)

Report a checksum error to the file system.

Parameters:

f - the file name containing the error

in - the stream open on the file

inPos - the position of the beginning of the bad data in the file

sums - the stream open on the checksum file

sumsPos - the position of the beginning of the bad data in the checksum file

Returns:

if retry is necessary

hasPathCapability

public boolean hasPathCapability(Path path,
                                 String capability)
                          throws IOException

Disable those operations which the checksummed FS blocks. The base FileSystem implementation generally has no knowledge of the capabilities of actual implementations. Unless it has a way to explicitly determine the capabilities, this method returns false. Probe for a specific capability under the given path. If the function returns true, this instance is explicitly declaring that the capability is available. If the function returns false, it can mean one of:

• The capability is not known.
• The capability is known but it is not supported.
• The capability is known but the filesystem does not know if it is supported under the supplied path.

The core guarantee which a caller can rely on is: if the predicate returns true, then the specific operation/behavior can be expected to be supported. However a specific call may be rejected for permission reasons, the actual file/directory not being present, or some other failure during the attempted execution of the operation.

Implementors: PathCapabilitiesSupport can be used to help implement this method.

Specified by:

hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities

Overrides:

hasPathCapability in class FilterFileSystem

Parameters:

path - path to query the capability of.

capability - non-null, non-empty string to query the path for support.

Returns:

true if the capability is supported under that part of the FS.

Throws:

IOException - this should not be raised, except on problems resolving paths or relaying the call.