org.apache.hadoop.fs.azure

Class NativeAzureFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.azure.NativeAzureFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable

@InterfaceAudience.Public
@InterfaceStability.Stable
public class NativeAzureFileSystem
extends FileSystem

A FileSystem for reading and writing files stored on Windows Azure. This implementation is blob-based and stores files on Azure in their native form so they can be read by other Azure tools.

Field Summary

Fields

Modifier and Type	Field and Description
static String	APPEND_SUPPORT_ENABLE_PROPERTY_NAME
static org.slf4j.Logger	LOG
static String	SKIP_AZURE_METRICS_PROPERTY_NAME

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX

Constructor Summary

Constructors

Constructor and Description
NativeAzureFileSystem()
NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store)

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.azure.SelfRenewingLease	acquireLease(Path path) Get a self-renewing lease on the specified file.
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress) This optional operation is not yet supported.
protected void	checkPath(Path path) Check that a Path belongs to this FileSystem.
void	close() No more filesystem operations are needed.
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(Path path) Deprecated.
boolean	delete(Path f, boolean recursive) Delete a file.
boolean	delete(Path f, boolean recursive, boolean skipParentFolderLastModifidedTimeUpdate) Delete the specified file or folder.
void	deleteFilesWithDanglingTempData(Path root) Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there.
protected void	finalize()
BlockLocation[]	getFileBlockLocations(FileStatus file, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
FileStatus	getFileStatus(Path f) Return a file status object that represents the path.
AzureFileSystemInstrumentation	getInstrumentation() Gets the metrics source for this file system.
String	getScheme() Return the protocol scheme for the FileSystem.
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore	getStore() For unit test purposes, retrieves the AzureNativeFileSystemStore store backing this file system.
URI	getUri() Returns a URI whose scheme and authority identify this FileSystem.
Path	getWorkingDirectory() Get the current working directory for the given file system
void	initialize(URI uri, Configuration conf) Called after a new FileSystem instance is constructed.
FileStatus[]	listStatus(Path f) Retrieve the status of a given path if it is a file, or of all the contained files if it is a directory.
Path	makeAbsolute(Path path) Get the absolute version of the path (fully qualified).
boolean	mkdirs(Path f, FsPermission permission) Make the given file and all non-existent parents into directories.
boolean	mkdirs(Path f, FsPermission permission, boolean noUmask)
static String	newMetricsSourceName() Creates a new metrics source name that's unique within this process.
FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
String	pathToKey(Path path) Convert the path to a key.
void	recoverFilesWithDanglingTempData(Path root, Path destination) Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there.
boolean	rename(Path src, Path dst) Renames Path src to Path dst.
void	setOwner(Path p, String username, String groupname) Set owner of a path (i.e.
void	setPermission(Path p, FsPermission permission) Set permission of a path.
void	setWorkingDirectory(Path newDir) Set the working directory to the given directory.

Methods inherited from class org.apache.hadoop.fs.FileSystem

addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createSnapshot, createSnapshot, createSymlink, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

SKIP_AZURE_METRICS_PROPERTY_NAME

public static final String SKIP_AZURE_METRICS_PROPERTY_NAME

See Also:

Constant Field Values

APPEND_SUPPORT_ENABLE_PROPERTY_NAME

public static final String APPEND_SUPPORT_ENABLE_PROPERTY_NAME

See Also:

Constant Field Values

Constructor Detail

NativeAzureFileSystem

public NativeAzureFileSystem()

NativeAzureFileSystem

public NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store)

Method Detail

getScheme

public String getScheme()

Description copied from class: FileSystem

Return the protocol scheme for the FileSystem.

This implementation throws an UnsupportedOperationException.

Overrides:

getScheme in class FileSystem

Returns:

the protocol scheme for the FileSystem.

newMetricsSourceName

public static String newMetricsSourceName()

Creates a new metrics source name that's unique within this process.

checkPath

protected void checkPath(Path path)

Description copied from class: FileSystem

Check that a Path belongs to this FileSystem.

Overrides:

checkPath in class FileSystem

Parameters:

path - to check

initialize

public void initialize(URI uri,
              Configuration conf)
                throws IOException,
                       IllegalArgumentException

Description copied from class: FileSystem

Called after a new FileSystem instance is constructed.

Overrides:

initialize in class FileSystem

Parameters:

uri - a uri whose authority section names the host, port, etc. for this FileSystem

conf - the configuration

Throws:

IOException

IllegalArgumentException

pathToKey

public String pathToKey(Path path)

Convert the path to a key. By convention, any leading or trailing slash is removed, except for the special case of a single slash.

makeAbsolute

public Path makeAbsolute(Path path)

Get the absolute version of the path (fully qualified). This is public for testing purposes.

Parameters:

path -

Returns:

fully qualified path

getStore

public org.apache.hadoop.fs.azure.AzureNativeFileSystemStore getStore()

For unit test purposes, retrieves the AzureNativeFileSystemStore store backing this file system.

Returns:

The store object.

getInstrumentation

public AzureFileSystemInstrumentation getInstrumentation()

Gets the metrics source for this file system. This is mainly here for unit testing purposes.

Returns:

the metrics source.

append

public FSDataOutputStream append(Path f,
                        int bufferSize,
                        Progressable progress)
                          throws IOException

This optional operation is not yet supported.

Specified by:

append in class FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

Throws:

IOException

create

public FSDataOutputStream create(Path f,
                        FsPermission permission,
                        boolean overwrite,
                        int bufferSize,
                        short replication,
                        long blockSize,
                        Progressable progress)
                          throws IOException

Description copied from class: FileSystem

Create an FSDataOutputStream at the indicated Path with write-progress reporting.

Specified by:

create in class FileSystem

Parameters:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

Throws:

IOException

See Also:

FileSystem.setPermission(Path, FsPermission)

acquireLease

public org.apache.hadoop.fs.azure.SelfRenewingLease acquireLease(Path path)
                                                          throws AzureException

Get a self-renewing lease on the specified file.

Throws:

AzureException

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                    FsPermission permission,
                                    boolean overwrite,
                                    int bufferSize,
                                    short replication,
                                    long blockSize,
                                    Progressable progress)
                                      throws IOException

Description copied from class: FileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

Overrides:

createNonRecursive in class FileSystem

Parameters:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

Throws:

IOException

See Also:

FileSystem.setPermission(Path, FsPermission)

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                    FsPermission permission,
                                    EnumSet<CreateFlag> flags,
                                    int bufferSize,
                                    short replication,
                                    long blockSize,
                                    Progressable progress)
                                      throws IOException

Description copied from class: FileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

Overrides:

createNonRecursive in class FileSystem

Parameters:

f - the file name to open

flags - CreateFlags to use for this stream.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

Throws:

IOException

See Also:

FileSystem.setPermission(Path, FsPermission)

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                    boolean overwrite,
                                    int bufferSize,
                                    short replication,
                                    long blockSize,
                                    Progressable progress)
                                      throws IOException

Description copied from class: FileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

Overrides:

createNonRecursive in class FileSystem

Parameters:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

Throws:

IOException

See Also:

FileSystem.setPermission(Path, FsPermission)

delete

@Deprecated
public boolean delete(Path path)
               throws IOException

Deprecated.

Description copied from class: FileSystem

Delete a file

Overrides:

delete in class FileSystem

Throws:

IOException

delete

public boolean delete(Path f,
             boolean recursive)
               throws IOException

Description copied from class: FileSystem

Delete a file.

Specified by:

delete in class FileSystem

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

IOException

delete

public boolean delete(Path f,
             boolean recursive,
             boolean skipParentFolderLastModifidedTimeUpdate)
               throws IOException

Delete the specified file or folder. The parameter skipParentFolderLastModifidedTimeUpdate is used in the case of atomic folder rename redo. In that case, there is a lease on the parent folder, so (without reworking the code) modifying the parent folder update time will fail because of a conflict with the lease. Since we are going to delete the folder soon anyway so accurate modified time is not necessary, it's easier to just skip the modified time update.

Parameters:

f -

recursive -

skipParentFolderLastModifidedTimeUpdate - If true, don't update the folder last modified time.

Returns:

true if and only if the file is deleted

Throws:

IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws FileNotFoundException,
                                IOException

Description copied from class: FileSystem

Return a file status object that represents the path.

Specified by:

getFileStatus in class FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

getUri

public URI getUri()

Description copied from class: FileSystem

Returns a URI whose scheme and authority identify this FileSystem.

Specified by:

getUri in class FileSystem

listStatus

public FileStatus[] listStatus(Path f)
                        throws FileNotFoundException,
                               IOException

Retrieve the status of a given path if it is a file, or of all the contained files if it is a directory.

Specified by:

listStatus in class FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

mkdirs

public boolean mkdirs(Path f,
             FsPermission permission)
               throws IOException

Description copied from class: FileSystem

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class FileSystem

Parameters:

f - path to create

permission - to apply to f

Throws:

IOException

mkdirs

public boolean mkdirs(Path f,
             FsPermission permission,
             boolean noUmask)
               throws IOException

Throws:

IOException

open

public FSDataInputStream open(Path f,
                     int bufferSize)
                       throws FileNotFoundException,
                              IOException

Description copied from class: FileSystem

Opens an FSDataInputStream at the indicated Path.

Specified by:

open in class FileSystem

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

FileNotFoundException

IOException

rename

public boolean rename(Path src,
             Path dst)
               throws FileNotFoundException,
                      IOException

Description copied from class: FileSystem

Renames Path src to Path dst. Can take place on local fs or remote DFS.

Specified by:

rename in class FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on failure

FileNotFoundException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(FileStatus file,
                                    long start,
                                    long len)
                                      throws IOException

Return an array containing hostnames, offset and size of portions of the given file. For WASB we'll just lie and give fake hosts to make sure we get many splits in MR jobs.

Overrides:

getFileBlockLocations in class FileSystem

Parameters:

file - FilesStatus to get data from

start - offset into the given file

len - length for which to get locations for

Throws:

IOException

setWorkingDirectory

public void setWorkingDirectory(Path newDir)

Set the working directory to the given directory.

Specified by:

setWorkingDirectory in class FileSystem

getWorkingDirectory

public Path getWorkingDirectory()

Description copied from class: FileSystem

Get the current working directory for the given file system

Specified by:

getWorkingDirectory in class FileSystem

Returns:

the directory pathname

setPermission

public void setPermission(Path p,
                 FsPermission permission)
                   throws FileNotFoundException,
                          IOException

Description copied from class: FileSystem

Set permission of a path.

Overrides:

setPermission in class FileSystem

Throws:

FileNotFoundException

IOException

setOwner

public void setOwner(Path p,
            String username,
            String groupname)
              throws IOException

Description copied from class: FileSystem

Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.

Overrides:

setOwner in class FileSystem

Parameters:

p - The path

username - If it is null, the original username remains unchanged.

groupname - If it is null, the original groupname remains unchanged.

Throws:

IOException

close

public void close()
           throws IOException

Description copied from class: FileSystem

No more filesystem operations are needed. Will release any held locks.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class FileSystem

Throws:

IOException

recoverFilesWithDanglingTempData

public void recoverFilesWithDanglingTempData(Path root,
                                    Path destination)
                                      throws IOException

Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there. If any are found, we move them to the destination given.

Parameters:

root - The root path to consider.

destination - The destination path to move any recovered files to.

Throws:

IOException

deleteFilesWithDanglingTempData

public void deleteFilesWithDanglingTempData(Path root)
                                     throws IOException

Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there. If any are found, we delete them.

Parameters:

root - The root path to consider.

Throws:

IOException

finalize

protected void finalize()
                 throws Throwable

Overrides:

finalize in class Object

Throws:

Throwable