@InterfaceAudience.Public @InterfaceStability.Stable public class NativeAzureFileSystem extends FileSystem
FileSystem
for reading and writing files stored on Windows Azure. This implementation is
blob-based and stores files on Azure in their native form so they can be read
by other Azure tools.Modifier and Type | Field and Description |
---|---|
static String |
APPEND_SUPPORT_ENABLE_PROPERTY_NAME |
static String |
AZURE_CHOWN_USERLIST_PROPERTY_NAME
Configuration property used to specify list of users that can perform
chown operation when authorization is enabled in WASB.
|
static String |
AZURE_DELETE_THREADS
The configuration property to set number of threads to be used for delete operation.
|
static String |
AZURE_RENAME_THREADS
The configuration property to set number of threads to be used for rename operation.
|
static int |
DEFAULT_AZURE_DELETE_THREADS
The default number of threads to be used for delete operation.
|
static int |
DEFAULT_AZURE_RENAME_THREADS
The default number of threads to be used for rename operation.
|
static String |
KEY_AZURE_AUTHORIZATION
Configuration key to enable authorization support in WASB.
|
static org.slf4j.Logger |
LOG |
static String |
SKIP_AZURE_METRICS_PROPERTY_NAME |
DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX
Constructor and Description |
---|
NativeAzureFileSystem() |
NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.fs.azure.SelfRenewingLease |
acquireLease(Path path)
Get a self-renewing lease on the specified file.
|
FSDataOutputStream |
append(Path f,
int bufferSize,
Progressable progress)
This optional operation is not yet supported.
|
protected void |
checkPath(Path path)
Check that a Path belongs to this FileSystem.
|
void |
close()
Close this FileSystem instance.
|
FSDataOutputStream |
create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
createNonRecursive(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
createNonRecursive(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
createNonRecursive(Path f,
FsPermission permission,
EnumSet<CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
boolean |
delete(Path path)
Deprecated.
|
boolean |
delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
delete(Path f,
boolean recursive,
boolean skipParentFolderLastModifiedTimeUpdate)
Delete the specified file or folder.
|
void |
deleteFilesWithDanglingTempData(Path root)
Looks under the given root path for any blob that are left "dangling",
meaning that they are place-holder blobs that we created while we upload
the data to a temporary blob, but for some reason we crashed in the middle
of the upload and left them there.
|
protected void |
finalize() |
Token<?> |
getDelegationToken(String renewer)
Get a delegation token from remote service endpoint if
'fs.azure.enable.kerberos.support' is set to 'true'.
|
BlockLocation[] |
getFileBlockLocations(FileStatus file,
long start,
long len)
Return an array containing hostnames, offset and size of
portions of the given file.
|
FileStatus |
getFileStatus(Path f)
Return a file status object that represents the path.
|
Path |
getHomeDirectory()
Return the current user's home directory in this FileSystem.
|
AzureFileSystemInstrumentation |
getInstrumentation()
Gets the metrics source for this file system.
|
String |
getOwnerForPath(Path absolutePath) |
String |
getScheme()
Return the protocol scheme for this FileSystem.
|
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore |
getStore()
For unit test purposes, retrieves the AzureNativeFileSystemStore store
backing this file system.
|
org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor |
getThreadPoolExecutor(int threadCount,
String threadNamePrefix,
String operation,
String key,
String config) |
URI |
getUri()
Returns a URI which identifies this FileSystem.
|
Path |
getWorkingDirectory()
Get the current working directory for the given FileSystem
|
void |
initialize(URI uri,
Configuration conf)
Initialize a FileSystem.
|
FileStatus[] |
listStatus(Path f)
Retrieve the status of a given path if it is a file, or of all the
contained files if it is a directory.
|
Path |
makeAbsolute(Path path)
Get the absolute version of the path (fully qualified).
|
boolean |
mkdirs(Path f,
FsPermission permission)
Make the given file and all non-existent parents into
directories.
|
boolean |
mkdirs(Path f,
FsPermission permission,
boolean noUmask) |
static String |
newMetricsSourceName()
Creates a new metrics source name that's unique within this process.
|
FSDataInputStream |
open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
String |
pathToKey(Path path)
Convert the path to a key.
|
void |
recoverFilesWithDanglingTempData(Path root,
Path destination)
Looks under the given root path for any blob that are left "dangling",
meaning that they are place-holder blobs that we created while we upload
the data to a temporary blob, but for some reason we crashed in the middle
of the upload and left them there.
|
boolean |
rename(Path src,
Path dst)
Renames Path src to Path dst.
|
void |
setOwner(Path p,
String username,
String groupname)
Set owner of a path (i.e.
|
void |
setPermission(Path p,
FsPermission permission)
Set permission of a path.
|
void |
setWorkingDirectory(Path newDir)
Set the working directory to the given directory.
|
void |
updateWasbAuthorizer(org.apache.hadoop.fs.azure.WasbAuthorizerInterface authorizer) |
addDelegationTokens, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createSnapshot, createSnapshot, createSymlink, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy
getConf, setConf
public static final org.slf4j.Logger LOG
public static final String AZURE_CHOWN_USERLIST_PROPERTY_NAME
public static final String SKIP_AZURE_METRICS_PROPERTY_NAME
public static final String APPEND_SUPPORT_ENABLE_PROPERTY_NAME
public static final String AZURE_RENAME_THREADS
public static final int DEFAULT_AZURE_RENAME_THREADS
public static final String AZURE_DELETE_THREADS
public static final int DEFAULT_AZURE_DELETE_THREADS
public static final String KEY_AZURE_AUTHORIZATION
public NativeAzureFileSystem()
public NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store)
public String getScheme()
FileSystem
This implementation throws an UnsupportedOperationException
.
getScheme
in class FileSystem
public static String newMetricsSourceName()
protected void checkPath(Path path)
FileSystem
checkPath
in class FileSystem
path
- to checkpublic void initialize(URI uri, Configuration conf) throws IOException, IllegalArgumentException
FileSystem
initialize
in class FileSystem
uri
- a URI whose authority section names the host, port, etc.
for this FileSystemconf
- the configurationIOException
- on any failure to initialize this instance.IllegalArgumentException
- if the URI is considered invalid.public Path getHomeDirectory()
FileSystem
"/user/$USER/"
.getHomeDirectory
in class FileSystem
public void updateWasbAuthorizer(org.apache.hadoop.fs.azure.WasbAuthorizerInterface authorizer)
public String pathToKey(Path path)
path
- path converted to a keypublic Path makeAbsolute(Path path)
path
- path to be absolute path.public org.apache.hadoop.fs.azure.AzureNativeFileSystemStore getStore()
public AzureFileSystemInstrumentation getInstrumentation()
public FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException
append
in class FileSystem
f
- the existing file to be appended.bufferSize
- the size of the buffer to be used.progress
- for reporting progress if it is not null.IOException
- IO failurepublic FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystem
create
in class FileSystem
f
- the file name to openpermission
- file permissionoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.blockSize
- block sizeprogress
- the progress reporterIOException
- IO failureFileSystem.setPermission(Path, FsPermission)
public org.apache.hadoop.fs.azure.SelfRenewingLease acquireLease(Path path) throws AzureException
path
- path whose lease to be renewed.AzureException
- when not being able to acquire a lease on the pathpublic FSDataOutputStream createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystem
createNonRecursive
in class FileSystem
f
- the file name to openpermission
- file permissionoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.blockSize
- block sizeprogress
- the progress reporterIOException
- IO failureFileSystem.setPermission(Path, FsPermission)
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystem
createNonRecursive
in class FileSystem
f
- the file name to openpermission
- file permissionflags
- CreateFlag
s to use for this stream.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.blockSize
- block sizeprogress
- the progress reporterIOException
- IO failureFileSystem.setPermission(Path, FsPermission)
public FSDataOutputStream createNonRecursive(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystem
createNonRecursive
in class FileSystem
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.blockSize
- block sizeprogress
- the progress reporterIOException
- IO failureFileSystem.setPermission(Path, FsPermission)
@Deprecated public boolean delete(Path path) throws IOException
FileSystem
delete
in class FileSystem
IOException
public boolean delete(Path f, boolean recursive) throws IOException
FileSystem
delete
in class FileSystem
f
- the path to delete.recursive
- if path is a directory and set to
true, the directory is deleted else throws an exception. In
case of a file the recursive can be set to either true or false.IOException
- IO failurepublic boolean delete(Path f, boolean recursive, boolean skipParentFolderLastModifiedTimeUpdate) throws IOException
f
- file path to be deleted.recursive
- specify deleting recursively or not.skipParentFolderLastModifiedTimeUpdate
- If true, don't update the folder last
modified time.IOException
- Thrown when fail to delete file or directory.public org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor getThreadPoolExecutor(int threadCount, String threadNamePrefix, String operation, String key, String config)
public FileStatus getFileStatus(Path f) throws FileNotFoundException, IOException
FileSystem
getFileStatus
in class FileSystem
f
- The path we want information fromFileNotFoundException
- when the path does not existIOException
- see specific implementationpublic URI getUri()
FileSystem
getUri
in class FileSystem
public FileStatus[] listStatus(Path f) throws FileNotFoundException, IOException
listStatus
in class FileSystem
f
- given pathFileNotFoundException
- when the path does not existIOException
- see specific implementationpublic boolean mkdirs(Path f, FsPermission permission) throws IOException
FileSystem
mkdirs
in class FileSystem
f
- path to createpermission
- to apply to fIOException
- IO failurepublic boolean mkdirs(Path f, FsPermission permission, boolean noUmask) throws IOException
IOException
public FSDataInputStream open(Path f, int bufferSize) throws FileNotFoundException, IOException
FileSystem
open
in class FileSystem
f
- the file name to openbufferSize
- the size of the buffer to be used.IOException
- IO failureFileNotFoundException
public boolean rename(Path src, Path dst) throws FileNotFoundException, IOException
FileSystem
rename
in class FileSystem
src
- path to be renameddst
- new path after renameIOException
- on failureFileNotFoundException
public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws IOException
getFileBlockLocations
in class FileSystem
file
- FilesStatus to get data fromstart
- offset into the given filelen
- length for which to get locations forIOException
- IO failurepublic void setWorkingDirectory(Path newDir)
setWorkingDirectory
in class FileSystem
newDir
- Path of new working directorypublic Path getWorkingDirectory()
FileSystem
getWorkingDirectory
in class FileSystem
public void setPermission(Path p, FsPermission permission) throws FileNotFoundException, IOException
FileSystem
setPermission
in class FileSystem
p
- The pathpermission
- permissionIOException
- IO failureFileNotFoundException
public void setOwner(Path p, String username, String groupname) throws IOException
FileSystem
setOwner
in class FileSystem
p
- The pathusername
- If it is null, the original username remains unchanged.groupname
- If it is null, the original groupname remains unchanged.IOException
- IO failurepublic void close() throws IOException
FileSystem
FileSystem.deleteOnExit(Path)
, and remove this FS instance
from the cache, if cached.
After this operation, the outcome of any method call on this FileSystem
instance, or any input/output stream created by it is undefined.close
in interface Closeable
close
in interface AutoCloseable
close
in class FileSystem
IOException
- IO failurepublic Token<?> getDelegationToken(String renewer) throws IOException
renewer
- the account name that is allowed to renew the token.IOException
- thrown when getting the current user.public void recoverFilesWithDanglingTempData(Path root, Path destination) throws IOException
root
- The root path to consider.destination
- The destination path to move any recovered files to.IOException
- Thrown when fail to recover files.public void deleteFilesWithDanglingTempData(Path root) throws IOException
root
- The root path to consider.IOException
- Thrown when fail to delete.protected void finalize() throws Throwable
public String getOwnerForPath(Path absolutePath) throws IOException
IOException
Copyright © 2017 Apache Software Foundation. All rights reserved.