org.apache.hadoop.fs.adl

Class AdlFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.adl.AdlFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable, BulkDeleteSource, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class AdlFileSystem
extends FileSystem

A FileSystem to access Azure Data Lake Store.

Field Summary

Fields

Modifier and Type	Field and Description
static String	SCHEME

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

TOKEN_LOG

Constructor Summary

Constructors

Constructor and Description
AdlFileSystem()

Method Summary

Modifier and Type	Method and Description
void	access(Path path, FsAction mode) Checks if the user can access a path.
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress) Append to an existing file (optional operation).
void	concat(Path trg, Path[] srcs) Concat existing files together.
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Create call semantic is handled differently in case of ADL.
FSDataOutputStream	createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) Deprecated. API only for 0.20-append
boolean	delete(Path path, boolean recursive) Delete a file.
static String	getAccountNameFromFQDN(String accountFQDN) Gets ADL account name from ADL FQDN.
AclStatus	getAclStatus(Path path) Gets the ACL of a file or directory.
com.microsoft.azure.datalake.store.ADLStoreClient	getAdlClient()
long	getBlockSize(Path f) Deprecated. Use getFileStatus() instead
ContentSummary	getContentSummary(Path f) Return the ContentSummary of a given Path.
protected AzureADTokenProvider	getCustomAccessTokenProvider(Configuration conf) This method is provided for convenience for derived classes to define custom AzureADTokenProvider instance.
long	getDefaultBlockSize() Deprecated. use getDefaultBlockSize(Path) instead
long	getDefaultBlockSize(Path f) Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
int	getDefaultPort() Get the default port for this FileSystem.
FileStatus	getFileStatus(Path f) Return a file status object that represents the path.
Path	getHomeDirectory() Constructing home directory locally is fine as long as Hadoop local user name and ADL user name relationship story is not fully baked yet.
short	getReplication(Path src) Deprecated. Use getFileStatus() instead
String	getScheme() Return the protocol scheme for this FileSystem.
protected String	getTransportScheme()
URI	getUri() Returns a URI which identifies this FileSystem.
Path	getWorkingDirectory() Get the current working directory for the given file system.
boolean	hasPathCapability(Path path, String capability) The base FileSystem implementation generally has no knowledge of the capabilities of actual implementations.
void	initialize(URI storeUri, Configuration originalConf) Called after a new FileSystem instance is constructed.
FileStatus[]	listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(Path path, FsPermission permission) Make the given file and all non-existent parents into directories.
void	modifyAclEntries(Path path, List<AclEntry> aclSpec) Modifies ACL entries of files and directories.
FSDataInputStream	open(Path f, int buffersize) Open call semantic is handled differently in case of ADL.
static Configuration	propagateAccountOptions(Configuration source, String accountName) Propagates account-specific settings into generic ADL configuration keys.
void	removeAcl(Path path) Removes all but the base ACL entries of files and directories.
void	removeAclEntries(Path path, List<AclEntry> aclSpec) Removes ACL entries from files and directories.
void	removeDefaultAcl(Path path) Removes all default ACL entries from files and directories.
boolean	rename(Path src, Path dst) Renames Path src to Path dst.
void	rename(Path src, Path dst, org.apache.hadoop.fs.Options.Rename... options) Deprecated.
void	setAcl(Path path, List<AclEntry> aclSpec) Fully replaces ACL of files and directories, discarding all existing entries.
void	setOwner(Path path, String owner, String group) Set owner of a path (i.e.
void	setPermission(Path path, FsPermission permission) Set permission of a path.
boolean	setReplication(Path p, short replication) Azure data lake does not support user configuration for data replication hence not leaving system to query on azure data lake.
void	setUserGroupRepresentationAsUPN(boolean enableUPN)
void	setWorkingDirectory(Path dir) Set the current working directory for the given file system.
boolean	supportsSymlinks() See AbstractFileSystem.supportsSymlinks().

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createBulkDelete, createDataOutputStreamBuilder, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAllStatistics, getAllStoragePolicies, getCanonicalServiceName, getCanonicalUri, getDefaultReplication, getDefaultReplication, getDefaultUri, getEnclosingRoot, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeXAttr, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageType, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, getAdditionalTokenIssuers, getDelegationToken

Field Detail

SCHEME

public static final String SCHEME

See Also:

Constant Field Values

Constructor Detail

AdlFileSystem

public AdlFileSystem()

Method Detail

getScheme

public String getScheme()

Description copied from class: FileSystem

Return the protocol scheme for this FileSystem.

This implementation throws an UnsupportedOperationException.

Overrides:

getScheme in class FileSystem

Returns:

the protocol scheme for this FileSystem.

getUri

public URI getUri()

Description copied from class: FileSystem

Returns a URI which identifies this FileSystem.

Specified by:

getUri in class FileSystem

Returns:

the URI of this filesystem.

getDefaultPort

public int getDefaultPort()

Description copied from class: FileSystem

Get the default port for this FileSystem.

Overrides:

getDefaultPort in class FileSystem

Returns:

the default port or 0 if there isn't one

supportsSymlinks

public boolean supportsSymlinks()

Description copied from class: FileSystem

See AbstractFileSystem.supportsSymlinks().

Overrides:

supportsSymlinks in class FileSystem

Returns:

if support symlinkls true, not false.

initialize

public void initialize(URI storeUri,
                       Configuration originalConf)
                throws IOException

Called after a new FileSystem instance is constructed.

Overrides:

initialize in class FileSystem

Parameters:

storeUri - a uri whose authority section names the host, port, etc. for this FileSystem

originalConf - the configuration to use for the FS. The account- specific options are patched over the base ones before any use is made of the config.

Throws:

IOException - on any failure to initialize this instance.

getCustomAccessTokenProvider

protected AzureADTokenProvider getCustomAccessTokenProvider(Configuration conf)
                                                     throws IOException

This method is provided for convenience for derived classes to define custom AzureADTokenProvider instance. In order to ensure secure hadoop infrastructure and user context for which respective AdlFileSystem instance is initialized, Loading AzureADTokenProvider is not sufficient. The order of loading AzureADTokenProvider is to first invoke getCustomAccessTokenProvider(Configuration), If method return null which means no implementation provided by derived classes, then configuration object is loaded to retrieve token configuration as specified is documentation. Custom token management takes the higher precedence during initialization.

Parameters:

conf - Configuration object

Returns:

null if the no custom AzureADTokenProvider token management is specified.

Throws:

IOException - if failed to initialize token provider.

getAdlClient

@VisibleForTesting
public com.microsoft.azure.datalake.store.ADLStoreClient getAdlClient()

getHomeDirectory

public Path getHomeDirectory()

Constructing home directory locally is fine as long as Hadoop local user name and ADL user name relationship story is not fully baked yet.

Overrides:

getHomeDirectory in class FileSystem

Returns:

Hadoop local user home directory.

create

public FSDataOutputStream create(Path f,
                                 FsPermission permission,
                                 boolean overwrite,
                                 int bufferSize,
                                 short replication,
                                 long blockSize,
                                 Progressable progress)
                          throws IOException

Create call semantic is handled differently in case of ADL. Create semantics is translated to Create/Append semantics. 1. No dedicated connection to server. 2. Buffering is locally done, Once buffer is full or flush is invoked on the by the caller. All the pending data is pushed to ADL as APPEND operation code. 3. On close - Additional call is send to server to close the stream, and release lock from the stream. Necessity of Create/Append semantics is 1. ADL backend server does not allow idle connection for longer duration . In case of slow writer scenario, observed connection timeout/Connection reset causing occasional job failures. 2. Performance boost to jobs which are slow writer, avoided network latency 3. ADL equally better performing with multiple of 4MB chunk as append calls.

Specified by:

create in class FileSystem

Parameters:

f - File path

permission - Access permission for the newly created file

overwrite - Remove existing file and recreate new one if true otherwise throw error if file exist

bufferSize - Buffer size, ADL backend does not honour

replication - Replication count, ADL backend does not honour

blockSize - Block size, ADL backend does not honour

progress - Progress indicator

Returns:

FSDataOutputStream OutputStream on which application can push stream of bytes

Throws:

IOException - when system error, internal server error or user error

See Also:

FileSystem.setPermission(Path, FsPermission)

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                             FsPermission permission,
                                             EnumSet<CreateFlag> flags,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             Progressable progress)
                                      throws IOException

Deprecated. API only for 0.20-append

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

Overrides:

createNonRecursive in class FileSystem

Parameters:

f - the file name to open

permission - Access permission for the newly created file

flags - CreateFlags to use for this stream.

bufferSize - the size of the buffer to be used. ADL backend does not honour

replication - required block replication for the file. ADL backend does not honour

blockSize - Block size, ADL backend does not honour

progress - Progress indicator

Returns:

output stream.

Throws:

IOException - when system error, internal server error or user error

See Also:

setPermission(Path, FsPermission)

append

public FSDataOutputStream append(Path f,
                                 int bufferSize,
                                 Progressable progress)
                          throws IOException

Append to an existing file (optional operation).

Specified by:

append in class FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used. ADL backend does not honour

progress - Progress indicator

Returns:

output stream.

Throws:

IOException - when system error, internal server error or user error

setReplication

public boolean setReplication(Path p,
                              short replication)
                       throws IOException

Azure data lake does not support user configuration for data replication hence not leaving system to query on azure data lake. Stub implementation

Overrides:

setReplication in class FileSystem

Parameters:

p - Not honoured

replication - Not honoured

Returns:

True hard coded since ADL file system does not support replication configuration

Throws:

IOException - No exception would not thrown in this case however aligning with parent api definition.

open

public FSDataInputStream open(Path f,
                              int buffersize)
                       throws IOException

Open call semantic is handled differently in case of ADL. Instead of network stream is returned to the user, Overridden FsInputStream is returned.

Specified by:

open in class FileSystem

Parameters:

f - File path

buffersize - Buffer size, Not honoured

Returns:

FSDataInputStream InputStream on which application can read stream of bytes

Throws:

IOException - when system error, internal server error or user error

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

IOException - when the path does not exist or any other error; IOException see specific implementation

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

IOException - when the path does not exist or any other error; IOException see specific implementation

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Renames Path src to Path dst. Can take place on local fs or remote DFS. ADLS support POSIX standard for rename operation.

Specified by:

rename in class FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on failure

rename

@Deprecated
public void rename(Path src,
                               Path dst,
                               org.apache.hadoop.fs.Options.Rename... options)
                        throws IOException

Deprecated.

Description copied from class: FileSystem

Renames Path src to Path dst

• Fails if src is a file and dst is a directory.
• Fails if src is a directory and dst is a file.
• Fails if the parent of dst does not exist or is a file.

If OVERWRITE option is not passed as an argument, rename fails if the dst already exists.

If OVERWRITE option is passed as an argument, rename overwrites the dst if it is a file or an empty directory. Rename fails if dst is a non-empty directory.

Note that atomicity of rename is dependent on the file system implementation. Please refer to the file system documentation for details. This default implementation is non atomic.

This method is deprecated since it is a temporary method added to support the transition from FileSystem to FileContext for user applications.

Overrides:

rename in class FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

options - rename options.

Throws:

FileNotFoundException - src path does not exist, or the parent path of dst does not exist.

FileAlreadyExistsException - dest path exists and is a file

ParentNotDirectoryException - if the parent path of dest is not a directory

IOException - on failure

concat

public void concat(Path trg,
                   Path[] srcs)
            throws IOException

Concat existing files together.

Overrides:

concat in class FileSystem

Parameters:

trg - the path to the target destination.

srcs - the paths to the sources to use for the concatenation.

Throws:

IOException - when system error, internal server error or user error

delete

public boolean delete(Path path,
                      boolean recursive)
               throws IOException

Delete a file.

Specified by:

delete in class FileSystem

Parameters:

path - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

IOException - when system error, internal server error or user error

mkdirs

public boolean mkdirs(Path path,
                      FsPermission permission)
               throws IOException

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class FileSystem

Parameters:

path - path to create

permission - to apply to path

Returns:

if mkdir success true, not false.

Throws:

IOException - IO failure

setOwner

public void setOwner(Path path,
                     String owner,
                     String group)
              throws IOException

Set owner of a path (i.e. a file or a directory). The parameters owner and group cannot both be null.

Overrides:

setOwner in class FileSystem

Parameters:

path - The path

owner - If it is null, the original username remains unchanged.

group - If it is null, the original groupname remains unchanged.

Throws:

IOException - IO failure

setPermission

public void setPermission(Path path,
                          FsPermission permission)
                   throws IOException

Set permission of a path.

Overrides:

setPermission in class FileSystem

Parameters:

path - The path

permission - Access permission

Throws:

IOException - IO failure

modifyAclEntries

public void modifyAclEntries(Path path,
                             List<AclEntry> aclSpec)
                      throws IOException

Modifies ACL entries of files and directories. This method can add new ACL entries or modify the permissions on existing ACL entries. All existing ACL entries that are not specified in this call are retained without changes. (Modifications are merged into the current ACL.)

Overrides:

modifyAclEntries in class FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing modifications

Throws:

IOException - if an ACL could not be modified

removeAclEntries

public void removeAclEntries(Path path,
                             List<AclEntry> aclSpec)
                      throws IOException

Removes ACL entries from files and directories. Other ACL entries are retained.

Overrides:

removeAclEntries in class FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing entries to remove

Throws:

IOException - if an ACL could not be modified

removeDefaultAcl

public void removeDefaultAcl(Path path)
                      throws IOException

Removes all default ACL entries from files and directories.

Overrides:

removeDefaultAcl in class FileSystem

Parameters:

path - Path to modify

Throws:

IOException - if an ACL could not be modified

removeAcl

public void removeAcl(Path path)
               throws IOException

Removes all but the base ACL entries of files and directories. The entries for user, group, and others are retained for compatibility with permission bits.

Overrides:

removeAcl in class FileSystem

Parameters:

path - Path to modify

Throws:

IOException - if an ACL could not be removed

setAcl

public void setAcl(Path path,
                   List<AclEntry> aclSpec)
            throws IOException

Fully replaces ACL of files and directories, discarding all existing entries.

Overrides:

setAcl in class FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing modifications, must include entries for user, group, and others for compatibility with permission bits.

Throws:

IOException - if an ACL could not be modified

getAclStatus

public AclStatus getAclStatus(Path path)
                       throws IOException

Gets the ACL of a file or directory.

Overrides:

getAclStatus in class FileSystem

Parameters:

path - Path to get

Returns:

AclStatus describing the ACL of the file or directory

Throws:

IOException - if an ACL could not be read

access

public void access(Path path,
                   FsAction mode)
            throws IOException

Checks if the user can access a path. The mode specifies which access checks to perform. If the requested permissions are granted, then the method returns normally. If access is denied, then the method throws an AccessControlException.

Parameters:

path - Path to check

mode - type of access to check

Throws:

AccessControlException - if access is denied

FileNotFoundException - if the path does not exist

IOException - see specific implementation

getContentSummary

public ContentSummary getContentSummary(Path f)
                                 throws IOException

Return the ContentSummary of a given Path.

Overrides:

getContentSummary in class FileSystem

Parameters:

f - path to use

Returns:

content summary.

Throws:

FileNotFoundException - if the path does not resolve

IOException - IO failure

getTransportScheme

@VisibleForTesting
protected String getTransportScheme()

getWorkingDirectory

public Path getWorkingDirectory()

Get the current working directory for the given file system.

Specified by:

getWorkingDirectory in class FileSystem

Returns:

the directory pathname

setWorkingDirectory

public void setWorkingDirectory(Path dir)

Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class FileSystem

Parameters:

dir - Working directory path.

getDefaultBlockSize

@Deprecated
public long getDefaultBlockSize()

Deprecated. use getDefaultBlockSize(Path) instead

Return the number of bytes that large input files should be optimally be split into to minimize i/o time.

Overrides:

getDefaultBlockSize in class FileSystem

Returns:

default block size.

getDefaultBlockSize

public long getDefaultBlockSize(Path f)

Return the number of bytes that large input files should be optimally be split into to minimize i/o time. The given path will be used to locate the actual filesystem. The full path does not have to exist.

Overrides:

getDefaultBlockSize in class FileSystem

Parameters:

f - path of file

Returns:

the default block size for the path's filesystem

getBlockSize

@Deprecated
public long getBlockSize(Path f)
                              throws IOException

Deprecated. Use getFileStatus() instead

Description copied from class: FileSystem

Get the block size for a particular file.

Overrides:

getBlockSize in class FileSystem

Parameters:

f - the filename

Returns:

the number of bytes in a block

Throws:

FileNotFoundException - if the path is not present

IOException - IO failure

getReplication

@Deprecated
public short getReplication(Path src)

Deprecated. Use getFileStatus() instead

Get replication.

Overrides:

getReplication in class FileSystem

Parameters:

src - file name

Returns:

file replication

setUserGroupRepresentationAsUPN

@VisibleForTesting
public void setUserGroupRepresentationAsUPN(boolean enableUPN)

getAccountNameFromFQDN

public static String getAccountNameFromFQDN(String accountFQDN)

Gets ADL account name from ADL FQDN.

Parameters:

accountFQDN - ADL account fqdn

Returns:

ADL account name

propagateAccountOptions

public static Configuration propagateAccountOptions(Configuration source,
                                                    String accountName)

Propagates account-specific settings into generic ADL configuration keys. This is done by propagating the values of the form fs.adl.account.${account_name}.key to fs.adl.key, for all values of "key" The source of the updated property is set to the key name of the account property, to aid in diagnostics of where things came from. Returns a new configuration. Why the clone? You can use the same conf for different filesystems, and the original values are not updated.

Parameters:

source - Source Configuration object

accountName - account name. Must not be empty

Returns:

a (potentially) patched clone of the original

hasPathCapability

public boolean hasPathCapability(Path path,
                                 String capability)
                          throws IOException

Description copied from class: FileSystem

The base FileSystem implementation generally has no knowledge of the capabilities of actual implementations. Unless it has a way to explicitly determine the capabilities, this method returns false. Probe for a specific capability under the given path. If the function returns true, this instance is explicitly declaring that the capability is available. If the function returns false, it can mean one of:

• The capability is not known.
• The capability is known but it is not supported.
• The capability is known but the filesystem does not know if it is supported under the supplied path.

The core guarantee which a caller can rely on is: if the predicate returns true, then the specific operation/behavior can be expected to be supported. However a specific call may be rejected for permission reasons, the actual file/directory not being present, or some other failure during the attempted execution of the operation.

Implementors: PathCapabilitiesSupport can be used to help implement this method.

Specified by:

hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities

Overrides:

hasPathCapability in class FileSystem

Parameters:

path - path to query the capability of.

capability - non-null, non-empty string to query the path for support.

Returns:

true if the capability is supported under that part of the FS.

Throws:

IOException - this should not be raised, except on problems resolving paths or relaying the call.