Package org.apache.hadoop.fs.azurebfs.constants

Class ConfigurationKeys

java.lang.Object
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys
@Public @Evolving public final class ConfigurationKeys extends Object
Responsible to keep all the Azure Blob File System configurations keys in Hadoop configuration file.

  • Field Details

    • FS_AZURE_ACCOUNT_IS_HNS_ENABLED

      public static final String FS_AZURE_ACCOUNT_IS_HNS_ENABLED
      Config to specify if the configured account is HNS enabled or not. If this config is not set, getacl call is made on account filesystem root path on DFS Endpoint to determine HNS status.
      See Also:

    • FS_AZURE_FNS_ACCOUNT_SERVICE_TYPE

      public static final String FS_AZURE_FNS_ACCOUNT_SERVICE_TYPE
      Config to specify which AbfsServiceType to use with HNS-Disabled Account type. Default value will be identified from URL used to initialize filesystem. This will allow an override to choose service endpoint in cases where any local DNS resolution is set for account and driver is unable to detect intended endpoint from the url used to initialize filesystem. If configured Blob for HNS-Enabled account, FS init will fail. Value "fs.azure.fns.account.service.type" case-insensitive "DFS" or "BLOB"
      See Also:

    • FS_AZURE_INGRESS_SERVICE_TYPE

      public static final String FS_AZURE_INGRESS_SERVICE_TYPE
      Config to specify which AbfsServiceType to use only for Ingress Operations. Other operations will continue to move to the FS configured service endpoint. Value "fs.azure.ingress.service.type" case-insensitive "DFS" or "BLOB"
      See Also:

    • FS_AZURE_ENABLE_DFSTOBLOB_FALLBACK

      public static final String FS_AZURE_ENABLE_DFSTOBLOB_FALLBACK
      Config to be set only for cases where traffic over dfs endpoint is experiencing compatibility issues and need to move to blob for mitigation. Value "fs.azure.enable.dfstoblob.fallback" case-insensitive "True" or "False"
      See Also:

    • FS_AZURE_ACCOUNT_IS_EXPECT_HEADER_ENABLED

      public static final String FS_AZURE_ACCOUNT_IS_EXPECT_HEADER_ENABLED
      Enable or disable expect hundred continue header. Value: "fs.azure.account.expect.header.enabled".
      See Also:

    • FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME

      public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME
      See Also:

    • FS_AZURE_METRICS_ACCOUNT_NAME

      public static final String FS_AZURE_METRICS_ACCOUNT_NAME
      Config to set separate metrics account in case user don't want to use existing storage account for metrics collection. Value: "fs.azure.metrics.account.name".
      See Also:

    • FS_AZURE_METRICS_ACCOUNT_KEY

      public static final String FS_AZURE_METRICS_ACCOUNT_KEY
      Config to set metrics account key for @FS_AZURE_METRICS_ACCOUNT_NAME. Value: "fs.azure.metrics.account.key".
      See Also:

    • FS_AZURE_METRICS_FORMAT

      public static final String FS_AZURE_METRICS_FORMAT
      Config to set metrics format. Possible values are MetricFormat Value: "fs.azure.metrics.format".
      See Also:

    • FS_AZURE_METRICS_COLLECTION_ENABLED

      public static final String FS_AZURE_METRICS_COLLECTION_ENABLED
      Config to enable or disable metrics collection. Value: "fs.azure.metrics.collection.enabled".
      See Also:

    • FS_AZURE_METRICS_SHOULD_EMIT_ON_IDLE_TIME

      public static final String FS_AZURE_METRICS_SHOULD_EMIT_ON_IDLE_TIME
      Config to enable or disable emitting metrics when idle time exceeds threshold. Value: "fs.azure.metrics.should.emit.on.idle.time".
      See Also:

    • FS_AZURE_METRICS_EMIT_THRESHOLD

      public static final String FS_AZURE_METRICS_EMIT_THRESHOLD
      Config to set threshold for emitting metrics when number of operations exceeds threshold. Value: "fs.azure.metrics.emit.threshold".
      See Also:

    • FS_AZURE_METRICS_EMIT_THRESHOLD_INTERVAL_SECS

      public static final String FS_AZURE_METRICS_EMIT_THRESHOLD_INTERVAL_SECS
      Config to set interval in seconds to check for threshold breach for emitting metrics. If the number of operations exceed threshold within this interval, metrics will be emitted. Value: "fs.azure.metrics.emit.threshold.interval.secs".
      See Also:

    • FS_AZURE_METRICS_EMIT_INTERVAL_MINS

      public static final String FS_AZURE_METRICS_EMIT_INTERVAL_MINS
      Config to set interval in minutes for emitting metrics in regular time intervals. Value: "fs.azure.metrics.emit.interval.mins".
      See Also:

    • FS_AZURE_METRICS_MAX_CALLS_PER_SECOND

      public static final String FS_AZURE_METRICS_MAX_CALLS_PER_SECOND
      Config to set maximum metrics calls per second. Value: "fs.azure.metrics.max.calls.per.second".
      See Also:

    • FS_AZURE_METRICS_BACKOFF_RETRY_ENABLED

      public static final String FS_AZURE_METRICS_BACKOFF_RETRY_ENABLED
      Config to enable or disable backoff retry metrics collection. Value: "fs.azure.metrics.backoff.retry.enabled".
      See Also:

    • FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME_REGX

      public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME_REGX
      See Also:

    • FS_AZURE_SECURE_MODE

      public static final String FS_AZURE_SECURE_MODE
      See Also:

    • FS_AZURE_ACCOUNT_LEVEL_THROTTLING_ENABLED

      public static final String FS_AZURE_ACCOUNT_LEVEL_THROTTLING_ENABLED
      See Also:

    • AZURE_MIN_BACKOFF_INTERVAL

      public static final String AZURE_MIN_BACKOFF_INTERVAL
      See Also:

    • AZURE_MAX_BACKOFF_INTERVAL

      public static final String AZURE_MAX_BACKOFF_INTERVAL
      See Also:

    • AZURE_STATIC_RETRY_FOR_CONNECTION_TIMEOUT_ENABLED

      public static final String AZURE_STATIC_RETRY_FOR_CONNECTION_TIMEOUT_ENABLED
      See Also:

    • AZURE_STATIC_RETRY_INTERVAL

      public static final String AZURE_STATIC_RETRY_INTERVAL
      See Also:

    • AZURE_BACKOFF_INTERVAL

      public static final String AZURE_BACKOFF_INTERVAL
      See Also:

    • AZURE_MAX_IO_RETRIES

      public static final String AZURE_MAX_IO_RETRIES
      See Also:

    • AZURE_CUSTOM_TOKEN_FETCH_RETRY_COUNT

      public static final String AZURE_CUSTOM_TOKEN_FETCH_RETRY_COUNT
      See Also:

    • AZURE_HTTP_CONNECTION_TIMEOUT

      public static final String AZURE_HTTP_CONNECTION_TIMEOUT
      Config to set HTTP Connection Timeout Value for Rest Operations. Value: "fs.azure.http.connection.timeout".
      See Also:

    • AZURE_HTTP_READ_TIMEOUT

      public static final String AZURE_HTTP_READ_TIMEOUT
      Config to set HTTP Read Timeout Value for Rest Operations. Value: "fs.azure.http.read.timeout".
      See Also:

    • AZURE_EXPECT_100CONTINUE_WAIT_TIMEOUT

      public static final String AZURE_EXPECT_100CONTINUE_WAIT_TIMEOUT
      Config to set HTTP Expect100-Continue Wait Timeout Value for Rest Operations. Value: "fs.azure.http.expect.100continue.wait.timeout".
      See Also:

    • AZURE_OAUTH_TOKEN_FETCH_RETRY_COUNT

      public static final String AZURE_OAUTH_TOKEN_FETCH_RETRY_COUNT
      See Also:

    • AZURE_OAUTH_TOKEN_FETCH_RETRY_MIN_BACKOFF

      public static final String AZURE_OAUTH_TOKEN_FETCH_RETRY_MIN_BACKOFF
      See Also:

    • AZURE_OAUTH_TOKEN_FETCH_RETRY_MAX_BACKOFF

      public static final String AZURE_OAUTH_TOKEN_FETCH_RETRY_MAX_BACKOFF
      See Also:

    • AZURE_OAUTH_TOKEN_FETCH_RETRY_DELTA_BACKOFF

      public static final String AZURE_OAUTH_TOKEN_FETCH_RETRY_DELTA_BACKOFF
      See Also:

    • AZURE_WRITE_MAX_CONCURRENT_REQUESTS

      public static final String AZURE_WRITE_MAX_CONCURRENT_REQUESTS
      See Also:

    • AZURE_WRITE_MAX_REQUESTS_TO_QUEUE

      public static final String AZURE_WRITE_MAX_REQUESTS_TO_QUEUE
      See Also:

    • AZURE_WRITE_BUFFER_SIZE

      public static final String AZURE_WRITE_BUFFER_SIZE
      See Also:

    • FS_AZURE_BLOCK_UPLOAD_ACTIVE_BLOCKS

      public static final String FS_AZURE_BLOCK_UPLOAD_ACTIVE_BLOCKS
      Maximum Number of blocks a single output stream can have active (uploading, or queued to the central FileSystem instance's pool of queued operations. This stops a single stream overloading the shared thread pool. "fs.azure.block.upload.active.blocks"

      Default is FileSystemConfigurations.BLOCK_UPLOAD_ACTIVE_BLOCKS_DEFAULT

      See Also:

    • FS_AZURE_BLOCK_UPLOAD_BUFFER_DIR

      public static final String FS_AZURE_BLOCK_UPLOAD_BUFFER_DIR
      Buffer directory path for uploading AbfsOutputStream data blocks. Value: "fs.azure.buffer.dir"
      See Also:

    • DATA_BLOCKS_BUFFER

      public static final String DATA_BLOCKS_BUFFER
      What data block buffer to use.
      Options include: "disk", "array", and "bytebuffer"(Default).
      Default is FileSystemConfigurations.DATA_BLOCKS_BUFFER_DEFAULT. Value: "fs.azure.data.blocks.buffer"
      See Also:

    • AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION

      public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION
      If the data size written by Hadoop app is small, i.e. data size : (a) before any of HFlush/HSync call is made or (b) between 2 HFlush/Hsync API calls is less than write buffer size, 2 separate calls, one for append and another for flush are made. By enabling the small write optimization, a single call will be made to perform both append and flush operations and hence reduce request count.
      See Also:

    • AZURE_READ_BUFFER_SIZE

      public static final String AZURE_READ_BUFFER_SIZE
      See Also:

    • AZURE_READ_SMALL_FILES_COMPLETELY

      public static final String AZURE_READ_SMALL_FILES_COMPLETELY
      See Also:

    • AZURE_READ_AHEAD_RANGE

      public static final String AZURE_READ_AHEAD_RANGE
      Read ahead range parameter which can be set by user. Default value is FileSystemConfigurations.DEFAULT_READ_AHEAD_RANGE. This might reduce number of calls to remote as next requested data could already be present in buffer "fs.azure.readahead.range".
      See Also:

    • AZURE_BLOCK_SIZE_PROPERTY_NAME

      public static final String AZURE_BLOCK_SIZE_PROPERTY_NAME
      See Also:

    • AZURE_BLOCK_LOCATION_HOST_PROPERTY_NAME

      public static final String AZURE_BLOCK_LOCATION_HOST_PROPERTY_NAME
      See Also:

    • AZURE_CONCURRENT_CONNECTION_VALUE_OUT

      public static final String AZURE_CONCURRENT_CONNECTION_VALUE_OUT
      See Also:

    • AZURE_CONCURRENT_CONNECTION_VALUE_IN

      public static final String AZURE_CONCURRENT_CONNECTION_VALUE_IN
      See Also:

    • AZURE_TOLERATE_CONCURRENT_APPEND

      public static final String AZURE_TOLERATE_CONCURRENT_APPEND
      See Also:

    • AZURE_LIST_MAX_RESULTS

      public static final String AZURE_LIST_MAX_RESULTS
      See Also:

    • AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION

      public static final String AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION
      See Also:

    • AZURE_SKIP_USER_GROUP_METADATA_DURING_INITIALIZATION

      public static final String AZURE_SKIP_USER_GROUP_METADATA_DURING_INITIALIZATION
      See Also:

    • FS_AZURE_ENABLE_AUTOTHROTTLING

      public static final String FS_AZURE_ENABLE_AUTOTHROTTLING
      See Also:

    • FS_AZURE_METRIC_IDLE_TIMEOUT

      public static final String FS_AZURE_METRIC_IDLE_TIMEOUT
      See Also:

    • FS_AZURE_METRIC_ANALYSIS_TIMEOUT

      public static final String FS_AZURE_METRIC_ANALYSIS_TIMEOUT
      See Also:

    • FS_AZURE_ACCOUNT_OPERATION_IDLE_TIMEOUT

      public static final String FS_AZURE_ACCOUNT_OPERATION_IDLE_TIMEOUT
      See Also:

    • FS_AZURE_ANALYSIS_PERIOD

      public static final String FS_AZURE_ANALYSIS_PERIOD
      See Also:

    • FS_AZURE_ALWAYS_USE_HTTPS

      public static final String FS_AZURE_ALWAYS_USE_HTTPS
      See Also:

    • FS_AZURE_ATOMIC_RENAME_KEY

      public static final String FS_AZURE_ATOMIC_RENAME_KEY
      See Also:

    • FS_AZURE_ENABLE_CONDITIONAL_CREATE_OVERWRITE

      public static final String FS_AZURE_ENABLE_CONDITIONAL_CREATE_OVERWRITE
      This config ensures that during create overwrite an existing file will be overwritten only if there is a match on the eTag of existing file.
      See Also:

    • FS_AZURE_ENABLE_MKDIR_OVERWRITE

      public static final String FS_AZURE_ENABLE_MKDIR_OVERWRITE
      See Also:

    • FS_AZURE_APPEND_BLOB_KEY

      public static final String FS_AZURE_APPEND_BLOB_KEY
      Provides a config to provide comma separated path prefixes on which Appendblob based files are created Default is empty.
      See Also:

    • FS_AZURE_INFINITE_LEASE_KEY

      public static final String FS_AZURE_INFINITE_LEASE_KEY
      Provides a config to provide comma separated path prefixes which support infinite leases. Files under these paths will be leased when created or opened for writing and the lease will be released when the file is closed. The lease may be broken with the breakLease method on AzureBlobFileSystem. Default is empty.
      See Also:

    • FS_AZURE_LEASE_THREADS

      public static final String FS_AZURE_LEASE_THREADS
      Provides a number of threads to use for lease operations for infinite lease directories. Must be set to a minimum of 1 if infinite lease directories are to be used. Default is 0.
      See Also:

    • FS_AZURE_READ_AHEAD_QUEUE_DEPTH

      public static final String FS_AZURE_READ_AHEAD_QUEUE_DEPTH
      See Also:

    • FS_AZURE_ALWAYS_READ_BUFFER_SIZE

      public static final String FS_AZURE_ALWAYS_READ_BUFFER_SIZE
      See Also:

    • FS_AZURE_READ_AHEAD_BLOCK_SIZE

      public static final String FS_AZURE_READ_AHEAD_BLOCK_SIZE
      See Also:

    • FS_AZURE_READ_POLICY

      public static final String FS_AZURE_READ_POLICY
      Provides hint for the read workload pattern. Possible Values Exposed in Options.OpenFileOptions.FS_OPTION_OPENFILE_READ_POLICIES
      See Also:

    • FS_AZURE_ENABLE_FLUSH

      public static final String FS_AZURE_ENABLE_FLUSH
      Provides a config control to enable or disable ABFS Flush operations - HFlush and HSync. Default is true.
      See Also:

    • FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH

      public static final String FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH
      Provides a config control to disable or enable OutputStream Flush API operations in AbfsOutputStream. Flush() will trigger actions that guarantee that buffered data is persistent with a perf cost while the API documentation does not have such expectations of data being persisted. Default value of this config is true.
      See Also:

    • FS_AZURE_USER_AGENT_PREFIX_KEY

      public static final String FS_AZURE_USER_AGENT_PREFIX_KEY
      See Also:

    • FS_AZURE_CLIENT_CORRELATIONID

      public static final String FS_AZURE_CLIENT_CORRELATIONID
      The client correlation ID provided over config that will be added to x-ms-client-request-Id header. Defaults to empty string if the length and character constraints are not satisfied.
      See Also:

    • FS_AZURE_TRACINGHEADER_FORMAT

      public static final String FS_AZURE_TRACINGHEADER_FORMAT
      See Also:

    • FS_AZURE_CLUSTER_NAME

      public static final String FS_AZURE_CLUSTER_NAME
      See Also:

    • FS_AZURE_CLUSTER_TYPE

      public static final String FS_AZURE_CLUSTER_TYPE
      See Also:

    • FS_AZURE_SSL_CHANNEL_MODE_KEY

      public static final String FS_AZURE_SSL_CHANNEL_MODE_KEY
      See Also:

    • FS_AZURE_ENABLE_CHECK_ACCESS

      public static final String FS_AZURE_ENABLE_CHECK_ACCESS
      Provides a config to enable/disable the checkAccess API. By default this will be FileSystemConfigurations.DEFAULT_ENABLE_CHECK_ACCESS.
      See Also:

    • FS_AZURE_USE_UPN

      public static final String FS_AZURE_USE_UPN
      See Also:

    • FS_AZURE_FILE_OWNER_ENABLE_SHORTNAME

      public static final String FS_AZURE_FILE_OWNER_ENABLE_SHORTNAME
      User principal names (UPNs) have the format “{alias}@{domain}”. If true, only {alias} is included when a UPN would otherwise appear in the output of APIs like getFileStatus, getOwner, getAclStatus, etc. Default is false.
      See Also:

    • FS_AZURE_FILE_OWNER_DOMAINNAME

      public static final String FS_AZURE_FILE_OWNER_DOMAINNAME
      If the domain name is specified and “fs.azure.identity.transformer.enable.short.name” is true, then the {alias} part of a UPN can be specified as input to APIs like setOwner and setAcl and it will be transformed to a UPN by appending @ and the domain specified by this configuration property.
      See Also:

    • FS_AZURE_OVERRIDE_OWNER_SP

      public static final String FS_AZURE_OVERRIDE_OWNER_SP
      An Azure Active Directory object ID (oid) used as the replacement for names contained in the list specified by “fs.azure.identity.transformer.service.principal.substitution.list. Notice that instead of setting oid, you can also set $superuser.
      See Also:

    • FS_AZURE_OVERRIDE_OWNER_SP_LIST

      public static final String FS_AZURE_OVERRIDE_OWNER_SP_LIST
      A comma separated list of names to be replaced with the service principal ID specified by “fs.default.identity.transformer.service.principal.id”. This substitution occurs when setOwner, setAcl, modifyAclEntries, or removeAclEntries are invoked with identities contained in the substitution list. Notice that when in non-secure cluster, asterisk symbol "*" can be used to match all user/group.
      See Also:

    • FS_AZURE_SKIP_SUPER_USER_REPLACEMENT

      public static final String FS_AZURE_SKIP_SUPER_USER_REPLACEMENT
      By default this is set as false, so “$superuser” is replaced with the current user when it appears as the owner or owning group of a file or directory. To disable it, set it as true.
      See Also:

    • AZURE_KEY_ACCOUNT_KEYPROVIDER

      public static final String AZURE_KEY_ACCOUNT_KEYPROVIDER
      See Also:

    • AZURE_KEY_ACCOUNT_SHELLKEYPROVIDER_SCRIPT

      public static final String AZURE_KEY_ACCOUNT_SHELLKEYPROVIDER_SCRIPT
      See Also:

    • FS_AZURE_ENABLE_READAHEAD

      public static final String FS_AZURE_ENABLE_READAHEAD
      Enable or disable readahead V1 in AbfsInputStream. Value: "fs.azure.enable.readahead".
      See Also:

    • FS_AZURE_ENABLE_READAHEAD_V2

      public static final String FS_AZURE_ENABLE_READAHEAD_V2
      Enable or disable readahead V2 in AbfsInputStream. This will work independent of V1. Value: "fs.azure.enable.readahead.v2".
      See Also:

    • FS_AZURE_ENABLE_READAHEAD_V2_DYNAMIC_SCALING

      public static final String FS_AZURE_ENABLE_READAHEAD_V2_DYNAMIC_SCALING
      Enable or disable dynamic scaling of thread pool and buffer pool of readahead V2. Value: "fs.azure.enable.readahead.v2.dynamic.scaling".
      See Also:

    • FS_AZURE_ENABLE_PREFETCH_REQUEST_PRIORITY

      public static final String FS_AZURE_ENABLE_PREFETCH_REQUEST_PRIORITY
      Enable or disable request priority for prefetch requests Value: "fs.azure.enable.prefetch.request.priority".
      See Also:

    • FS_AZURE_PREFETCH_REQUEST_PRIORITY_VALUE

      public static final String FS_AZURE_PREFETCH_REQUEST_PRIORITY_VALUE
      Request priority value for prefetch requests Value: "fs.azure.prefetch.request.priority.value".
      See Also:

    • FS_AZURE_READAHEAD_V2_MIN_THREAD_POOL_SIZE

      public static final String FS_AZURE_READAHEAD_V2_MIN_THREAD_POOL_SIZE
      Minimum number of prefetch threads in the thread pool for readahead V2. "fs.azure.readahead.v2.min.thread.pool.size"
      See Also:

    • FS_AZURE_READAHEAD_V2_MAX_THREAD_POOL_SIZE

      public static final String FS_AZURE_READAHEAD_V2_MAX_THREAD_POOL_SIZE
      Maximum number of prefetch threads in the thread pool for readahead V2. "fs.azure.readahead.v2.max.thread.pool.size"
      See Also:

    • FS_AZURE_READAHEAD_V2_MIN_BUFFER_POOL_SIZE

      public static final String FS_AZURE_READAHEAD_V2_MIN_BUFFER_POOL_SIZE
      Minimum size of the buffer pool for caching prefetched data for readahead V2. "fs.azure.readahead.v2.min.buffer.pool.size"
      See Also:

    • FS_AZURE_READAHEAD_V2_MAX_BUFFER_POOL_SIZE

      public static final String FS_AZURE_READAHEAD_V2_MAX_BUFFER_POOL_SIZE
      Maximum size of the buffer pool for caching prefetched data for readahead V2. "fs.azure.readahead.v2.max.buffer.pool.size"
      See Also:

    • FS_AZURE_READAHEAD_V2_CPU_MONITORING_INTERVAL_MILLIS

      public static final String FS_AZURE_READAHEAD_V2_CPU_MONITORING_INTERVAL_MILLIS
      Interval in milliseconds for periodic monitoring of CPU usage and up/down scaling thread pool size accordingly. "fs.azure.readahead.v2.cpu.monitoring.interval.millis"
      See Also:

    • FS_AZURE_READAHEAD_V2_THREAD_POOL_UPSCALE_PERCENTAGE

      public static final String FS_AZURE_READAHEAD_V2_THREAD_POOL_UPSCALE_PERCENTAGE
      Percentage by which the thread pool size should be upscaled when CPU usage is low.
      See Also:

    • FS_AZURE_READAHEAD_V2_THREAD_POOL_DOWNSCALE_PERCENTAGE

      public static final String FS_AZURE_READAHEAD_V2_THREAD_POOL_DOWNSCALE_PERCENTAGE
      Percentage by which the thread pool size should be downscaled when CPU usage is high.
      See Also:

    • FS_AZURE_READAHEAD_V2_MEMORY_MONITORING_INTERVAL_MILLIS

      public static final String FS_AZURE_READAHEAD_V2_MEMORY_MONITORING_INTERVAL_MILLIS
      Interval in milliseconds for periodic monitoring of memory usage and up/down scaling buffer pool size accordingly. "fs.azure.readahead.v2.memory.monitoring.interval.millis"
      See Also:

    • FS_AZURE_READAHEAD_V2_EXECUTOR_SERVICE_TTL_MILLIS

      public static final String FS_AZURE_READAHEAD_V2_EXECUTOR_SERVICE_TTL_MILLIS
      TTL in milliseconds for the idle threads in executor service used by read ahead v2.
      See Also:

    • FS_AZURE_READAHEAD_V2_CACHED_BUFFER_TTL_MILLIS

      public static final String FS_AZURE_READAHEAD_V2_CACHED_BUFFER_TTL_MILLIS
      TTL in milliseconds for the cached buffers in buffer pool used by read ahead v2.
      See Also:

    • FS_AZURE_READAHEAD_V2_CPU_USAGE_THRESHOLD_PERCENT

      public static final String FS_AZURE_READAHEAD_V2_CPU_USAGE_THRESHOLD_PERCENT
      Threshold percentage for CPU usage to scale up/down the thread pool size in read ahead v2.
      See Also:

    • FS_AZURE_READAHEAD_V2_MEMORY_USAGE_THRESHOLD_PERCENT

      public static final String FS_AZURE_READAHEAD_V2_MEMORY_USAGE_THRESHOLD_PERCENT
      Threshold percentage for memory usage to scale up/down the buffer pool size in read ahead v2.
      See Also:

    • FS_AZURE_ENABLE_ABFS_LIST_ITERATOR

      public static final String FS_AZURE_ENABLE_ABFS_LIST_ITERATOR
      Setting this true will make the driver use it's own RemoteIterator implementation
      See Also:

    • FS_AZURE_ENCRYPTION_ENCODED_CLIENT_PROVIDED_KEY

      public static final String FS_AZURE_ENCRYPTION_ENCODED_CLIENT_PROVIDED_KEY
      Server side encryption key encoded in Base6format "fs.azure.encryption.encoded.client-provided-key".
      See Also:

    • FS_AZURE_ENCRYPTION_ENCODED_CLIENT_PROVIDED_KEY_SHA

      public static final String FS_AZURE_ENCRYPTION_ENCODED_CLIENT_PROVIDED_KEY_SHA
      SHA256 hash of encryption key encoded in Base64format
      See Also:

    • FS_AZURE_ENCRYPTION_CONTEXT_PROVIDER_TYPE

      public static final String FS_AZURE_ENCRYPTION_CONTEXT_PROVIDER_TYPE
      Custom EncryptionContextProvider type
      See Also:

    • AZURE_ABFS_ENDPOINT

      public static final String AZURE_ABFS_ENDPOINT
      End point of ABFS account: "fs.azure.abfs.endpoint".
      See Also:

    • FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME

      public static final String FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME
      Key for auth type properties: "fs.azure.account.auth.type".
      See Also:

    • FS_AZURE_ACCOUNT_TOKEN_PROVIDER_TYPE_PROPERTY_NAME

      public static final String FS_AZURE_ACCOUNT_TOKEN_PROVIDER_TYPE_PROPERTY_NAME
      Key for oauth token provider type: "fs.azure.account.oauth.provider.type".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID

      public static final String FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID
      Key for oauth AAD client id: "fs.azure.account.oauth2.client.id".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_CLIENT_SECRET

      public static final String FS_AZURE_ACCOUNT_OAUTH_CLIENT_SECRET
      Key for oauth AAD client secret: "fs.azure.account.oauth2.client.secret".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_CLIENT_ENDPOINT

      public static final String FS_AZURE_ACCOUNT_OAUTH_CLIENT_ENDPOINT
      Key for oauth AAD client endpoint: "fs.azure.account.oauth2.client.endpoint".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT

      public static final String FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT
      Key for oauth msi tenant id: "fs.azure.account.oauth2.msi.tenant".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT

      public static final String FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT
      Key for oauth msi endpoint: "fs.azure.account.oauth2.msi.endpoint".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_MSI_AUTHORITY

      public static final String FS_AZURE_ACCOUNT_OAUTH_MSI_AUTHORITY
      Key for oauth msi Authority: "fs.azure.account.oauth2.msi.authority".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_USER_NAME

      public static final String FS_AZURE_ACCOUNT_OAUTH_USER_NAME
      Key for oauth user name: "fs.azure.account.oauth2.user.name".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_USER_PASSWORD

      public static final String FS_AZURE_ACCOUNT_OAUTH_USER_PASSWORD
      Key for oauth user password: "fs.azure.account.oauth2.user.password".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN

      public static final String FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN
      Key for oauth refresh token: "fs.azure.account.oauth2.refresh.token".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN_ENDPOINT

      public static final String FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN_ENDPOINT
      Key for oauth AAD refresh token endpoint: "fs.azure.account.oauth2.refresh.token.endpoint".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_TOKEN_FILE

      public static final String FS_AZURE_ACCOUNT_OAUTH_TOKEN_FILE
      Key for oauth AAD workload identity token file path: "fs.azure.account.oauth2.token.file".
      See Also:

    • FS_AZURE_ACCOUNT_OAUTH_CLIENT_ASSERTION_PROVIDER_TYPE

      public static final String FS_AZURE_ACCOUNT_OAUTH_CLIENT_ASSERTION_PROVIDER_TYPE
      Key for custom client assertion provider class for WorkloadIdentityTokenProvider
      See Also:

    • FS_AZURE_ABFS_LATENCY_TRACK

      public static final String FS_AZURE_ABFS_LATENCY_TRACK
      Key for enabling the tracking of ABFS API latency and sending the latency numbers to the ABFS API service
      See Also:

    • FS_AZURE_ABFS_IO_RATE_LIMIT

      public static final String FS_AZURE_ABFS_IO_RATE_LIMIT
      Key for rate limit capacity, as used by IO operations which try to throttle themselves.
      See Also:

    • FS_AZURE_ABFS_RENAME_RESILIENCE

      public static final String FS_AZURE_ABFS_RENAME_RESILIENCE
      Add extra resilience to rename failures, at the expense of performance.
      See Also:

    • FS_AZURE_ENABLE_PAGINATED_DELETE

      public static final String FS_AZURE_ENABLE_PAGINATED_DELETE
      Specify whether paginated behavior is to be expected or not in delete path. "fs.azure.enable.paginated.delete"
      See Also:

    • FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION

      public static final String FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION
      Add extra layer of verification of the integrity of the request content during transport: "fs.azure.enable.checksum.validation".
      See Also:

    • FS_AZURE_ENABLE_FULL_BLOB_CHECKSUM_VALIDATION

      public static final String FS_AZURE_ENABLE_FULL_BLOB_CHECKSUM_VALIDATION
      Add extra layer of verification of the integrity of the full blob request content during transport: "fs.azure.enable.full.blob.checksum.validation".
      See Also:

    • FS_AZURE_ENABLE_DELEGATION_TOKEN

      public static final String FS_AZURE_ENABLE_DELEGATION_TOKEN
      See Also:

    • FS_AZURE_DELEGATION_TOKEN_PROVIDER_TYPE

      public static final String FS_AZURE_DELEGATION_TOKEN_PROVIDER_TYPE
      See Also:

    • FS_AZURE_SAS_FIXED_TOKEN

      public static final String FS_AZURE_SAS_FIXED_TOKEN
      Key for fixed SAS token: "fs.azure.sas.fixed.token".
      See Also:

    • FS_AZURE_SAS_TOKEN_PROVIDER_TYPE

      public static final String FS_AZURE_SAS_TOKEN_PROVIDER_TYPE
      Key for SAS token provider: "fs.azure.sas.token.provider.type".
      See Also:

    • FS_AZURE_SAS_TOKEN_RENEW_PERIOD_FOR_STREAMS

      public static final String FS_AZURE_SAS_TOKEN_RENEW_PERIOD_FOR_STREAMS
      For performance, AbfsInputStream/AbfsOutputStream re-use SAS tokens until the expiry is within this number of seconds.
      See Also:

    • FS_AZURE_IDENTITY_TRANSFORM_CLASS

      public static final String FS_AZURE_IDENTITY_TRANSFORM_CLASS
      Key to enable custom identity transformation.
      See Also:

    • FS_AZURE_LOCAL_USER_SP_MAPPING_FILE_PATH

      public static final String FS_AZURE_LOCAL_USER_SP_MAPPING_FILE_PATH
      Key for Local User to Service Principal file location.
      See Also:

    • FS_AZURE_LOCAL_GROUP_SG_MAPPING_FILE_PATH

      public static final String FS_AZURE_LOCAL_GROUP_SG_MAPPING_FILE_PATH
      Key for Local Group to Service Group file location.
      See Also:

    • FS_AZURE_BUFFERED_PREAD_DISABLE

      public static final String FS_AZURE_BUFFERED_PREAD_DISABLE
      Optional config to enable a lock free pread which will bypass buffer in AbfsInputStream. This is not a config which can be set at cluster level. It can be used as an option on FutureDataInputStreamBuilder.
      See Also:

    • FS_AZURE_NETWORKING_LIBRARY

      public static final String FS_AZURE_NETWORKING_LIBRARY
      Defines what network library to use for server IO calls: "fs.azure.networking.library"
      See Also:

    • FS_AZURE_APACHE_HTTP_CLIENT_MAX_IO_EXCEPTION_RETRIES

      public static final String FS_AZURE_APACHE_HTTP_CLIENT_MAX_IO_EXCEPTION_RETRIES
      Maximum number of IOExceptions retries for a single server call on ApacheHttpClient. Breach of this count would turn off future uses of the ApacheHttpClient library in the JVM lifecycle: "fs.azure.apache.http.client.max.io.exception.retries"
      See Also:

    • FS_AZURE_APACHE_HTTP_CLIENT_MAX_CACHE_SIZE

      public static final String FS_AZURE_APACHE_HTTP_CLIENT_MAX_CACHE_SIZE
      Maximum ApacheHttpClient-connection cache size at filesystem level: "fs.azure.apache.http.client.max.cache.size"
      See Also:

    • FS_AZURE_APACHE_HTTP_CLIENT_CACHE_WARMUP_COUNT

      public static final String FS_AZURE_APACHE_HTTP_CLIENT_CACHE_WARMUP_COUNT
      Defines number of connections to establish during warmup phase of ApacheHttpClient connection cache: "fs.azure.apache.http.client.cache.warmup.count"
      See Also:

    • FS_AZURE_APACHE_HTTP_CLIENT_CACHE_REFRESH_COUNT

      public static final String FS_AZURE_APACHE_HTTP_CLIENT_CACHE_REFRESH_COUNT
      Defines number of connections to establish during refresh phase of ApacheHttpClient connection cache: "fs.azure.apache.http.client.cache.refresh.count"
      See Also:

    • FS_AZURE_APACHE_HTTP_CLIENT_MAX_REFRESH_WAIT_TIME_MILLIS

      public static final String FS_AZURE_APACHE_HTTP_CLIENT_MAX_REFRESH_WAIT_TIME_MILLIS
      Defines time duration to wait for ApacheHttpClient connection cache to warmup/ refresh: "fs.azure.apache.http.client.max.refresh.wait.time.millis"
      See Also:

    • FS_AZURE_APACHE_HTTP_CLIENT_MIN_TRIGGER_REFRESH_COUNT

      public static final String FS_AZURE_APACHE_HTTP_CLIENT_MIN_TRIGGER_REFRESH_COUNT
      Minimum number of cached connections in ApacheHttpClient cache below which refresh will be triggered. "fs.azure.apache.http.client.min.trigger.refresh.count"
      See Also:

    • FS_AZURE_APACHE_HTTP_CLIENT_WARMUP_CACHE_TIMEOUT_MILLIS

      public static final String FS_AZURE_APACHE_HTTP_CLIENT_WARMUP_CACHE_TIMEOUT_MILLIS
      Time duration to wait for ApacheHttpClient connection cache to warmup/refresh: "fs.azure.apache.http.client.warmup.cache.timeout.millis"
      See Also:

    • FS_AZURE_BLOB_COPY_PROGRESS_WAIT_MILLIS

      public static final String FS_AZURE_BLOB_COPY_PROGRESS_WAIT_MILLIS
      Blob copy API is an async API, this configuration defines polling duration for checking copy status: "fs.azure.blob.copy.progress.wait.millis"
      See Also:

    • FS_AZURE_BLOB_COPY_MAX_WAIT_MILLIS

      public static final String FS_AZURE_BLOB_COPY_MAX_WAIT_MILLIS
      Maximum time to wait for a blob copy operation to complete: "fs.azure.blob.copy.max.wait.millis"
      See Also:

    • FS_AZURE_BLOB_ATOMIC_RENAME_LEASE_REFRESH_DURATION

      public static final String FS_AZURE_BLOB_ATOMIC_RENAME_LEASE_REFRESH_DURATION
      Blob rename lease refresh duration: "fs.azure.blob.atomic.rename.lease.refresh.duration"
      See Also:

    • FS_AZURE_PRODUCER_QUEUE_MAX_SIZE

      public static final String FS_AZURE_PRODUCER_QUEUE_MAX_SIZE
      Maximum number of blob information enqueued in memory for rename or delete orchestration: "fs.azure.blob.dir.list.producer.queue.max.size"
      See Also:

    • FS_AZURE_CONSUMER_MAX_LAG

      public static final String FS_AZURE_CONSUMER_MAX_LAG
      Maximum consumer lag (count of blob information which is yet to be taken for operation) in blob listing which can be tolerated before making producer to wait for consumer lag to become tolerable: "fs.azure.blob.dir.list.consumer.max.lag".
      See Also:

    • FS_AZURE_BLOB_DIR_RENAME_MAX_THREAD

      public static final String FS_AZURE_BLOB_DIR_RENAME_MAX_THREAD
      Maximum number of thread per blob-rename orchestration: "fs.azure.blob.dir.rename.max.thread"
      See Also:

    • FS_AZURE_BLOB_DIR_DELETE_MAX_THREAD

      public static final String FS_AZURE_BLOB_DIR_DELETE_MAX_THREAD
      Maximum number of thread per blob-delete orchestration: "fs.azure.blob.dir.delete.max.thread"
      See Also:

    • FS_AZURE_ENABLE_CLIENT_TRANSACTION_ID

      public static final String FS_AZURE_ENABLE_CLIENT_TRANSACTION_ID
      Flag to enable/disable sending client transactional ID during create/rename operations: "fs.azure.enable.client.transaction.id"
      See Also:

    • FS_AZURE_ENABLE_CREATE_BLOB_IDEMPOTENCY

      public static final String FS_AZURE_ENABLE_CREATE_BLOB_IDEMPOTENCY
      Flag to enable/disable create idempotency during create operation: "fs.azure.enable.create.blob.idempotency"
      See Also:

    • FS_AZURE_ENABLE_TAIL_LATENCY_TRACKER

      public static final String FS_AZURE_ENABLE_TAIL_LATENCY_TRACKER
      Flag to enable/disable tail latency tracker for AbfsRestOperation. When enabled, Client observed E2E latency will be tracked by a histogram. Regular p50, p99 and configured percentile latencies will be reported.
      See Also:

    • FS_AZURE_ENABLE_TAIL_LATENCY_REQUEST_TIMEOUT

      public static final String FS_AZURE_ENABLE_TAIL_LATENCY_REQUEST_TIMEOUT
      Flag to enable/disable tail latency based timeout for AbfsRestOperation. When enabled, if an operation's latency exceeds the currently reported tail latency by the tracker, the ongoing socket connection will be closed and the operation will be retried, up to the configured max retry count: "fs.azure.enable.tail.latency.timeout"
      See Also:

    • FS_AZURE_TAIL_LATENCY_PERCENTILE

      public static final String FS_AZURE_TAIL_LATENCY_PERCENTILE
      The percentile value to be considered as tail latency value. Default is 99.0 (99th percentile): "fs.azure.tail.latency.percentile"
      See Also:

    • FS_AZURE_TAIL_LATENCY_MIN_DEVIATION

      public static final String FS_AZURE_TAIL_LATENCY_MIN_DEVIATION
      The minimum deviation (in percentage) between p50 and tail latency percentile to trigger tail latency based request timeout: "fs.azure.tail.latency.min.deviation"
      See Also:

    • FS_AZURE_TAIL_LATENCY_MIN_SAMPLE_SIZE

      public static final String FS_AZURE_TAIL_LATENCY_MIN_SAMPLE_SIZE
      The minimum sample size required before the histogram starts reporting latency data: "fs.azure.tail.latency.min.sample.size"
      See Also:

    • FS_AZURE_TAIL_LATENCY_ANALYSIS_WINDOW_MILLIS

      public static final String FS_AZURE_TAIL_LATENCY_ANALYSIS_WINDOW_MILLIS
      The time window (in milliseconds) over which the tail latency analysis is performed. Until the whole window is filled, the histogram will not report any latency data: "fs.azure.tail.latency.analysis.window.millis"
      See Also:

    • FS_AZURE_TAIL_LATENCY_ANALYSIS_WINDOW_GRANULARITY

      public static final String FS_AZURE_TAIL_LATENCY_ANALYSIS_WINDOW_GRANULARITY
      The granularity (in milliseconds) at which the tail latency analysis window is divided. This is to make sliding window calculations efficient and robust: "fs.azure.tail.latency.analysis.window.granularity"
      See Also:

    • FS_AZURE_TAIL_LATENCY_PERCENTILE_COMPUTATION_INTERVAL_MILLIS

      public static final String FS_AZURE_TAIL_LATENCY_PERCENTILE_COMPUTATION_INTERVAL_MILLIS
      Interval (in milliseconds) at which the tail latency percentile is computed and updated by the background thread for each operation type: "fs.azure.tail.latency.percentile.computation.interval.millis"
      See Also:

    • FS_AZURE_TAIL_LATENCY_MAX_RETRY_COUNT

      public static final String FS_AZURE_TAIL_LATENCY_MAX_RETRY_COUNT
      Maximum number of retries for an operation when tail latency based timeout occur: "fs.azure.tail.latency.max.retry.count"
      See Also:

  • Method Details