Package org.apache.hadoop.fs.s3a

Class Constants

java.lang.Object
org.apache.hadoop.fs.s3a.Constants
@Public @Evolving public final class Constants extends Object
Constants used with the S3AFileSystem. Some of the strings are marked as Unstable. This means that they may be Unsupported in future; at which point they will be marked as deprecated and simply ignored. All S3Guard related constants are marked as Deprecated and either ignored (ddb config) or rejected (setting the metastore to anything other than the null store).

Timeout default values are declared as integers or long values in milliseconds and occasionally seconds. There are now Duration constants for these default values; the original fields are retained for compatibility, and derive their value from the Duration equivalent.

New timeout/duration constants do not get the equivalent integer/long fields.

  • Field Details

    • HADOOP_TMP_DIR

      public static final String HADOOP_TMP_DIR
      default hadoop temp dir on local system: "hadoop.tmp.dir".
      See Also:

    • MULTIPART_MIN_SIZE

      public static final int MULTIPART_MIN_SIZE
      The minimum multipart size which S3 supports.
      See Also:

    • ACCESS_KEY

      public static final String ACCESS_KEY
      See Also:

    • SECRET_KEY

      public static final String SECRET_KEY
      See Also:

    • AWS_CREDENTIALS_PROVIDER

      public static final String AWS_CREDENTIALS_PROVIDER
      See Also:

    • AWS_CREDENTIALS_PROVIDER_MAPPING

      public static final String AWS_CREDENTIALS_PROVIDER_MAPPING
      AWS credentials providers mapping with key/value pairs. Value = "fs.s3a.aws.credentials.provider.mapping"
      See Also:

    • S3A_SECURITY_CREDENTIAL_PROVIDER_PATH

      public static final String S3A_SECURITY_CREDENTIAL_PROVIDER_PATH
      Extra set of security credentials which will be prepended to that set in "hadoop.security.credential.provider.path". This extra option allows for per-bucket overrides.
      See Also:

    • SESSION_TOKEN

      public static final String SESSION_TOKEN
      session token for when using TemporaryAWSCredentialsProvider: : "fs.s3a.session.token".
      See Also:

    • ASSUMED_ROLE_ARN

      public static final String ASSUMED_ROLE_ARN
      ARN of AWS Role to request: "fs.s3a.assumed.role.arn".
      See Also:

    • ASSUMED_ROLE_EXTERNAL_ID

      public static final String ASSUMED_ROLE_EXTERNAL_ID
      external id for assume role request: "fs.s3a.assumed.role.external.id".
      See Also:

    • ASSUMED_ROLE_SESSION_NAME

      public static final String ASSUMED_ROLE_SESSION_NAME
      Session name for the assumed role, must be valid characters according to the AWS APIs: "fs.s3a.assumed.role.session.name". If not set, one is generated from the current Hadoop/Kerberos username.
      See Also:

    • ASSUMED_ROLE_SESSION_DURATION

      public static final String ASSUMED_ROLE_SESSION_DURATION
      Duration of assumed roles before a refresh is attempted.
      See Also:

    • ASSUMED_ROLE_STS_ENDPOINT

      public static final String ASSUMED_ROLE_STS_ENDPOINT
      Security Token Service Endpoint: "fs.s3a.assumed.role.sts.endpoint". If unset, uses the default endpoint.
      See Also:

    • DEFAULT_ASSUMED_ROLE_STS_ENDPOINT

      public static final String DEFAULT_ASSUMED_ROLE_STS_ENDPOINT
      Default endpoint for session tokens: "". This is the central STS endpoint which, for v3 signing, can issue STS tokens for any region.
      See Also:

    • ASSUMED_ROLE_STS_ENDPOINT_REGION

      public static final String ASSUMED_ROLE_STS_ENDPOINT_REGION
      Region for the STS endpoint; needed if the endpoint is set to anything other then the central one.: "fs.s3a.assumed.role.sts.endpoint.region".
      See Also:

    • ASSUMED_ROLE_STS_ENDPOINT_REGION_DEFAULT

      public static final String ASSUMED_ROLE_STS_ENDPOINT_REGION_DEFAULT
      Default value for the STS endpoint region; needed for v4 signing: "".
      See Also:

    • ASSUMED_ROLE_SESSION_DURATION_DEFAULT

      public static final String ASSUMED_ROLE_SESSION_DURATION_DEFAULT
      Default duration of an assumed role: "1h".
      See Also:

    • ASSUMED_ROLE_CREDENTIALS_PROVIDER

      public static final String ASSUMED_ROLE_CREDENTIALS_PROVIDER
      List of providers to authenticate for the assumed role: "fs.s3a.assumed.role.credentials.provider".
      See Also:

    • ASSUMED_ROLE_POLICY

      public static final String ASSUMED_ROLE_POLICY
      JSON policy containing the policy to apply to the role: "fs.s3a.assumed.role.policy". This is not used for delegation tokens, which generate the policy automatically, and restrict it to the S3 and KMS services needed.
      See Also:

    • ASSUMED_ROLE_CREDENTIALS_DEFAULT

      public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT
      See Also:

    • MAX_TOTAL_TASKS

      public static final String MAX_TOTAL_TASKS
      The maximum number of tasks queued (other than prefetcher tasks) if all threads are busy: "fs.s3a.max.total.tasks".
      See Also:

    • DEFAULT_MAX_TOTAL_TASKS

      public static final int DEFAULT_MAX_TOTAL_TASKS
      Default value for MAX_TOTAL_TASKS: 32.
      See Also:

    • MAXIMUM_CONNECTIONS

      public static final String MAXIMUM_CONNECTIONS
      Number of simultaneous connections to S3: "fs.s3a.connection.maximum".
      See Also:

    • DEFAULT_MAXIMUM_CONNECTIONS

      public static final int DEFAULT_MAXIMUM_CONNECTIONS
      Default value for MAXIMUM_CONNECTIONS: 500. Future releases are likely to increase this value. Keep in sync with the value in core-default.xml
      See Also:

    • CONNECTION_TTL

      public static final String CONNECTION_TTL
      Configuration option to configure expiration time of S3 http connection from the connection pool: "fs.s3a.connection.ttl".
      See Also:

    • DEFAULT_CONNECTION_TTL_DURATION

      public static final Duration DEFAULT_CONNECTION_TTL_DURATION
      Default duration for CONNECTION_TTL: 5 minutes.

    • DEFAULT_CONNECTION_TTL

      public static final long DEFAULT_CONNECTION_TTL
      Deprecated.
      use DEFAULT_CONNECTION_TTL_DURATION
      Default value in millis for CONNECTION_TTL: 5 minutes.

    • SECURE_CONNECTIONS

      public static final String SECURE_CONNECTIONS
      See Also:

    • DEFAULT_SECURE_CONNECTIONS

      public static final boolean DEFAULT_SECURE_CONNECTIONS
      See Also:

    • ALLOW_REQUESTER_PAYS

      public static final String ALLOW_REQUESTER_PAYS
      Configuration option for S3 Requester Pays feature: "fs.s3a.requester.pays.enabled".
      See Also:

    • DEFAULT_ALLOW_REQUESTER_PAYS

      public static final boolean DEFAULT_ALLOW_REQUESTER_PAYS
      Default configuration for "fs.s3a.requester.pays.enabled": false.
      See Also:

    • SSL_CHANNEL_MODE

      public static final String SSL_CHANNEL_MODE
      See Also:

    • DEFAULT_SSL_CHANNEL_MODE

      public static final DelegatingSSLSocketFactory.SSLChannelMode DEFAULT_SSL_CHANNEL_MODE

    • ENDPOINT

      public static final String ENDPOINT
      Endpoint. For v4 signing and/or better performance, this should be the specific endpoint of the region in which the bucket is hosted.
      See Also:

    • DEFAULT_ENDPOINT

      public static final String DEFAULT_ENDPOINT
      Default value of s3 endpoint: "". It tells the AWS client to work it out by asking the central endpoint where the bucket lives; caching that value in the client for the life of the process.

      Note: previously this constant was defined as CENTRAL_ENDPOINT, however the actual S3A client code used "" as the default when ENDPOINT was unset. As core-default.xml also set the endpoint to "", the empty string has long been the real default value.

      See Also:

    • CENTRAL_ENDPOINT

      public static final String CENTRAL_ENDPOINT
      The central endpoint :"s3.amazonaws.com".
      See Also:

    • PATH_STYLE_ACCESS

      public static final String PATH_STYLE_ACCESS
      See Also:

    • PROXY_HOST

      public static final String PROXY_HOST
      See Also:

    • PROXY_PORT

      public static final String PROXY_PORT
      See Also:

    • PROXY_USERNAME

      public static final String PROXY_USERNAME
      See Also:

    • PROXY_PASSWORD

      public static final String PROXY_PASSWORD
      See Also:

    • PROXY_DOMAIN

      public static final String PROXY_DOMAIN
      See Also:

    • PROXY_WORKSTATION

      public static final String PROXY_WORKSTATION
      See Also:

    • PROXY_SECURED

      public static final String PROXY_SECURED
      Is the proxy secured(proxyProtocol = HTTPS)?
      See Also:

    • MAX_ERROR_RETRIES

      public static final String MAX_ERROR_RETRIES
      Number of times the AWS client library should retry errors before escalating to the S3A code: "fs.s3a.attempts.maximum". The S3A connector does its own selective retries; the only time the AWS SDK operations are not wrapped is during multipart copy via the AWS SDK transfer manager.
      See Also:

    • DEFAULT_MAX_ERROR_RETRIES

      public static final int DEFAULT_MAX_ERROR_RETRIES
      Default number of times the AWS client library should retry errors before escalating to the S3A code: 5.
      See Also:

    • EXPERIMENTAL_AWS_INTERNAL_THROTTLING

      @Unstable public static final String EXPERIMENTAL_AWS_INTERNAL_THROTTLING
      Experimental/Unstable feature: should the AWS client library retry throttle responses before escalating to the S3A code: "fs.s3a.experimental.aws.s3.throttling". When set to false, the S3A connector sees all S3 throttle events, And so can update it counters and the metrics, and use its own retry policy. However, this may have adverse effects on some operations where the S3A code cannot retry as efficiently as the AWS client library. This only applies to S3 operations, not to DynamoDB or other services.
      See Also:

    • EXPERIMENTAL_AWS_INTERNAL_THROTTLING_DEFAULT

      @Unstable public static final boolean EXPERIMENTAL_AWS_INTERNAL_THROTTLING_DEFAULT
      Default value of EXPERIMENTAL_AWS_INTERNAL_THROTTLING, value: true.
      See Also:

    • MINIMUM_NETWORK_OPERATION_DURATION

      public static final Duration MINIMUM_NETWORK_OPERATION_DURATION
      This is the minimum operation duration unless programmatically set. It ensures that even if a configuration has mistaken a millisecond option for seconds, a viable duration will actually be used. Value: 15s.

    • ESTABLISH_TIMEOUT

      public static final String ESTABLISH_TIMEOUT
      Milliseconds until a connection is established: "fs.s3a.connection.establish.timeout".
      See Also:

    • DEFAULT_ESTABLISH_TIMEOUT_DURATION

      public static final Duration DEFAULT_ESTABLISH_TIMEOUT_DURATION
      Default TCP/(and TLS?) establish timeout: 30 seconds.

    • DEFAULT_ESTABLISH_TIMEOUT

      public static final int DEFAULT_ESTABLISH_TIMEOUT
      Deprecated.
      use DEFAULT_ESTABLISH_TIMEOUT_DURATION
      Default establish timeout in millis: 30 seconds.

    • SOCKET_TIMEOUT

      public static final String SOCKET_TIMEOUT
      Milliseconds until we give up on a connection to s3: "fs.s3a.connection.timeout".
      See Also:

    • DEFAULT_SOCKET_TIMEOUT_DURATION

      public static final Duration DEFAULT_SOCKET_TIMEOUT_DURATION
      Default socket timeout: 200 seconds.

    • DEFAULT_SOCKET_TIMEOUT

      public static final int DEFAULT_SOCKET_TIMEOUT
      Deprecated.
      use DEFAULT_SOCKET_TIMEOUT_DURATION
      Default socket timeout: DEFAULT_SOCKET_TIMEOUT_DURATION.

    • REQUEST_TIMEOUT

      public static final String REQUEST_TIMEOUT
      How long should the SDK retry/wait on a response from an S3 store: "fs.s3a.connection.request.timeout" including the time needed to sign the request.

      This is time to response, so for a GET request it is "time to 200 response" not the time limit to download the requested data. This makes it different from REQUEST_TIMEOUT, which is for total HTTP request.

      Default unit is milliseconds.

      There is a minimum duration set in MINIMUM_NETWORK_OPERATION_DURATION; it is impossible to set a delay less than this, even for testing. Why so? Too many deployments where the configuration assumed the timeout was in seconds and that "120" was a reasonable value rather than "too short to work reliably"

      Note for anyone writing tests which need to set a low value for this: to avoid the minimum duration overrides, call AWSClientConfig.setMinimumOperationDuration() and set a low value before creating the filesystem.

      See Also:

    • DEFAULT_REQUEST_TIMEOUT_DURATION

      public static final Duration DEFAULT_REQUEST_TIMEOUT_DURATION
      Default duration of a request before it is timed out: 60s.

    • DEFAULT_REQUEST_TIMEOUT

      public static final int DEFAULT_REQUEST_TIMEOUT
      Deprecated.
      use DEFAULT_REQUEST_TIMEOUT_DURATION
      Default duration of a request before it is timed out: Zero.

    • CONNECTION_ACQUISITION_TIMEOUT

      public static final String CONNECTION_ACQUISITION_TIMEOUT
      Acquisition timeout for connections from the pool: "fs.s3a.connection.acquisition.timeout". Default unit is milliseconds for consistency with other options.
      See Also:

    • DEFAULT_CONNECTION_ACQUISITION_TIMEOUT_DURATION

      public static final Duration DEFAULT_CONNECTION_ACQUISITION_TIMEOUT_DURATION
      Default acquisition timeout: 60 seconds.

    • PART_UPLOAD_TIMEOUT

      public static final String PART_UPLOAD_TIMEOUT
      Timeout for uploading all of a small object or a single part of a larger one. "fs.s3a.connection.part.upload.timeout". Default unit is milliseconds for consistency with other options.
      See Also:

    • DEFAULT_PART_UPLOAD_TIMEOUT

      public static final Duration DEFAULT_PART_UPLOAD_TIMEOUT
      Default part upload timeout: 15 minutes.

    • CONNECTION_KEEPALIVE

      public static final String CONNECTION_KEEPALIVE
      Should TCP Keepalive be enabled on the socket? This adds some network IO, but finds failures faster. "fs.s3a.connection.keepalive".
      See Also:

    • DEFAULT_CONNECTION_KEEPALIVE

      public static final boolean DEFAULT_CONNECTION_KEEPALIVE
      Default value of CONNECTION_KEEPALIVE: false.
      See Also:

    • CONNECTION_IDLE_TIME

      public static final String CONNECTION_IDLE_TIME
      Maximum idle time for connections in the pool: "fs.s3a.connection.idle.time".

      Too low: overhead of creating connections. Too high, risk of stale connections and inability to use the adaptive load balancing of the S3 front end.

      Default unit is milliseconds for consistency with other options.

      See Also:

    • DEFAULT_CONNECTION_IDLE_TIME_DURATION

      public static final Duration DEFAULT_CONNECTION_IDLE_TIME_DURATION
      Default idle time: 60 seconds.

    • CONNECTION_EXPECT_CONTINUE

      public static final String CONNECTION_EXPECT_CONTINUE
      Should PUT requests await a 100 CONTINUE responses before uploading data?

      Value: "fs.s3a.connection.expect.continue".

      See Also:

    • CONNECTION_EXPECT_CONTINUE_DEFAULT

      public static final boolean CONNECTION_EXPECT_CONTINUE_DEFAULT
      Default value for CONNECTION_EXPECT_CONTINUE.
      See Also:

    • SOCKET_SEND_BUFFER

      public static final String SOCKET_SEND_BUFFER
      See Also:

    • DEFAULT_SOCKET_SEND_BUFFER

      public static final int DEFAULT_SOCKET_SEND_BUFFER
      See Also:

    • SOCKET_RECV_BUFFER

      public static final String SOCKET_RECV_BUFFER
      See Also:

    • DEFAULT_SOCKET_RECV_BUFFER

      public static final int DEFAULT_SOCKET_RECV_BUFFER
      See Also:

    • MAX_PAGING_KEYS

      public static final String MAX_PAGING_KEYS
      See Also:

    • DEFAULT_MAX_PAGING_KEYS

      public static final int DEFAULT_MAX_PAGING_KEYS
      See Also:

    • MAX_THREADS

      public static final String MAX_THREADS
      The maximum number of threads to allow in the pool used by S3A. Value: "fs.s3a.threads.max".
      See Also:

    • DEFAULT_MAX_THREADS

      public static final int DEFAULT_MAX_THREADS
      Default value of MAX_THREADS: 96.
      See Also:

    • KEEPALIVE_TIME

      public static final String KEEPALIVE_TIME
      The time an idle thread waits before terminating: "fs.s3a.threads.keepalivetime". This is in SECONDS unless the optional unit is given.
      See Also:

    • DEFAULT_KEEPALIVE_TIME_DURATION

      public static final Duration DEFAULT_KEEPALIVE_TIME_DURATION
      Default value of KEEPALIVE_TIME: 60s.

    • DEFAULT_KEEPALIVE_TIME

      public static final int DEFAULT_KEEPALIVE_TIME
      Deprecated.
      use DEFAULT_KEEPALIVE_TIME_DURATION
      Default value of KEEPALIVE_TIME: 60s.

    • MULTIPART_SIZE

      public static final String MULTIPART_SIZE
      See Also:

    • DEFAULT_MULTIPART_SIZE

      public static final long DEFAULT_MULTIPART_SIZE
      See Also:

    • MIN_MULTIPART_THRESHOLD

      public static final String MIN_MULTIPART_THRESHOLD
      See Also:

    • DEFAULT_MIN_MULTIPART_THRESHOLD

      public static final long DEFAULT_MIN_MULTIPART_THRESHOLD
      See Also:

    • ENABLE_MULTI_DELETE

      public static final String ENABLE_MULTI_DELETE
      See Also:

    • BULK_DELETE_PAGE_SIZE

      public static final String BULK_DELETE_PAGE_SIZE
      Number of objects to delete in a single multi-object delete "fs.s3a.bulk.delete.page.size". Max: 1000. A bigger value it means fewer POST requests when deleting a directory tree with many objects. However, as you are limited to only a a few thousand requests per second against a single partition of an S3 bucket, a large page size can easily overload the bucket and so trigger throttling. Furthermore, as the reaction to this request is being throttled is simply to retry it -it can take a while for the situation to go away. While a large value may give better numbers on tests and benchmarks where only a single operations being executed, once multiple applications start working with the same bucket these large deletes can be highly disruptive.
      See Also:

    • BULK_DELETE_PAGE_SIZE_DEFAULT

      public static final int BULK_DELETE_PAGE_SIZE_DEFAULT
      Default Number of objects to delete in a single multi-object delete: 250.
      See Also:

    • BUFFER_DIR

      public static final String BUFFER_DIR
      See Also:

    • FAST_UPLOAD

      @Deprecated public static final String FAST_UPLOAD
      Deprecated.
      See Also:

    • DEFAULT_FAST_UPLOAD

      @Deprecated public static final boolean DEFAULT_FAST_UPLOAD
      Deprecated.
      See Also:

    • FAST_BUFFER_SIZE

      @Deprecated public static final String FAST_BUFFER_SIZE
      Deprecated.
      See Also:

    • DEFAULT_FAST_BUFFER_SIZE

      public static final int DEFAULT_FAST_BUFFER_SIZE
      See Also:

    • FAST_UPLOAD_BUFFER

      public static final String FAST_UPLOAD_BUFFER
      What buffer to use. Default is FAST_UPLOAD_BUFFER_DISK Value: "fs.s3a.fast.upload.buffer"
      See Also:

    • FAST_UPLOAD_BUFFER_DISK

      public static final String FAST_UPLOAD_BUFFER_DISK
      Buffer blocks to disk: "disk". Capacity is limited to available disk space.
      See Also:

    • FAST_UPLOAD_BUFFER_ARRAY

      public static final String FAST_UPLOAD_BUFFER_ARRAY
      Use an in-memory array. Fast but will run of heap rapidly: "array".
      See Also:

    • FAST_UPLOAD_BYTEBUFFER

      public static final String FAST_UPLOAD_BYTEBUFFER
      Use a byte buffer. May be more memory efficient than the FAST_UPLOAD_BUFFER_ARRAY: "bytebuffer".
      See Also:

    • DEFAULT_FAST_UPLOAD_BUFFER

      public static final String DEFAULT_FAST_UPLOAD_BUFFER
      Default buffer option: "disk".
      See Also:

    • FAST_UPLOAD_ACTIVE_BLOCKS

      public static final String FAST_UPLOAD_ACTIVE_BLOCKS
      Maximum Number of blocks a single output stream can have active (uploading, or queued to the central FileSystem instance's pool of queued operations. This stops a single stream overloading the shared thread pool. "fs.s3a.fast.upload.active.blocks"

      Default is DEFAULT_FAST_UPLOAD_ACTIVE_BLOCKS

      See Also:

    • DEFAULT_FAST_UPLOAD_ACTIVE_BLOCKS

      public static final int DEFAULT_FAST_UPLOAD_ACTIVE_BLOCKS
      Limit of queued block upload operations before writes block. Value: 4
      See Also:

    • DOWNGRADE_SYNCABLE_EXCEPTIONS

      public static final String DOWNGRADE_SYNCABLE_EXCEPTIONS
      Rather than raise an exception when an attempt is made to call the Syncable APIs, warn and downgrade. Value: "fs.s3a.downgrade.syncable.exceptions".
      See Also:

    • DOWNGRADE_SYNCABLE_EXCEPTIONS_DEFAULT

      public static final boolean DOWNGRADE_SYNCABLE_EXCEPTIONS_DEFAULT
      Default value for syncable invocation. Value: true.
      See Also:

    • EXECUTOR_CAPACITY

      public static final String EXECUTOR_CAPACITY
      The capacity of executor queues for operations other than block upload, where FAST_UPLOAD_ACTIVE_BLOCKS is used instead. This should be less than MAX_THREADS for fair submission. Value: "fs.s3a.executor.capacity".
      See Also:

    • DEFAULT_EXECUTOR_CAPACITY

      public static final int DEFAULT_EXECUTOR_CAPACITY
      The capacity of executor queues for operations other than block upload, where FAST_UPLOAD_ACTIVE_BLOCKS is used instead. Value: 16
      See Also:

    • CANNED_ACL

      public static final String CANNED_ACL
      See Also:

    • DEFAULT_CANNED_ACL

      public static final String DEFAULT_CANNED_ACL
      See Also:

    • CONTENT_ENCODING

      public static final String CONTENT_ENCODING
      Content encoding: gzip, deflate, compress, br, etc. Value "fs.s3a.object.content.encoding".
      See Also:

    • STORAGE_CLASS

      public static final String STORAGE_CLASS
      S3 storage class: standard, reduced_redundancy, intelligent_tiering etc. Value "fs.s3a.create.storage.class".
      See Also:

    • STORAGE_CLASS_STANDARD

      public static final String STORAGE_CLASS_STANDARD
      S3 Storage option: "standard".
      See Also:

    • STORAGE_CLASS_REDUCED_REDUNDANCY

      public static final String STORAGE_CLASS_REDUCED_REDUNDANCY
      S3 Storage option: "reduced_redundancy".
      See Also:

    • STORAGE_CLASS_GLACIER

      public static final String STORAGE_CLASS_GLACIER
      S3 Storage option: "glacier".
      See Also:

    • STORAGE_CLASS_STANDARD_INFREQUENT_ACCESS

      public static final String STORAGE_CLASS_STANDARD_INFREQUENT_ACCESS
      S3 Storage option: "standard_ia".
      See Also:

    • STORAGE_CLASS_ONEZONE_INFREQUENT_ACCESS

      public static final String STORAGE_CLASS_ONEZONE_INFREQUENT_ACCESS
      S3 Storage option: "onezone_ia".
      See Also:

    • STORAGE_CLASS_INTELLIGENT_TIERING

      public static final String STORAGE_CLASS_INTELLIGENT_TIERING
      S3 Storage option: "intelligent_tiering".
      See Also:

    • STORAGE_CLASS_DEEP_ARCHIVE

      public static final String STORAGE_CLASS_DEEP_ARCHIVE
      S3 Storage option: "deep_archive".
      See Also:

    • STORAGE_CLASS_OUTPOSTS

      public static final String STORAGE_CLASS_OUTPOSTS
      S3 Storage option: "outposts".
      See Also:

    • STORAGE_CLASS_GLACIER_INSTANT_RETRIEVAL

      public static final String STORAGE_CLASS_GLACIER_INSTANT_RETRIEVAL
      S3 Storage option: "glacier_ir".
      See Also:

    • PURGE_EXISTING_MULTIPART

      public static final String PURGE_EXISTING_MULTIPART
      See Also:

    • DEFAULT_PURGE_EXISTING_MULTIPART

      public static final boolean DEFAULT_PURGE_EXISTING_MULTIPART
      See Also:

    • PURGE_EXISTING_MULTIPART_AGE

      public static final String PURGE_EXISTING_MULTIPART_AGE
      purge any multipart uploads older than this number of seconds.
      See Also:

    • DEFAULT_PURGE_EXISTING_MULTIPART_AGE

      public static final long DEFAULT_PURGE_EXISTING_MULTIPART_AGE
      Default Age.

    • SERVER_SIDE_ENCRYPTION_ALGORITHM

      @Deprecated public static final String SERVER_SIDE_ENCRYPTION_ALGORITHM
      Deprecated.
      s3 server-side encryption, see S3AEncryptionMethods for valid options. "fs.s3a.server-side-encryption-algorithm"
      See Also:

    • SERVER_SIDE_ENCRYPTION_AES256

      @Deprecated public static final String SERVER_SIDE_ENCRYPTION_AES256
      Deprecated.
      The standard encryption algorithm AWS supports. Different implementations may support others (or none). Use the S3AEncryptionMethods instead when configuring which Server Side Encryption to use. Value: ""AES256"".
      See Also:

    • SERVER_SIDE_ENCRYPTION_KEY

      @Deprecated public static final String SERVER_SIDE_ENCRYPTION_KEY
      Deprecated.
      Used to specify which AWS KMS key to use if SERVER_SIDE_ENCRYPTION_ALGORITHM is SSE-KMS (will default to aws/s3 master key if left blank). With with SSE_C, the base-64 encoded AES 256 key. May be set within a JCEKS file. Value: ""fs.s3a.server-side-encryption.key"".
      See Also:

    • S3_ENCRYPTION_ALGORITHM

      public static final String S3_ENCRYPTION_ALGORITHM
      Set S3-server side encryption(SSE) or S3-Client side encryption(CSE) algorithm. Check S3AEncryptionMethods for valid options.
      value: "fs.s3a.encryption.algorithm"
      See Also:

    • S3_ENCRYPTION_KEY

      public static final String S3_ENCRYPTION_KEY
      Set S3-SSE or S3-CSE encryption Key if required.
      Note: value:"fs.s3a.encryption.key"
      See Also:

    • S3_ENCRYPTION_CONTEXT

      public static final String S3_ENCRYPTION_CONTEXT
      Set S3-SSE encryption context. The value of this property is a set of non-secret comma-separated key-value pairs of additional contextual information about the data that are separated by equal operator (=). value:"fs.s3a.encryption.context"
      See Also:

    • S3_ENCRYPTION_CSE_CUSTOM_KEYRING_CLASS_NAME

      public static final String S3_ENCRYPTION_CSE_CUSTOM_KEYRING_CLASS_NAME
      Client side encryption (CSE-CUSTOM) with custom cryptographic material manager class name. Custom keyring class name for CSE-KMS. value:"fs.s3a.encryption.cse.custom.keyring.class.name"
      See Also:

    • S3_ENCRYPTION_CSE_V1_COMPATIBILITY_ENABLED

      public static final String S3_ENCRYPTION_CSE_V1_COMPATIBILITY_ENABLED
      Config to provide backward compatibility with V1 encryption client. Enabling this configuration will invoke the followings 1. Unencrypted s3 objects will be read using unencrypted/base s3 client when CSE is enabled. 2. Size of encrypted object will be fetched from object header if present or calculated using ranged S3 GET calls. value:"fs.s3a.encryption.cse.v1.compatibility.enabled"
      See Also:

    • S3_ENCRYPTION_CSE_V1_COMPATIBILITY_ENABLED_DEFAULT

      public static final boolean S3_ENCRYPTION_CSE_V1_COMPATIBILITY_ENABLED_DEFAULT
      Default value : false.
      See Also:

    • S3_ENCRYPTION_CSE_KMS_REGION

      public static final String S3_ENCRYPTION_CSE_KMS_REGION
      S3 CSE-KMS KMS region config.
      See Also:

    • CUSTOM_SIGNERS

      public static final String CUSTOM_SIGNERS
      List of custom Signers. The signer class will be loaded, and the signer name will be associated with this signer class in the S3 SDK. Examples CustomSigner -> 'CustomSigner:org.apache...CustomSignerClass' CustomSigners -> 'CSigner1:CSigner1Class,CSigner2:CSigner2Class' Initializer -> 'CSigner1:CSigner1Class:CSigner1InitializerClass' With Existing -> 'AWS4Signer,CSigner1,CSigner2:CSigner2Class'
      See Also:

    • SIGNING_ALGORITHM

      public static final String SIGNING_ALGORITHM
      Multiple parameters can be used to specify a non-default signing algorithm.
      fs.s3a.signing-algorithm - This property has existed for the longest time. If specified, without other properties being specified, this signing algorithm will be used for all services.
      Another property overrides this value for S3.
      fs.s3a.s3.signing-algorithm - Allows overriding the S3 Signing algorithm. Specifying this property without specifying fs.s3a.signing-algorithm will only update the signing algorithm for S3 requests. fs.s3a.sts.signing-algorithm: algorithm to use for STS interaction.
      See Also:

    • SIGNING_ALGORITHM_S3

      public static final String SIGNING_ALGORITHM_S3

    • SIGNING_ALGORITHM_DDB

      @Deprecated public static final String SIGNING_ALGORITHM_DDB
      Deprecated.

    • SIGNING_ALGORITHM_STS

      public static final String SIGNING_ALGORITHM_STS

    • S3N_FOLDER_SUFFIX

      @Deprecated public static final String S3N_FOLDER_SUFFIX
      Deprecated.
      See Also:

    • FS_S3A_BLOCK_SIZE

      public static final String FS_S3A_BLOCK_SIZE
      See Also:

    • FS_S3A

      public static final String FS_S3A
      See Also:

    • FS_S3A_PREFIX

      public static final String FS_S3A_PREFIX
      Prefix for all S3A properties: "fs.s3a.".
      See Also:

    • FS_S3A_BUCKET_PREFIX

      public static final String FS_S3A_BUCKET_PREFIX
      Prefix for S3A bucket-specific properties: "fs.s3a.bucket.".
      See Also:

    • S3A_DEFAULT_PORT

      public static final int S3A_DEFAULT_PORT
      Default port for this is 443: HTTPS.
      See Also:

    • USER_AGENT_PREFIX

      public static final String USER_AGENT_PREFIX
      See Also:

    • AUTHORITATIVE_PATH

      @Deprecated public static final String AUTHORITATIVE_PATH
      Deprecated.
      Paths considered "authoritative". When S3guard was supported, this skipped checks to s3 on directory listings. It was also possilbe to use to optionally disable marker retentation purely on these paths -a feature which is no longer available. As no feature uses this any more, it is declared as deprecated.
      See Also:

    • DEFAULT_AUTHORITATIVE_PATH

      @Deprecated public static final String[] DEFAULT_AUTHORITATIVE_PATH
      Deprecated.

    • METADATASTORE_AUTHORITATIVE

      @Deprecated public static final String METADATASTORE_AUTHORITATIVE
      Deprecated.
      no longer supported
      Whether or not to allow MetadataStore to be source of truth.
      See Also:

    • DEFAULT_METADATASTORE_AUTHORITATIVE

      @Deprecated public static final boolean DEFAULT_METADATASTORE_AUTHORITATIVE
      Deprecated.
      See Also:

    • S3A_BUCKET_PROBE

      public static final String S3A_BUCKET_PROBE
      Bucket validation parameter which can be set by client. This will be used in S3AFileSystem.initialize(URI, Configuration). Value: "fs.s3a.bucket.probe"
      See Also:

    • S3A_BUCKET_PROBE_DEFAULT

      public static final int S3A_BUCKET_PROBE_DEFAULT
      Default value of bucket validation parameter. An existence of bucket will be validated using S3AFileSystem.verifyBucketExistsV2(). Value: 0
      See Also:

    • METADATASTORE_METADATA_TTL

      @Deprecated public static final String METADATASTORE_METADATA_TTL
      Deprecated.
      no longer supported
      How long a directory listing in the MS is considered as authoritative.
      See Also:

    • DEFAULT_METADATASTORE_METADATA_TTL

      @Deprecated public static final long DEFAULT_METADATASTORE_METADATA_TTL
      Deprecated.
      Default TTL in milliseconds: 15 minutes.

    • READAHEAD_RANGE

      public static final String READAHEAD_RANGE
      read ahead buffer size to prevent connection re-establishments.
      See Also:

    • DEFAULT_READAHEAD_RANGE

      public static final long DEFAULT_READAHEAD_RANGE
      See Also:

    • ASYNC_DRAIN_THRESHOLD

      public static final String ASYNC_DRAIN_THRESHOLD
      The threshold at which drain operations switch to being asynchronous with the schedule/wait overhead compared to synchronous. Value: "fs.s3a.input.async.drain.threshold"
      See Also:

    • DEFAULT_ASYNC_DRAIN_THRESHOLD

      public static final int DEFAULT_ASYNC_DRAIN_THRESHOLD
      This is a number based purely on experimentation in ITestS3AInputStreamPerformance. Value: 16000
      See Also:

    • INPUT_FADVISE

      public static final String INPUT_FADVISE
      Which input strategy to use for buffering, seeking and similar when reading data. Value: "fs.s3a.experimental.input.fadvise"
      See Also:

    • INPUT_FADV_DEFAULT

      public static final String INPUT_FADV_DEFAULT
      Deprecated.
      use the Options.OpenFileOptions value in code which only needs to be compiled against newer hadoop releases.
      The default value for this FS. Which for S3A, is adaptive. Value: "default"
      See Also:

    • INPUT_FADV_NORMAL

      public static final String INPUT_FADV_NORMAL
      General input. Some seeks, some reads. The policy name "default" is standard across different stores, and should be preferred. Value: "normal"
      See Also:

    • INPUT_FADV_SEQUENTIAL

      public static final String INPUT_FADV_SEQUENTIAL
      Deprecated.
      use the Options.OpenFileOptions value in code which only needs to be compiled against newer hadoop releases.
      Optimized for sequential access. Value: "sequential"
      See Also:

    • INPUT_FADV_RANDOM

      public static final String INPUT_FADV_RANDOM
      Deprecated.
      use the Options.OpenFileOptions value in code which only needs to be compiled against newer hadoop releases.
      Optimized purely for random seek+read/positionedRead operations; The performance of sequential IO may be reduced in exchange for more efficient seek() operations. Value: "random"
      See Also:

    • STREAM_READ_GAUGE_INPUT_POLICY

      public static final String STREAM_READ_GAUGE_INPUT_POLICY
      Gauge name for the input policy : "stream_read_gauge_input_policy". This references an enum currently exclusive to the S3A stream.
      See Also:

    • S3_CLIENT_FACTORY_IMPL

      @Private @Unstable public static final String S3_CLIENT_FACTORY_IMPL
      S3 Client Factory implementation class: "fs.s3a.s3.client.factory.impl". Unstable and incompatible between v1 and v2 SDK versions.
      See Also:

    • DEFAULT_S3_CLIENT_FACTORY_IMPL

      @Private @Unstable public static final Class<? extends org.apache.hadoop.fs.s3a.S3ClientFactory> DEFAULT_S3_CLIENT_FACTORY_IMPL
      Default factory: org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.

    • MAX_MULTIPART_COUNT

      @Private public static final int MAX_MULTIPART_COUNT
      Maximum number of partitions in a multipart upload: 10000.
      See Also:

    • S3_METADATA_STORE_IMPL

      @Deprecated public static final String S3_METADATA_STORE_IMPL
      Deprecated.
      See Also:

    • FAIL_ON_METADATA_WRITE_ERROR

      @Deprecated public static final String FAIL_ON_METADATA_WRITE_ERROR
      Deprecated.
      See Also:

    • FAIL_ON_METADATA_WRITE_ERROR_DEFAULT

      @Deprecated public static final boolean FAIL_ON_METADATA_WRITE_ERROR_DEFAULT
      Deprecated.
      See Also:

    • S3GUARD_CLI_PRUNE_AGE

      @Unstable @Deprecated public static final String S3GUARD_CLI_PRUNE_AGE
      Deprecated.
      See Also:

    • S3GUARD_DDB_REGION_KEY

      @Deprecated public static final String S3GUARD_DDB_REGION_KEY
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_NAME_KEY

      @Deprecated public static final String S3GUARD_DDB_TABLE_NAME_KEY
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_TAG

      @Deprecated public static final String S3GUARD_DDB_TABLE_TAG
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_CREATE_KEY

      @Deprecated public static final String S3GUARD_DDB_TABLE_CREATE_KEY
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_CAPACITY_READ_KEY

      @Deprecated public static final String S3GUARD_DDB_TABLE_CAPACITY_READ_KEY
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_CAPACITY_READ_DEFAULT

      @Deprecated public static final long S3GUARD_DDB_TABLE_CAPACITY_READ_DEFAULT
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_CAPACITY_WRITE_KEY

      @Deprecated public static final String S3GUARD_DDB_TABLE_CAPACITY_WRITE_KEY
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_CAPACITY_WRITE_DEFAULT

      @Deprecated public static final long S3GUARD_DDB_TABLE_CAPACITY_WRITE_DEFAULT
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_SSE_ENABLED

      @Deprecated public static final String S3GUARD_DDB_TABLE_SSE_ENABLED
      Deprecated.
      See Also:

    • S3GUARD_DDB_TABLE_SSE_CMK

      @Deprecated public static final String S3GUARD_DDB_TABLE_SSE_CMK
      Deprecated.
      See Also:

    • S3GUARD_DDB_BATCH_WRITE_REQUEST_LIMIT

      @Deprecated public static final int S3GUARD_DDB_BATCH_WRITE_REQUEST_LIMIT
      Deprecated.
      See Also:

    • S3GUARD_DDB_MAX_RETRIES

      @Deprecated public static final String S3GUARD_DDB_MAX_RETRIES
      Deprecated.
      See Also:

    • S3GUARD_DDB_MAX_RETRIES_DEFAULT

      @Deprecated public static final int S3GUARD_DDB_MAX_RETRIES_DEFAULT
      Deprecated.
      See Also:

    • S3GUARD_DDB_THROTTLE_RETRY_INTERVAL

      @Deprecated public static final String S3GUARD_DDB_THROTTLE_RETRY_INTERVAL
      Deprecated.
      See Also:

    • S3GUARD_DDB_THROTTLE_RETRY_INTERVAL_DEFAULT

      @Deprecated public static final String S3GUARD_DDB_THROTTLE_RETRY_INTERVAL_DEFAULT
      Deprecated.
      See Also:

    • S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_KEY

      @Deprecated @Unstable public static final String S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_KEY
      Deprecated.
      See Also:

    • S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_DEFAULT

      @Deprecated public static final int S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_DEFAULT
      Deprecated.
      See Also:

    • S3GUARD_METASTORE_NULL

      @Deprecated public static final String S3GUARD_METASTORE_NULL
      Deprecated.
      The default "Null" metadata store: "org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore".
      See Also:

    • S3GUARD_METASTORE_LOCAL

      @Deprecated @Unstable public static final String S3GUARD_METASTORE_LOCAL
      Deprecated.
      See Also:

    • S3GUARD_METASTORE_LOCAL_MAX_RECORDS

      @Unstable @Deprecated public static final String S3GUARD_METASTORE_LOCAL_MAX_RECORDS
      Deprecated.
      See Also:

    • DEFAULT_S3GUARD_METASTORE_LOCAL_MAX_RECORDS

      @Deprecated public static final int DEFAULT_S3GUARD_METASTORE_LOCAL_MAX_RECORDS
      Deprecated.
      See Also:

    • S3GUARD_METASTORE_LOCAL_ENTRY_TTL

      @Unstable @Deprecated public static final String S3GUARD_METASTORE_LOCAL_ENTRY_TTL
      Deprecated.
      See Also:

    • DEFAULT_S3GUARD_METASTORE_LOCAL_ENTRY_TTL

      @Deprecated public static final int DEFAULT_S3GUARD_METASTORE_LOCAL_ENTRY_TTL
      Deprecated.
      See Also:

    • S3GUARD_METASTORE_DYNAMO

      @Deprecated public static final String S3GUARD_METASTORE_DYNAMO
      Deprecated.
      See Also:

    • S3GUARD_DISABLED_WARN_LEVEL

      @Deprecated public static final String S3GUARD_DISABLED_WARN_LEVEL
      Deprecated.
      See Also:

    • DEFAULT_S3GUARD_DISABLED_WARN_LEVEL

      @Deprecated public static final String DEFAULT_S3GUARD_DISABLED_WARN_LEVEL
      Deprecated.
      See Also:

    • FAIL_INJECT_INCONSISTENCY_KEY

      @Deprecated public static final String FAIL_INJECT_INCONSISTENCY_KEY
      Deprecated.
      Inconsistency (visibility delay) injection settings. No longer used.
      See Also:

    • FAIL_INJECT_INCONSISTENCY_MSEC

      @Deprecated public static final String FAIL_INJECT_INCONSISTENCY_MSEC
      Deprecated.
      See Also:

    • FAIL_INJECT_INCONSISTENCY_PROBABILITY

      @Deprecated public static final String FAIL_INJECT_INCONSISTENCY_PROBABILITY
      Deprecated.
      See Also:

    • LIST_VERSION

      @Unstable public static final String LIST_VERSION
      S3 API level parameters.
      See Also:

    • DEFAULT_LIST_VERSION

      @Unstable public static final int DEFAULT_LIST_VERSION
      See Also:

    • FAIL_INJECT_THROTTLE_PROBABILITY

      @Unstable public static final String FAIL_INJECT_THROTTLE_PROBABILITY
      See Also:

    • FAIL_INJECT_CLIENT_FACTORY

      @Unstable public static final String FAIL_INJECT_CLIENT_FACTORY
      See Also:

    • RETRY_LIMIT

      public static final String RETRY_LIMIT
      Number of times to retry any repeatable S3 client request on failure, excluding throttling requests: "fs.s3a.retry.limit".
      See Also:

    • RETRY_LIMIT_DEFAULT

      public static final int RETRY_LIMIT_DEFAULT
      Default retry limit: 7.
      See Also:

    • RETRY_INTERVAL

      public static final String RETRY_INTERVAL
      Interval between retry attempts.: "fs.s3a.retry.interval".
      See Also:

    • RETRY_INTERVAL_DEFAULT

      public static final String RETRY_INTERVAL_DEFAULT
      Default retry interval: "500ms".
      See Also:

    • RETRY_THROTTLE_LIMIT

      public static final String RETRY_THROTTLE_LIMIT
      Number of times to retry any throttled request: "fs.s3a.retry.throttle.limit".
      See Also:

    • RETRY_THROTTLE_LIMIT_DEFAULT

      public static final int RETRY_THROTTLE_LIMIT_DEFAULT
      Default throttled retry limit: 20.
      See Also:

    • RETRY_THROTTLE_INTERVAL

      public static final String RETRY_THROTTLE_INTERVAL
      Interval between retry attempts on throttled requests: "fs.s3a.retry.throttle.interval".
      See Also:

    • RETRY_THROTTLE_INTERVAL_DEFAULT

      public static final String RETRY_THROTTLE_INTERVAL_DEFAULT
      Default throttled retry interval: "500ms".
      See Also:

    • RETRY_HTTP_5XX_ERRORS

      public static final String RETRY_HTTP_5XX_ERRORS
      Should S3A connector retry on all 5xx errors which don't have explicit support: "fs.s3a.retry.http.5xx.errors"?

      This is in addition to any retries the AWS SDK itself does, which is known to retry on many of these (e.g. 500).

      See Also:

    • DEFAULT_RETRY_HTTP_5XX_ERRORS

      public static final boolean DEFAULT_RETRY_HTTP_5XX_ERRORS
      Default value for RETRY_HTTP_5XX_ERRORS: true.
      See Also:

    • ETAG_CHECKSUM_ENABLED

      public static final String ETAG_CHECKSUM_ENABLED
      Should etags be exposed as checksums?
      See Also:

    • ETAG_CHECKSUM_ENABLED_DEFAULT

      public static final boolean ETAG_CHECKSUM_ENABLED_DEFAULT
      Default value: false.
      See Also:

    • CHANGE_DETECT_SOURCE

      public static final String CHANGE_DETECT_SOURCE
      Where to get the value to use in change detection. E.g. eTag, or versionId?
      See Also:

    • CHANGE_DETECT_SOURCE_ETAG

      public static final String CHANGE_DETECT_SOURCE_ETAG
      eTag as the change detection mechanism.
      See Also:

    • CHANGE_DETECT_SOURCE_VERSION_ID

      public static final String CHANGE_DETECT_SOURCE_VERSION_ID
      Object versionId as the change detection mechanism.
      See Also:

    • CHANGE_DETECT_SOURCE_DEFAULT

      public static final String CHANGE_DETECT_SOURCE_DEFAULT
      Default change detection mechanism: eTag.
      See Also:

    • CHANGE_DETECT_MODE

      public static final String CHANGE_DETECT_MODE
      Mode to run change detection in. Server side comparison? Client side comparison? Client side compare and warn rather than exception? Don't bother at all?
      See Also:

    • CHANGE_DETECT_MODE_CLIENT

      public static final String CHANGE_DETECT_MODE_CLIENT
      Change is detected on the client side by comparing the returned id with the expected id. A difference results in RemoteFileChangedException.
      See Also:

    • CHANGE_DETECT_MODE_SERVER

      public static final String CHANGE_DETECT_MODE_SERVER
      Change is detected by passing the expected value in the GetObject request. If the expected value is unavailable, RemoteFileChangedException is thrown.
      See Also:

    • CHANGE_DETECT_MODE_WARN

      public static final String CHANGE_DETECT_MODE_WARN
      Change is detected on the client side by comparing the returned id with the expected id. A difference results in a WARN level message being logged.
      See Also:

    • CHANGE_DETECT_MODE_NONE

      public static final String CHANGE_DETECT_MODE_NONE
      Change detection is turned off. Readers may see inconsistent results due to concurrent writes without any exception or warning messages. May be useful with third-party S3 API implementations that don't support one of the change detection modes.
      See Also:

    • CHANGE_DETECT_MODE_DEFAULT

      public static final String CHANGE_DETECT_MODE_DEFAULT
      Default change detection mode: server.
      See Also:

    • CHANGE_DETECT_REQUIRE_VERSION

      public static final String CHANGE_DETECT_REQUIRE_VERSION
      If true, raises a RemoteFileChangedException exception when S3 doesn't provide the attribute defined by fs.s3a.change.detection.source. For example, if source is versionId, but object versioning is not enabled on the bucket, or alternatively if source is eTag and a third-party S3 implementation that doesn't return eTag is used.

      When false, only a warning message will be logged for this condition.

      See Also:

    • CHANGE_DETECT_REQUIRE_VERSION_DEFAULT

      public static final boolean CHANGE_DETECT_REQUIRE_VERSION_DEFAULT
      Default change detection require version: true.
      See Also:

    • S3GUARD_CONSISTENCY_RETRY_LIMIT

      @Deprecated public static final String S3GUARD_CONSISTENCY_RETRY_LIMIT
      Deprecated.
      Number of times to retry any repeatable S3 client request on failure, excluding throttling requests: "fs.s3a.s3guard.consistency.retry.limit".
      See Also:

    • S3GUARD_CONSISTENCY_RETRY_LIMIT_DEFAULT

      @Deprecated public static final int S3GUARD_CONSISTENCY_RETRY_LIMIT_DEFAULT
      Deprecated.
      Default retry limit: 7.
      See Also:

    • S3GUARD_CONSISTENCY_RETRY_INTERVAL

      @Deprecated public static final String S3GUARD_CONSISTENCY_RETRY_INTERVAL
      Deprecated.
      Initial retry interval: "fs.s3a.s3guard.consistency.retry.interval".
      See Also:

    • S3GUARD_CONSISTENCY_RETRY_INTERVAL_DEFAULT

      @Deprecated public static final String S3GUARD_CONSISTENCY_RETRY_INTERVAL_DEFAULT
      Deprecated.
      Default initial retry interval: "2s". The consistency retry probe uses exponential backoff, because each probe can cause the S3 load balancers to retain any 404 in its cache for longer. See HADOOP-16490.
      See Also:

    • AWS_SERVICE_IDENTIFIER_S3

      public static final String AWS_SERVICE_IDENTIFIER_S3
      See Also:

    • AWS_SERVICE_IDENTIFIER_DDB

      @Deprecated public static final String AWS_SERVICE_IDENTIFIER_DDB
      Deprecated.
      See Also:

    • AWS_SERVICE_IDENTIFIER_STS

      public static final String AWS_SERVICE_IDENTIFIER_STS
      See Also:

    • FS_S3A_CLIENT_PREFIX

      public static final String FS_S3A_CLIENT_PREFIX
      Prefix for S3A client-specific properties. value: "fs.s3a.client."
      See Also:

    • CUSTOM_HEADERS_POSTFIX

      public static final String CUSTOM_HEADERS_POSTFIX
      Custom headers postfix. value: ".custom.headers"
      See Also:

    • CUSTOM_HEADERS_STS

      public static final String CUSTOM_HEADERS_STS
      List of custom headers to be set on the service client. Multiple parameters can be used to specify custom headers. 
       Usage:
 fs.s3a.client.s3.custom.headers - Headers to add on all the S3 requests.
 fs.s3a.client.sts.custom.headers - Headers to add on all the STS requests.

 Examples:
 CustomHeader -> 'Header1:Value1'
 CustomHeaders -> 'Header1=Value1;Value2,Header2=Value1'

    • CUSTOM_HEADERS_S3

      public static final String CUSTOM_HEADERS_S3

    • THREAD_POOL_SHUTDOWN_DELAY_SECONDS

      public static final int THREAD_POOL_SHUTDOWN_DELAY_SECONDS
      How long to wait for the thread pool to terminate when cleaning up. Value: 30 seconds.
      See Also:

    • DIRECTORY_MARKER_POLICY

      @Deprecated public static final String DIRECTORY_MARKER_POLICY
      Deprecated.
      Policy for directory markers. No longer supported as "keep" is the sole policy.
      See Also:

    • DIRECTORY_MARKER_POLICY_DELETE

      @Deprecated public static final String DIRECTORY_MARKER_POLICY_DELETE
      Deprecated.
      Delete directory markers. No longer supported as "keep" is the sole policy. Value: "delete".
      See Also:

    • DIRECTORY_MARKER_POLICY_KEEP

      @Deprecated public static final String DIRECTORY_MARKER_POLICY_KEEP
      Deprecated.
      Retain directory markers. No longer needed, so marked as deprecated to flag usages. Value: "keep".
      See Also:

    • DIRECTORY_MARKER_POLICY_AUTHORITATIVE

      @Deprecated public static final String DIRECTORY_MARKER_POLICY_AUTHORITATIVE
      Deprecated.
      Retain directory markers in authoritative directory trees only. No longer required as "keep" is the sole policy. Value: "authoritative".
      See Also:

    • DEFAULT_DIRECTORY_MARKER_POLICY

      @Deprecated public static final String DEFAULT_DIRECTORY_MARKER_POLICY
      Deprecated.
      Default retention policy: "keep". No longer required as "keep" is the sole policy.
      See Also:

    • STORE_CAPABILITY_DIRECTORY_MARKER_AWARE

      public static final String STORE_CAPABILITY_DIRECTORY_MARKER_AWARE
      PathCapabilities probe to verify that an S3A Filesystem has the changes needed to safely work with buckets where directory markers have not been deleted. Value: "fs.s3a.capability.directory.marker.aware".
      See Also:

    • STORE_CAPABILITY_DIRECTORY_MARKER_POLICY_KEEP

      public static final String STORE_CAPABILITY_DIRECTORY_MARKER_POLICY_KEEP
      PathCapabilities probe to indicate that the filesystem keeps directory markers. Value: "fs.s3a.capability.directory.marker.policy.keep".
      See Also:

    • STORE_CAPABILITY_DIRECTORY_MARKER_POLICY_DELETE

      @Deprecated public static final String STORE_CAPABILITY_DIRECTORY_MARKER_POLICY_DELETE
      Deprecated.
      PathCapabilities probe to indicate that the filesystem deletes directory markers. Always false. Value: "fs.s3a.capability.directory.marker.policy.delete".
      See Also:

    • STORE_CAPABILITY_DIRECTORY_MARKER_POLICY_AUTHORITATIVE

      @Deprecated public static final String STORE_CAPABILITY_DIRECTORY_MARKER_POLICY_AUTHORITATIVE
      Deprecated.
      PathCapabilities probe to indicate that the filesystem keeps directory markers in authoritative paths only. This probe always returns false. Value: "fs.s3a.capability.directory.marker.policy.authoritative".
      See Also:

    • STORE_CAPABILITY_DIRECTORY_MARKER_ACTION_KEEP

      public static final String STORE_CAPABILITY_DIRECTORY_MARKER_ACTION_KEEP
      PathCapabilities probe to indicate that a path keeps directory markers. This probe always returns true. Value: "fs.s3a.capability.directory.marker.action.keep".
      See Also:

    • STORE_CAPABILITY_DIRECTORY_MARKER_ACTION_DELETE

      public static final String STORE_CAPABILITY_DIRECTORY_MARKER_ACTION_DELETE
      PathCapabilities probe to indicate that a path deletes directory markers. This probe always returns false. Value: "fs.s3a.capability.directory.marker.action.delete".
      See Also:

    • XA_HEADER_PREFIX

      public static final String XA_HEADER_PREFIX
      To comply with the XAttr rules, all headers of the object retrieved through the getXAttr APIs have the prefix: "header.".
      See Also:

    • AWS_S3_CROSS_REGION_ACCESS_ENABLED

      public static final String AWS_S3_CROSS_REGION_ACCESS_ENABLED
      S3 cross region access enabled ? Value: "fs.s3a.cross.region.access.enabled".
      See Also:

    • AWS_S3_CROSS_REGION_ACCESS_ENABLED_DEFAULT

      public static final boolean AWS_S3_CROSS_REGION_ACCESS_ENABLED_DEFAULT
      Default value for S3 cross region access enabled: true.
      See Also:

    • AWS_REGION

      public static final String AWS_REGION
      AWS S3 region for the bucket. When set bypasses the construction of region through endpoint url.
      See Also:

    • AWS_S3_CENTRAL_REGION

      public static final String AWS_S3_CENTRAL_REGION
      The special S3 region which can be used to talk to any bucket. Value "us-east-1".
      See Also:

    • AWS_S3_DEFAULT_REGION

      public static final String AWS_S3_DEFAULT_REGION
      The default S3 region when using cross region client. Value "us-east-2".
      See Also:

    • FIPS_ENDPOINT

      public static final String FIPS_ENDPOINT
      Is the endpoint a FIPS endpoint? Can be queried as a path capability. Value "fs.s3a.endpoint.fips".
      See Also:

    • ENDPOINT_FIPS_DEFAULT

      public static final boolean ENDPOINT_FIPS_DEFAULT
      See Also:

    • AWS_S3_ACCESSPOINT_REQUIRED

      public static final String AWS_S3_ACCESSPOINT_REQUIRED
      Require that all S3 access is made through Access Points.
      See Also:

    • FS_S3A_CREATE_PERFORMANCE

      public static final String FS_S3A_CREATE_PERFORMANCE
      Flag for create performance. This can be set in the {code createFile()} builder. Value "fs.s3a.create.performance".
      See Also:

    • FS_S3A_CREATE_PERFORMANCE_DEFAULT

      public static final boolean FS_S3A_CREATE_PERFORMANCE_DEFAULT
      Default value for create performance in an S3A FS. Value false.
      See Also:

    • FS_S3A_CREATE_PERFORMANCE_ENABLED

      public static final String FS_S3A_CREATE_PERFORMANCE_ENABLED
      Capability to indicate that the FS has been instantiated with FS_S3A_CREATE_PERFORMANCE set to true. Value "fs.s3a.create.performance.enabled".
      See Also:

    • FS_S3A_PERFORMANCE_FLAGS

      public static final String FS_S3A_PERFORMANCE_FLAGS
      Comma separated list of performance flags.
      See Also:

    • FS_S3A_CONDITIONAL_CREATE_ENABLED

      public static final String FS_S3A_CONDITIONAL_CREATE_ENABLED
      Is the create overwrite feature enabled or not? A configuration option and a path status probe. Value "fs.s3a.create.conditional.enabled".
      See Also:

    • DEFAULT_FS_S3A_CONDITIONAL_CREATE_ENABLED

      public static final boolean DEFAULT_FS_S3A_CONDITIONAL_CREATE_ENABLED
      Default value for FS_S3A_CONDITIONAL_CREATE_ENABLED. Value true.
      See Also:

    • FS_S3A_CREATE_MULTIPART

      public static final String FS_S3A_CREATE_MULTIPART
      createFile() boolean option toreate a multipart file, always: "fs.s3a.create.multipart".

      This is inefficient and will not work on a store which doesn't support that feature, so is primarily for testing.

      See Also:

    • FS_S3A_CREATE_HEADER

      public static final String FS_S3A_CREATE_HEADER
      Prefix for adding a header to the object when created. The actual value must have a "." suffix and then the actual header. This is *not* a configuration option; it is only for use in the {code createFile()} builder. Value "fs.s3a.create.header".
      See Also:

    • AWS_S3_VECTOR_READS_MIN_SEEK_SIZE

      public static final String AWS_S3_VECTOR_READS_MIN_SEEK_SIZE
      What is the smallest reasonable seek in bytes such that we group ranges together during vectored read operation. Value : "fs.s3a.vectored.read.min.seek.size".
      See Also:

    • AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE

      public static final String AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE
      What is the largest merged read size in bytes such that we group ranges together during vectored read. Setting this value to 0 will disable merging of ranges. Value : "fs.s3a.vectored.read.max.merged.size".
      See Also:

    • DEFAULT_AWS_S3_VECTOR_READS_MIN_SEEK_SIZE

      public static final int DEFAULT_AWS_S3_VECTOR_READS_MIN_SEEK_SIZE
      Default minimum seek in bytes during vectored reads: 131072.
      See Also:

    • DEFAULT_AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE

      public static final int DEFAULT_AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE
      Default maximum read size in bytes during vectored reads : 2097152.
      See Also:

    • AWS_S3_VECTOR_ACTIVE_RANGE_READS

      public static final String AWS_S3_VECTOR_ACTIVE_RANGE_READS
      Maximum number of range reads a single input stream can have active (downloading, or queued) to the central FileSystem instance's pool of queued operations. This stops a single stream overloading the shared thread pool. "fs.s3a.vectored.active.ranged.reads"

      Default is DEFAULT_AWS_S3_VECTOR_ACTIVE_RANGE_READS

      See Also:

    • DEFAULT_AWS_S3_VECTOR_ACTIVE_RANGE_READS

      public static final int DEFAULT_AWS_S3_VECTOR_ACTIVE_RANGE_READS
      Limit of queued range data download operations during vectored read. Value: 4
      See Also:

    • AWS_AUTH_CLASS_PREFIX

      public static final String AWS_AUTH_CLASS_PREFIX
      Prefix of auth classes in AWS SDK V1.
      See Also:

    • INPUT_STREAM_TYPE

      public static final String INPUT_STREAM_TYPE
      Input stream type: "fs.s3a.input.stream.type".
      See Also:

    • INPUT_STREAM_TYPE_CLASSIC

      public static final String INPUT_STREAM_TYPE_CLASSIC
      The classic input stream: "classic".
      See Also:

    • INPUT_STREAM_TYPE_PREFETCH

      public static final String INPUT_STREAM_TYPE_PREFETCH
      The prefetching input stream: "prefetch".
      See Also:

    • INPUT_STREAM_TYPE_ANALYTICS

      public static final String INPUT_STREAM_TYPE_ANALYTICS
      The analytics input stream: "analytics".
      See Also:

    • INPUT_STREAM_TYPE_DEFAULT

      public static final String INPUT_STREAM_TYPE_DEFAULT
      Request the default input stream, whatever it is for this release: "default".
      See Also:

    • INPUT_STREAM_TYPE_CUSTOM

      public static final String INPUT_STREAM_TYPE_CUSTOM
      The custom input stream type: "custom"". If set, the classname is loaded from INPUT_STREAM_CUSTOM_FACTORY.

      This option is primarily for testing as it can be used to generated failures.

      See Also:

    • INPUT_STREAM_CUSTOM_FACTORY

      public static final String INPUT_STREAM_CUSTOM_FACTORY
      Classname of the factory to instantiate for custom streams: "fs.s3a.input.stream.custom.factory".
      See Also:

    • PREFETCH_ENABLED_KEY

      @Deprecated public static final String PREFETCH_ENABLED_KEY
      Deprecated.
      Controls whether the prefetching input stream is enabled.
      See Also:

    • PREFETCH_ENABLED_DEFAULT

      @Deprecated public static final boolean PREFETCH_ENABLED_DEFAULT
      Deprecated.
      Default option as to whether the prefetching input stream is enabled.
      See Also:

    • PREFETCH_BLOCK_SIZE_KEY

      public static final String PREFETCH_BLOCK_SIZE_KEY
      The size of a single prefetched block in number of bytes.
      See Also:

    • PREFETCH_BLOCK_DEFAULT_SIZE

      public static final int PREFETCH_BLOCK_DEFAULT_SIZE
      See Also:

    • PREFETCH_BLOCK_COUNT_KEY

      public static final String PREFETCH_BLOCK_COUNT_KEY
      Maximum number of blocks prefetched at any given time.
      See Also:

    • PREFETCH_BLOCK_DEFAULT_COUNT

      public static final int PREFETCH_BLOCK_DEFAULT_COUNT
      See Also:

    • MULTIPART_UPLOADS_ENABLED

      public static final String MULTIPART_UPLOADS_ENABLED
      Option to enable or disable the multipart uploads. Value: "fs.s3a.multipart.uploads.enabled".

      Default is DEFAULT_MULTIPART_UPLOAD_ENABLED.

      See Also:

    • DEFAULT_MULTIPART_UPLOAD_ENABLED

      public static final boolean DEFAULT_MULTIPART_UPLOAD_ENABLED
      Default value for multipart uploads. true
      See Also:

    • STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED

      public static final String STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED
      Stream supports multipart uploads to the given path.
      See Also:

    • STORE_CAPABILITY_DIRECTORY_MARKER_MULTIPART_UPLOAD_ENABLED

      @Deprecated public static final String STORE_CAPABILITY_DIRECTORY_MARKER_MULTIPART_UPLOAD_ENABLED
      Deprecated.
      Stream supports multipart uploads to the given path. This name is wrong, but it has shipped so must be retained.
      See Also:

    • PREFETCH_MAX_BLOCKS_COUNT

      public static final String PREFETCH_MAX_BLOCKS_COUNT
      Prefetch max blocks count config. Value = "fs.s3a.prefetch.max.blocks.count"
      See Also:

    • DEFAULT_PREFETCH_MAX_BLOCKS_COUNT

      public static final int DEFAULT_PREFETCH_MAX_BLOCKS_COUNT
      Default value for max blocks count config. Value = 4
      See Also:

    • BUCKET_REGION_HEADER

      public static final String BUCKET_REGION_HEADER
      The bucket region header.
      See Also:

    • DIRECTORY_OPERATIONS_PURGE_UPLOADS

      public static final String DIRECTORY_OPERATIONS_PURGE_UPLOADS
      Should directory operations purge uploads? This adds at least one parallelized list operation to the call, plus the overhead of deletions. Value: "fs.s3a.directory.operations.purge.uploads".
      See Also:

    • DIRECTORY_OPERATIONS_PURGE_UPLOADS_DEFAULT

      public static final boolean DIRECTORY_OPERATIONS_PURGE_UPLOADS_DEFAULT
      Default value of DIRECTORY_OPERATIONS_PURGE_UPLOADS: false.
      See Also:

    • OPTIMIZED_COPY_FROM_LOCAL

      public static final String OPTIMIZED_COPY_FROM_LOCAL
      Is the higher performance copy from local file to S3 enabled? This switch allows for it to be disabled if there are problems. Value: "fs.s3a.optimized.copy.from.local.enabled".
      See Also:

    • OPTIMIZED_COPY_FROM_LOCAL_DEFAULT

      public static final boolean OPTIMIZED_COPY_FROM_LOCAL_DEFAULT
      Default value for OPTIMIZED_COPY_FROM_LOCAL. Value: true.
      See Also:

    • STORE_CAPABILITY_AWS_V2

      public static final String STORE_CAPABILITY_AWS_V2
      Is this a v2 SDK build? value "fs.s3a.capability.aws.v2".
      See Also:

    • S3EXPRESS_CREATE_SESSION

      public static final String S3EXPRESS_CREATE_SESSION
      Use the S3 Express createSession() operation to authenticate with S3Express storage?

      Value: "fs.s3a.s3express.create.session".

      This is preferred as it is faster, but it does require extra IAM permissions and is not suited to some deployments, including some of the hadoop-aws test suites.

      See Also:

    • S3EXPRESS_CREATE_SESSION_DEFAULT

      public static final boolean S3EXPRESS_CREATE_SESSION_DEFAULT
      Default value of S3EXPRESS_CREATE_SESSION. Value: true.
      See Also:

    • HTTP_SIGNER_ENABLED

      public static final String HTTP_SIGNER_ENABLED
      Flag to switch to a v2 SDK HTTP signer. Value "fs.s3a.http.signer.enabled".
      See Also:

    • HTTP_SIGNER_ENABLED_DEFAULT

      public static final boolean HTTP_SIGNER_ENABLED_DEFAULT
      Default value of HTTP_SIGNER_ENABLED: false.
      See Also:

    • HTTP_SIGNER_CLASS_NAME

      public static final String HTTP_SIGNER_CLASS_NAME
      Classname of the http signer to use when HTTP_SIGNER_ENABLED is true: "fs.s3a.http.signer.class".
      See Also:

    • CHECKSUM_VALIDATION

      public static final String CHECKSUM_VALIDATION
      Should checksums be validated on download? This is slower and not needed on TLS connections. Value: "fs.s3a.checksum.validation".
      See Also:

    • CHECKSUM_VALIDATION_DEFAULT

      public static final boolean CHECKSUM_VALIDATION_DEFAULT
      Default value of CHECKSUM_VALIDATION. Value: false.
      See Also:

    • CHECKSUM_GENERATION

      public static final String CHECKSUM_GENERATION
      Should checksums always be generated? Not all third-party stores like this being enabled for every request. Value: "fs.s3a.checksum.generation".
      See Also:

    • DEFAULT_CHECKSUM_GENERATION

      public static final boolean DEFAULT_CHECKSUM_GENERATION
      Default value of CHECKSUM_GENERATION. Value: false.
      See Also:

    • CHECKSUM_ALGORITHM

      public static final String CHECKSUM_ALGORITHM
      Indicates the algorithm used to create the checksum for the object to be uploaded to S3. Unset by default. It supports the following values: 'CRC32', 'CRC32C', 'SHA1', 'SHA256', 'CRC64_NVME 'NONE', ''. When checksum calculation is enabled this MUST be set to a valid algorithm. value:"fs.s3a.create.checksum.algorithm"
      See Also:

    • DEFAULT_CHECKSUM_ALGORITHM

      public static final String DEFAULT_CHECKSUM_ALGORITHM
      Default checksum algorithm: "NONE".
      See Also:

    • REQUEST_MD5_HEADER

      public static final String REQUEST_MD5_HEADER
      Send a Content-MD5 header with every request. This is required when performing some operations with third party stores For example: bulk delete). It is supported by AWS S3, though has unexpected behavior with AWS S3 Express storage. See https://github.com/aws/aws-sdk-java-v2/issues/6459 for details.
      See Also:

    • DEFAULT_REQUEST_MD5_HEADER

      public static final boolean DEFAULT_REQUEST_MD5_HEADER
      Default value of REQUEST_MD5_HEADER. Value: true.
      See Also:

    • AWS_S3_CLASSLOADER_ISOLATION

      public static final String AWS_S3_CLASSLOADER_ISOLATION
      Are extensions classes, such as fs.s3a.aws.credentials.provider, going to be loaded from the same classloader that loaded the S3AFileSystem? It is useful to turn classloader isolation off for Apache Spark applications that might load S3AFileSystem from the Spark distribution (Launcher classloader) while users might want to provide custom extensions (loaded by Spark MutableClassloader). Value: "fs.s3a.classloader.isolation".
      See Also:

    • DEFAULT_AWS_S3_CLASSLOADER_ISOLATION

      public static final boolean DEFAULT_AWS_S3_CLASSLOADER_ISOLATION
      Default value for AWS_S3_CLASSLOADER_ISOLATION. Value: true.
      See Also:

    • AWS_S3_ACCESS_GRANTS_ENABLED

      public static final String AWS_S3_ACCESS_GRANTS_ENABLED
      Flag "fs.s3a.access.grants.enabled" to enable S3 Access Grants to control authorization to S3 data. More information: https://aws.amazon.com/s3/features/access-grants/ and https://github.com/aws/aws-s3-accessgrants-plugin-java-v2/
      See Also:

    • AWS_S3_ACCESS_GRANTS_FALLBACK_TO_IAM_ENABLED

      public static final String AWS_S3_ACCESS_GRANTS_FALLBACK_TO_IAM_ENABLED
      Flag "fs.s3a.access.grants.fallback.to.iam" to enable jobs fall back to the Job Execution IAM role in case they get Access Denied from the S3 Access Grants call. More information: https://github.com/aws/aws-s3-accessgrants-plugin-java-v2/
      See Also:

    • DEFAULT_S3A_IO_RATE_LIMIT

      public static final int DEFAULT_S3A_IO_RATE_LIMIT
      Default value for S3A_IO_RATE_LIMIT. Value: 0. 0 means no rate limiting.
      See Also:

    • S3A_IO_RATE_LIMIT

      public static final String S3A_IO_RATE_LIMIT
      Config to set the rate limit for S3A IO operations. Value: "fs.s3a.io.rate.limit".
      See Also:

    • ANALYTICS_ACCELERATOR_CONFIGURATION_PREFIX

      public static final String ANALYTICS_ACCELERATOR_CONFIGURATION_PREFIX
      Prefix to configure Analytics Accelerator Library. Value: "fs.s3a.analytics.accelerator".
      See Also:

    • IF_NONE_MATCH_STAR

      public static final String IF_NONE_MATCH_STAR
      Value for the If-None-Match HTTP header in S3 requests. Value: "*". More information: AWS S3 PutObject API Documentation
      See Also: