Working with Delegation Tokens

S3A Session Delegation Tokens

A Session Delegation Token is created by asking the AWS Security Token Service to issue an AWS session password and identifier for a limited duration. These AWS session credentials are valid until the end of that time period. They are marshalled into the S3A Delegation Token.

Other S3A connectors can extract these credentials and use them to talk to S3 and related services.

Issued tokens cannot be renewed or revoked.

See GetSessionToken for specifics details on the (current) token lifespan.

S3A Role Delegation Tokens

A Role Delegation Token is created by asking the AWS Security Token Service for a set of “Assumed Role” session credentials with a limited lifetime, belonging to a given IAM Role. The resulting session credentials are restricted to grant access to all KMS keys, and to the specific S3 bucket. They are marshalled into the S3A Delegation Token.

Other S3A connectors can extract these credentials and use them to talk to S3 and related services. They may only work with the explicit AWS resources identified when the token was generated.

Issued tokens cannot be renewed or revoked.

S3A Full-Credential Delegation Tokens

Full Credential Delegation Tokens contain the full AWS login details (access key and secret key) needed to access a bucket.

They never expire, so are the equivalent of storing the AWS account credentials in a Hadoop, Hive, Spark configuration or similar.

The differences are:

1. They are automatically passed from the client/user to the application. A remote application can use them to access data on behalf of the user.
2. When a remote application destroys the filesystem connector instances and tokens of a user, the secrets are destroyed too.
3. Secrets in the AWS_ environment variables on the client will be picked up and automatically propagated.
4. They do not use the AWS STS service, so may work against third-party implementations of the S3 protocol.

Configuration Parameters

Warnings

Enabling Session Delegation Tokens

For session tokens, set fs.s3a.delegation.token.binding to org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding

There some further configuration options.

The XML settings needed to enable session tokens are:

<property>
  <name>fs.s3a.delegation.token.binding</name>
  <value>org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding</value>
</property>
<property>
  <name>fs.s3a.assumed.role.session.duration</name>
  <value>1h</value>
</property>

1. If the application requesting a token has full AWS credentials for the relevant bucket, then a new session token will be issued.
2. If the application requesting a token is itself authenticating with a session delegation token, then the existing token will be forwarded. The life of the token will not be extended.
3. If the application requesting a token does not have either of these, the token cannot be issued: the operation will fail with an error.

The endpoint for STS requests are set by the same configuration property as for the AssumedRole credential provider and for Role Delegation tokens.

<!-- Optional -->
<property>
  <name>fs.s3a.assumed.role.sts.endpoint</name>
  <value>sts.amazonaws.com</value>
</property>
<property>
  <name>fs.s3a.assumed.role.sts.endpoint.region</name>
  <value>us-west-1</value>
</property>

If the fs.s3a.assumed.role.sts.endpoint option is set, or set to something other than the central sts.amazonaws.com endpoint, then the region property must be set.

Both the Session and the Role Delegation Token bindings use the option fs.s3a.aws.credentials.provider to define the credential providers to authenticate to the AWS STS with.

Here is the effective list of providers if none are declared:

<property>
  <name>fs.s3a.aws.credentials.provider</name>
  <value>
    org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
    org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
    software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,
    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
  </value>
</property>

Not all these authentication mechanisms provide the full set of credentials STS needs. The session token provider will simply forward any session credentials it is authenticated with; the role token binding will fail.

Enabling Role Delegation Tokens

For role delegation tokens, set fs.s3a.delegation.token.binding to org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding

There are some further configuration options:

The option fs.s3a.assumed.role.arn must be set to a role which the user can assume. It must have permissions to access the bucket and any KMS encryption keys. The actual requested role will be this role, explicitly restricted to the specific bucket.

The XML settings needed to enable session tokens are:

<property>
  <name>fs.s3a.delegation.token.binding</name>
  <value>org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding</value>
</property>
<property>
  <name>fs.s3a.assumed.role.arn</name>
  <value>ARN of role to request</value>
  <value>REQUIRED ARN</value>
</property>
<property>
  <name>fs.s3a.assumed.role.session.duration</name>
  <value>1h</value>
</property>

A JSON role policy for the role/session will automatically be generated which will consist of:

1. Full access to the S3 bucket for all operations used by the S3A client (read, write, list, multipart operations, get bucket location, etc).
2. Full user access to KMS keys. This is to be able to decrypt any data in the bucket encrypted with SSE-KMS, as well as encrypt new data if that is the encryption policy.

Enabling Full Delegation Tokens

This passes the full credentials in, falling back to any session credentials which were used to configure the S3A FileSystem instance.

For Full Credential Delegation tokens, set fs.s3a.delegation.token.binding to org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding

There are no other configuration options.

<property>
  <name>fs.s3a.delegation.token.binding</name>
  <value>org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding</value>
</property>

Key points:

1. If the application requesting a token has full AWS credentials for the relevant bucket, then a full credential token will be issued.
2. If the application requesting a token is itself authenticating with a session delegation token, then the existing token will be forwarded. The life of the token will not be extended.
3. If the application requesting a token does not have either of these, the tokens cannot be issued: the operation will fail with an error.

Submitted job cannot authenticate

There are many causes for this; delegation tokens add some more.

Tokens are not issued

• This user is not kinit-ed in to Kerberos. Use klist and hadoop kdiag to see the Kerberos authentication state of the logged in user.
• The filesystem instance on the client does not have a token binding set in fs.s3a.delegation.token.binding, so does not attempt to issue any.
• The job submission is not aware that access to the specific S3 buckets are required. Review the application’s submission mechanism to determine how to list source and destination paths. For example, for MapReduce, tokens for the cluster filesystem (fs.defaultFS) and all filesystems referenced as input and output paths will be queried for delegation tokens.

For Apache Spark, the cluster filesystem and any filesystems listed in the property spark.yarn.access.hadoopFileSystems are queried for delegation tokens in secure clusters. See Running on Yarn.

Error No AWS login credentials

The client does not have any valid credentials to request a token from the Amazon STS service.

Tokens Expire before job completes

The default duration of session and role tokens as set in fs.s3a.assumed.role.session.duration is one hour, “1h”.

For session tokens, this can be increased to any time up to 36 hours.

For role tokens, it can be increased up to 12 hours, but only if the role is configured in the AWS IAM Console to have a longer lifespan.

Error DelegationTokenIOException: Token mismatch

org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException:
 Token mismatch: expected token for s3a://example-bucket
 of type S3ADelegationToken/Session but got a token of type S3ADelegationToken/Full

  at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.lookupToken(S3ADelegationTokens.java:379)
  at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.selectTokenFromActiveUser(S3ADelegationTokens.java:300)
  at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.bindToExistingDT(S3ADelegationTokens.java:160)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:423)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:265)

The value of fs.s3a.delegation.token.binding is different in the remote service than in the local client. As a result, the remote service cannot use the token supplied by the client to authenticate.

Fix: reference the same token binding class at both ends.

Warning Forwarding existing session credentials

This message is printed when an S3A filesystem instance has been asked for a Session Delegation Token, and it is itself only authenticated with a set of AWS session credentials (such as those issued by the IAM metadata service).

The created token will contain these existing credentials, credentials which can be used until the existing session expires.

The duration of this existing session is unknown: the message is warning you that it may expire without warning.

Error Cannot issue S3A Role Delegation Tokens without full AWS credentials

An S3A filesystem instance has been asked for a Role Delegation Token, but the instance is only authenticated with session tokens. This means that a set of role tokens cannot be requested.

Note: no attempt is made to convert the existing set of session tokens into a delegation token, unlike the Session Delegation Tokens. This is because the role of the current session (if any) is unknown.

Architecture

Concepts:

1. The S3A FileSystem can create delegation tokens when requested.
2. These can be marshalled as per other Hadoop Delegation Tokens.
3. At the far end, they can be retrieved, unmarshalled and used to authenticate callers.
4. DT binding plugins can then use these directly, or, somehow, manage authentication and token issue through other services (for example: Kerberos)
5. Token Renewal and Revocation are not supported.

There’s support for different back-end token bindings through the org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokenManager

Every implementation of this must return a subclass of org.apache.hadoop.fs.s3a.auth.delegation.AbstractS3ATokenIdentifier when asked to create a delegation token; this subclass must be registered in META-INF/services/org.apache.hadoop.security.token.TokenIdentifier for unmarshalling.

This identifier must contain all information needed at the far end to authenticate the caller with AWS services used by the S3A client: AWS S3 and potentially AWS KMS (for SSE-KMS).

It must have its own unique Token Kind, to ensure that it can be distinguished from the other token identifiers when tokens are being unmarshalled.

If implementing an external binding:

1. Follow the security requirements below.
2. Define a new token identifier; there is no requirement for the S3ADelegationToken/ prefix —but it is useful for debugging.
3. Token Renewal and Revocation is not integrated with the binding mechanism; if the operations are supported, implementation is left as an exercise.
4. Be aware of the stability guarantees of the module “LimitedPrivate/Unstable”.

Security

S3A DTs contain secrets valuable for a limited period (session secrets) or long-lived secrets with no explicit time limit.

• The toString() operations on token identifiers MUST NOT print secrets; this is needed to keep them out of logs.
• Secrets MUST NOT be logged, even at debug level.
• Prefer short-lived session secrets over long-term secrets.
• Try to restrict the permissions to what a client with the delegated token may perform to those needed to access data in the S3 bucket.
• Implementations need to be resistant to attacks which pass in invalid data as their token identifier: validate the types of the unmarshalled data; set limits on the size of all strings and other arrays to read in, etc.

Resilience

Implementations need to handle transient failures of any remote authentication service, and the risk of a large-cluster startup overloading it.

• All get/renew/cancel operations should be considered idempotent.
• And clients to repeat with backoff & jitter on recoverable connectivity failures.
• While failing fast on the unrecoverable failures (DNS, authentication).

Scalability limits of AWS STS service

There is currently no documented rate limit for token requests against the AWS STS service.

We have two tests which attempt to generate enough requests for delegation tokens that the AWS STS service will throttle requests for tokens by that AWS account for that specific STS endpoint (ILoadTestRoleCredentials and ILoadTestSessionCredentials).

In the initial results of these tests:

• A few hundred requests a second can be made before STS block the caller.
• The throttling does not last very long (seconds)
• It does not appear to affect any other STS endpoints.

If developers wish to experiment with these tests and provide more detailed analysis, we would welcome this. Do bear in mind that all users of the same AWS account in that region will be throttled. Your colleagues may notice, especially if the applications they are running do not retry on throttle responses from STS (it’s not a common occurrence after all…).

Steps

1. Come up with a token “Kind”; a unique name for the delegation token identifier.
2. Implement a subclass of AbstractS3ATokenIdentifier which adds all information which is marshalled from client to remote services. This must subclass the Writable methods to read and write the data to a data stream: these subclasses must call the superclass methods first.
3. Add a resource META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
4. And list in it, the classname of your new identifier.
5. Implement a subclass of AbstractDelegationTokenBinding

Implementing AbstractS3ATokenIdentifier

Look at the other examples to see what to do; SessionTokenIdentifier does most of the work.

Having a toString() method which is informative is ideal for the hdfs creds command as well as debugging: but do not print secrets.

Important: Add no references to any AWS SDK class, to ensure it can be safely deserialized whenever the relevant token identifier is examined. Best practise is: avoid any references to classes which may not be on the classpath of core Hadoop services, especially the YARN Resource Manager and Node Managers.

AWSCredentialProviderList deployUnbonded()

1. Perform all initialization needed on an “unbonded” deployment to authenticate with the store.
2. Return a list of AWS Credential providers which can be used to authenticate the caller.

Tip: consider not doing all the checks to verify that DTs can be issued. That can be postponed until a DT is issued -as in any deployments where a DT is not actually needed, failing at this point is overkill. As an example, RoleTokenBinding cannot issue DTs if it only has a set of session credentials, but it will deploy without them, so allowing hadoop fs commands to work on an EC2 VM with IAM role credentials.

Tip: The class org.apache.hadoop.fs.s3a.auth.MarshalledCredentials holds a set of marshalled credentials and so can be used within your own Token Identifier if you want to include a set of full/session AWS credentials in your token identifier.

AWSCredentialProviderList bindToTokenIdentifier(AbstractS3ATokenIdentifier id)

The identifier passed in will be the one for the current filesystem URI and of your token kind.

1. Use convertTokenIdentifier to cast it to your DT type, or fail with a meaningful IOException.
2. Extract the secrets needed to authenticate with the object store (or whatever service issues object store credentials).
3. Return a list of AWS Credential providers which can be used to authenticate the caller with the extracted secrets.

AbstractS3ATokenIdentifier createEmptyIdentifier()

Return an empty instance of your token identifier.

AbstractS3ATokenIdentifier createTokenIdentifier(Optional<RoleModel.Policy> policy, EncryptionSecrets secrets)

Create the delegation token.

If non-empty, the policy argument contains an AWS policy model to grant access to:

• The target S3 bucket.
• KMS key "kms:GenerateDataKey and kms:Decryptpermissions for all KMS keys.

This can be converted to a string and passed to the AWS assumeRole operation.

The secrets argument contains encryption policy and secrets: this should be passed to the superclass constructor as is; it is retrieved and used to set the encryption policy on the newly created filesystem.

Tip: Use AbstractS3ATokenIdentifier.createDefaultOriginMessage() to create an initial message for the origin of the token —this is useful for diagnostics.

Testing

Use the tests org.apache.hadoop.fs.s3a.auth.delegation as examples. You’ll have to copy and paste some of the test base classes over; hadoop-common’s test JAR is published to Maven Central, but not the S3A one (a fear of leaking AWS credentials).