CredentialProvider API Guide

Why is it used?

There are certain deployments that are very sensitive to how sensitive tokens like passwords are stored and managed within the cluster. For instance, there may be security best practices and policies in place that require such things to never be stored in clear text, for example. Enterprise deployments may be required to use a preferred solution for managing credentials and we need a way to plug in integrations for them.

General Usage Pattern

There are numerous places within the Hadoop project and ecosystem that can leverage the credential provider API today and the number continues to grow. In general, the usage pattern consists of the same requirements and flow.

1. Provision credentials within provider specific stores. This provisioning may be accomplished through the hadoop credential command or possibly through provider specific management tools.
2. Configure the credential provider path property. The provider path property hadoop.security.credential.provider.path is a comma separated list of one or more credential provider URIs that is traversed while trying to resolve a credential alias.
   • This property may be configured within core-site.xml or a component specific configuration file that is merged with core-site.xml.
   • For command line interfaces, such as that for DistCp, the property can be added with a hadoop system property (“-D property=value”) and dynamically added to the Configuration.
3. Features or components that leverage the new Configuration.getPassword method to resolve their credentials will automatically pick up support for the credential provider API.
   • By using the same property names as are used for existing clear text passwords, this mechanism allows for the migration to credential providers while providing backward compatibility for clear text.
   • The entire credential provider path is interrogated before falling back to clear text passwords in config.
4. Features or components that do not use the hadoop Configuration class for config or have other internal uses for the credential providers may choose to write to the CredentialProvider API itself. An example of its use will be included in this document but may also be found within Configuration.getPassword and within the unit tests of features that have added support and need to provision credentials for the tests.

Provision Credentials

Example: ssl.server.keystore.password

    hadoop credential create ssl.server.keystore.password -value 123
      -provider localjceks://file/home/lmccay/aws.jceks

Note that the alias names are the same as the configuration properties that were used to get the credentials from the Configuration.get method. Reusing these names allows for intuitive migration to the use of credential providers and fall back logic for backward compatibility.

Configuring the Provider Path

Now, we need to make sure that this provisioned credential store is known at runtime by the Configuration.getPassword method. If there is no credential provider path configuration then getPassword will skip the credential provider API interrogation. So, it is important that the following be configured within core-site.xml or your component’s equivalent.

    <property>
      <name>hadoop.security.credential.provider.path</name>
      <value>localjceks://file/home/lmccay/aws.jceks</value>
      <description>Path to interrogate for protected credentials.</description>
    </property>

A couple additional things to note about the provider path:

1. The scheme is used to indicate the type of provider in the above case the localjceks provider does not have a dependency on the Hadoop fs abstraction and is needed sometimes to avoid a recursive dependency. Another provider represented by jceks, does use the Hadoop fs abstraction and therefore has support for keystores provisioned within HDFS. A third provider type is the user type. This provider can manage credentials stored within the Credentials file for a process.
2. The path configuration accepts a comma separated path of providers or credential stores. The Configuration.getPassword method will walk through all of the providers until it resolves the alias or exhausts the list. Depending on the runtime needs for credentials, we may need to configure a chain of providers to check.

In summary, first, provision the credentials into a provider then configure the provider for use by a feature or component and it will often just be picked up through the use of the Configuration.getPassword method.

Supported Features

Options

Mechanics

Extremely important to consider that all of the runtime consumers of the credential being protected (mapreduce jobs/applications) will need to have access to the password used to protect the keystore providers. Communicating this password can be done a number of ways and they are described in the Options section above.

The use of the default password means that no additional communication/synchronization to runtime consumers needs to be done. The default password is known but file permissions are the primary protection of the keystore.

When file permissions are thwarted, unlike “side files”, there are no standard tools that can expose the protected credentials - even with the password known. Keytool requires a password that is six characters or more and doesn’t know how to retrieve general secrets from a keystore. It is also limited to PKI keypairs. Editors will not review the secrets stored within the keystore, nor will cat, more or any other standard tools. This is why the keystore providers are better than “side file” storage of credentials.

That said, it is trivial for someone to write code to access the credentials stored within a keystore-based credential provider using the API. Again, when using the default password, the password is merely a formality of persisting the keystore. The only protection is file permissions and OS level access policy.

Users may decide to use a password “side file” to store the password for the keystores themselves and this is supported. It is just really important to be aware of the mechanics required for this level of correctness.