The submitter of any patch is required to run all the integration tests and declare which S3 region/implementation they used.

This is important: patches which do not include this declaration will be ignored

This policy has proven to be the only mechanism to guarantee full regression testing of code changes. Why the declaration of region? Two reasons

1. It helps us identify regressions which only surface against specific endpoints or third-party implementations of the S3 protocol.
2. It forces the submitters to be more honest about their testing. It’s easy to lie, “yes, I tested this”. To say “yes, I tested this against S3 US-west” is a more specific lie and harder to make. And, if you get caught out: you lose all credibility with the project.

You don’t need to test from a VM within the AWS infrastructure; with the -Dparallel=tests option the non-scale tests complete in under ten minutes. Because the tests clean up after themselves, they are also designed to be low cost. It’s neither hard nor expensive to run the tests; if you can’t, there’s no guarantee your patch works. The reviewers have enough to do, and don’t have the time to do these tests, especially as every failure will simply make for a slow iterative development.

Please: run the tests. And if you don’t, we are sorry for declining your patch, but we have to.

What if there’s an intermittent failure of a test?

Some of the tests do fail intermittently, especially in parallel runs. If this happens, try to run the test on its own to see if the test succeeds.

If it still fails, include this fact in your declaration. We know some tests are intermittently unreliable.

What if the tests are timing out or failing over my network connection?

The tests and the S3A client are designed to be configurable for different timeouts. If you are seeing problems and this configuration isn’t working, that’s a sign of the configuration mechanism isn’t complete. If it’s happening in the production code, that could be a sign of a problem which may surface over long-haul connections. Please help us identify and fix these problems — especially as you are the one best placed to verify the fixes work.

File core-site.xml

This file pre-exists and sources the configurations created under auth-keys.xml.

For most purposes you will not need to edit this file unless you need to apply a specific, non-default property change during the tests.

File auth-keys.xml

The presence of this file triggers the testing of the S3 classes.

Without this file, none of the integration tests in this module will be executed.

The XML file must contain all the ID/key information needed to connect each of the filesystem clients to the object stores, and a URL for each filesystem for its testing.

1. test.fs.s3a.name : the URL of the bucket for S3a tests
2. fs.contract.test.fs.s3a : the URL of the bucket for S3a filesystem contract tests

The contents of the bucket will be destroyed during the test process: do not use the bucket for any purpose other than testing. Furthermore, for s3a, all in-progress multi-part uploads to the bucket will be aborted at the start of a test (by forcing fs.s3a.multipart.purge=true) to clean up the temporary state of previously failed tests.

Example:

<configuration>

  <property>
    <name>test.fs.s3a.name</name>
    <value>s3a://test-aws-s3a/</value>
  </property>

  <property>
    <name>fs.contract.test.fs.s3a</name>
    <value>${test.fs.s3a.name}</value>
  </property>

  <property>
    <name>fs.s3a.access.key</name>
    <description>AWS access key ID. Omit for IAM role-based authentication.</description>
    <value>DONOTCOMMITTHISKEYTOSCM</value>
  </property>

  <property>
    <name>fs.s3a.secret.key</name>
    <description>AWS secret key. Omit for IAM role-based authentication.</description>
    <value>DONOTEVERSHARETHISSECRETKEY!</value>
  </property>

  <property>
    <name>test.sts.endpoint</name>
    <description>Specific endpoint to use for STS requests.</description>
    <value>sts.amazonaws.com</value>
  </property>

</configuration>

Configuring S3a Encryption

For S3a encryption tests to run correctly, the fs.s3a.server-side-encryption.key must be configured in the s3a contract xml file with a AWS KMS encryption key arn as this value is different for each AWS KMS.

Example:

<property>
  <name>fs.s3a.server-side-encryption.key</name>
  <value>arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01</value>
</property>

You can also force all the tests to run with a specific SSE encryption method by configuring the property fs.s3a.server-side-encryption-algorithm in the s3a contract file.

Testing against different regions

S3A can connect to different regions —the tests support this. Simply define the target region in auth-keys.xml.

<property>
  <name>fs.s3a.endpoint</name>
  <value>s3.eu-central-1.amazonaws.com</value>
</property>

This is used for all tests expect for scale tests using a Public CSV.gz file (see below)

CSV Data Tests

The TestS3AInputStreamPerformance tests require read access to a multi-MB text file. The default file for these tests is one published by amazon, s3a://landsat-pds.s3.amazonaws.com/scene_list.gz. This is a gzipped CSV index of other files which amazon serves for open use.

The path to this object is set in the option fs.s3a.scale.test.csvfile,

<property>
  <name>fs.s3a.scale.test.csvfile</name>
  <value>s3a://landsat-pds/scene_list.gz</value>
</property>

1. If the option is not overridden, the default value is used. This is hosted in Amazon’s US-east datacenter.
2. If fs.s3a.scale.test.csvfile is empty, tests which require it will be skipped.
3. If the data cannot be read for any reason then the test will fail.
4. If the property is set to a different path, then that data must be readable and “sufficiently” large.

(the reason the space or newline is needed is to add “an empty entry”; an empty <value/> would be considered undefined and pick up the default)

Of using a test file in an S3 region requiring a different endpoint value set in fs.s3a.endpoint, a bucket-specific endpoint must be defined. For the default test dataset, hosted in the landsat-pds bucket, this is:

<property>
  <name>fs.s3a.bucket.landsat-pds.endpoint</name>
  <value>s3.amazonaws.com</value>
  <description>The endpoint for s3a://landsat-pds URLs</description>
</property>

Enabling the Scale Tests

The tests are enabled if the scale property is set in the maven build this can be done regardless of whether or not the parallel test profile is used

mvn verify -Dscale

mvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8

The most bandwidth intensive tests (those which upload data) always run sequentially; those which are slow due to HTTPS setup costs or server-side actionsare included in the set of parallelized tests.

Tuning scale optins from Maven

Some of the tests can be tuned from the maven build or from the configuration file used to run the tests.

mvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dfs.s3a.scale.test.huge.filesize=128M

The algorithm is

1. The value is queried from the configuration file, using a default value if it is not set.
2. The value is queried from the JVM System Properties, where it is passed down by maven.
3. If the system property is null, an empty string, or it has the value unset, then the configuration value is used. The unset option is used to work round a quirk in maven property propagation.

Only a few properties can be set this way; more will be added.

The file and partition sizes are numeric values with a k/m/g/t/p suffix depending on the desired size. For example: 128M, 128m, 2G, 2G, 4T or even 1P.

Scale test configuration options

Some scale tests perform multiple operations (such as creating many directories).

The exact number of operations to perform is configurable in the option scale.test.operation.count

<property>
  <name>scale.test.operation.count</name>
  <value>10</value>
</property>

Larger values generate more load, and are recommended when testing locally, or in batch runs.

Smaller values results in faster test runs, especially when the object store is a long way away.

Operations which work on directories have a separate option: this controls the width and depth of tests creating recursive directories. Larger values create exponentially more directories, with consequent performance impact.

<property>
  <name>scale.test.directory.count</name>
  <value>2</value>
</property>

DistCp tests targeting S3A support a configurable file size. The default is 10 MB, but the configuration value is expressed in KB so that it can be tuned smaller to achieve faster test runs.

<property>
  <name>scale.test.distcp.file.size.kb</name>
  <value>10240</value>
</property>

S3A specific scale test properties are

fs.s3a.scale.test.huge.filesize: size in MB for “Huge file tests”.

The Huge File tests validate S3A’s ability to handle large files —the property fs.s3a.scale.test.huge.filesize declares the file size to use.

<property>
  <name>fs.s3a.scale.test.huge.filesize</name>
  <value>200M</value>
</property>

Amazon S3 handles files larger than 5GB differently than smaller ones. Setting the huge filesize to a number greater than that) validates support for huge files.

<property>
  <name>fs.s3a.scale.test.huge.filesize</name>
  <value>6G</value>
</property>

Tests at this scale are slow: they are best executed from hosts running in the cloud infrastructure where the S3 endpoint is based. Otherwise, set a large timeout in fs.s3a.scale.test.timeout

<property>
  <name>fs.s3a.scale.test.timeout</name>
  <value>432000</value>
</property>

The tests are executed in an order to only clean up created files after the end of all the tests. If the tests are interrupted, the test data will remain.

Disabling the encryption tests

If the endpoint doesn’t support server-side-encryption, these will fail. They can be turned off.

<property>
  <name>test.fs.s3a.encryption.enabled</name>
  <value>false</value>
</property>

Encryption is only used for those specific test suites with Encryption in their classname.

Configuring the CSV file read tests**

To test on alternate infrastructures supporting the same APIs, the option fs.s3a.scale.test.csvfile must either be set to " ", or an object of at least 10MB is uploaded to the object store, and the fs.s3a.scale.test.csvfile option set to its path.

<property>
  <name>fs.s3a.scale.test.csvfile</name>
  <value> </value>
</property>

(yes, the space is necessary. The Hadoop Configuration class treats an empty value as “do not override the default”).

Testing Session Credentials

The test TestS3ATemporaryCredentials requests a set of temporary credentials from the STS service, then uses them to authenticate with S3.

If an S3 implementation does not support STS, then the functional test cases must be disabled:

<property>
  <name>test.fs.s3a.sts.enabled</name>
  <value>false</value>
</property>

These tests reqest a temporary set of credentials from the STS service endpoint. An alternate endpoint may be defined in test.fs.s3a.sts.endpoint.

<property>
  <name>test.fs.s3a.sts.endpoint</name>
  <value>https://sts.example.org/</value>
</property>

The default is ""; meaning “use the amazon default value”.

Subclasses Existing Shared Base Classes

Extend AbstractS3ATestBase or AbstractSTestS3AHugeFiles unless justifiable. These set things up for testing against the object stores, provide good threadnames, help generate isolated paths, and for AbstractSTestS3AHugeFiles subclasses, only run if -Dscale is set.

Key features of AbstractS3ATestBase

• getFileSystem() returns the S3A Filesystem bonded to the contract test Filesystem defined in fs.s3a.contract.test
• will automatically skip all tests if that URL is unset.
• Extends AbstractFSContractTestBase and Assert for all their methods.

Having shared base classes may help reduce future maintenance too. Please use them/

Secure

Don’t ever log credentials. The credential tests go out of their way to not provide meaningful logs or assertion messages precisely to avoid this.

Efficient of Time and Money

This means efficient in test setup/teardown, and, ideally, making use of existing public datasets to save setup time and tester cost.

Strategies of particular note are:

1. ITestS3ADirectoryPerformance: a single test case sets up the directory tree then performs different list operations, measuring the time taken.
2. AbstractSTestS3AHugeFiles: marks the test suite as @FixMethodOrder(MethodSorters.NAME_ASCENDING) then orders the test cases such that each test case expects the previous test to have completed (here: uploaded a file, renamed a file, …). This provides for independent tests in the reports, yet still permits an ordered sequence of operations. Do note the use of Assume.assume() to detect when the preconditions for a single test case are not met, hence, the tests become skipped, rather than fail with a trace which is really a false alarm.

The ordered test case mechanism of AbstractSTestS3AHugeFiles is probably the most elegant way of chaining test setup/teardown.

Regarding reusing existing data, we tend to use the landsat archive of AWS US-East for our testing of input stream operations. This doesn’t work against other regions, or with third party S3 implementations. Thus the URL can be overridden for testing elsewhere.

Works With Other S3 Endpoints

Don’t assume AWS S3 US-East only, do allow for working with external S3 implementations. Those may be behind the latest S3 API features, not support encryption, session APIs, etc.

Works Over Long-haul Links

As well as making file size and operation counts scaleable, this includes making test timeouts adequate. The Scale tests make this configurable; it’s hard coded to ten minutes in AbstractS3ATestBase(); subclasses can change this by overriding getTestTimeoutMillis().

Equally importantly: support proxies, as some testers need them.

Provides Diagnostics and timing information

1. Give threads useful names.
2. Create logs, log things. Know that the S3AFileSystem and its input and output streams all provide useful statistics in their {{toString()}} calls; logging them is useful on its own.
3. you can use AbstractS3ATestBase.describe(format-stringm, args) here.; it adds some newlines so as to be easier to spot.
4. Use ContractTestUtils.NanoTimer to measure the duration of operations, and log the output.

Fails Meaningfully

The ContractTestUtils class contains a whole set of assertions for making statements about the expected state of a filesystem, e.g. assertPathExists(FS, path), assertPathDoesNotExists(FS, path), and others. These do their best to provide meaningful diagnostics on failures (e.g. directory listings, file status, …), so help make failures easier to understand.

At the very least, do not use assertTrue() or assertFalse() without including error messages.

Cleans Up Afterwards

Keeps costs down.

1. Do not only cleanup if a test case completes successfully; test suite teardown must do it.
2. That teardown code must check for the filesystem and other fields being null before the cleanup. Why? If test setup fails, the teardown methods still get called.

Works Reliably

We really appreciate this — you will too.

How to keep your credentials really safe

Although the auth-keys.xml file is marked as ignored in git and subversion, it is still in your source tree, and there’s always that risk that it may creep out.

You can avoid this by keeping your keys outside the source tree and using an absolute XInclude reference to it.

<configuration>

  <include xmlns="http://www.w3.org/2001/XInclude"
    href="file:///users/ubuntu/.auth-keys.xml" />

</configuration>

Failure Injection

Warning do not enable any type of failure injection in production. The following settings are for testing only.

One of the challenges with S3A integration tests is the fact that S3 is an eventually-consistent storage system. In practice, we rarely see delays in visibility of recently created objects both in listings (listStatus()) and when getting a single file’s metadata (getFileStatus()). Since this behavior is rare and non-deterministic, thorough integration testing is challenging.

To address this, S3A supports a shim layer on top of the AmazonS3Client class which artificially delays certain paths from appearing in listings. This is implemented in the class InconsistentAmazonS3Client.

Enabling the InconsistentAmazonS3CClient

There are two ways of enabling the InconsistentAmazonS3Client: at config-time, or programmatically. For an example of programmatic test usage, see ITestS3GuardListConsistency.

To enable the fault-injecting client via configuration, switch the S3A client to use the “Inconsistent S3 Client Factory” when connecting to S3:

<property>
  <name>fs.s3a.s3.client.factory.impl</name>
  <value>org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory</value>
</property>

The inconsistent client works by:

1. Choosing which objects will be “inconsistent” at the time the object is created or deleted.
2. When listObjects() is called, any keys that we have marked as inconsistent above will not be returned in the results (until the configured delay has elapsed). Similarly, deleted items may be added to missing results to delay the visibility of the delete.

There are two ways of choosing which keys (filenames) will be affected: By substring, and by random probability.

<property>
  <name>fs.s3a.failinject.inconsistency.key.substring</name>
  <value>DELAY_LISTING_ME</value>
</property>

<property>
  <name>fs.s3a.failinject.inconsistency.probability</name>
  <value>1.0</value>
</property>

By default, any object which has the substring “DELAY_LISTING_ME” in its key will subject to delayed visibility. For example, the path s3a://my-bucket/test/DELAY_LISTING_ME/file.txt would match this condition. To match all keys use the value “*” (a single asterisk). This is a special value: We don’t support arbitrary wildcards.

The default probability of delaying an object is 1.0. This means that all keys that match the substring will get delayed visibility. Note that we take the logical and of the two conditions (substring matches and probability random chance occurs). Here are some example configurations:

| substring | probability |  behavior                                  |
|-----------|-------------|--------------------------------------------|
|           | 0.001       | An empty <value> tag in .xml config will   |
|           |             | be interpreted as unset and revert to the  |
|           |             | default value, "DELAY_LISTING_ME"          |
|           |             |                                            |
| *         | 0.001       | 1/1000 chance of *any* key being delayed.  |
|           |             |                                            |
| delay     | 0.01        | 1/100 chance of any key containing "delay" |
|           |             |                                            |
| delay     | 1.0         | All keys containing substring "delay" ..   |

You can also configure how long you want the delay in visibility to last. The default is 5000 milliseconds (five seconds).

<property>
  <name>fs.s3a.failinject.inconsistency.msec</name>
  <value>5000</value>
</property>

Future versions of this client will introduce new failure modes, with simulation of S3 throttling exceptions the next feature under development.

Limitations of Inconsistency Injection

Although InconsistentAmazonS3Client can delay the visibility of an object or parent directory, it does not prevent the key of that object from appearing in all prefix searches. For example, if we create the following object with the default configuration above, in an otherwise empty bucket:

s3a://bucket/a/b/c/DELAY_LISTING_ME

Then the following paths will still be visible as directories (ignoring possible real-world inconsistencies):

s3a://bucket/a
s3a://bucket/a/b

Whereas getFileStatus() on the following will be subject to delayed visibility (FileNotFoundException until delay has elapsed):

s3a://bucket/a/b/c
s3a://bucket/a/b/c/DELAY_LISTING_ME

In real-life S3 inconsistency, however, we expect that all the above paths (including a and b) will be subject to delayed visiblity.

Using the InconsistentAmazonS3CClient in downstream integration tests

The inconsistent client is shipped in the hadoop-aws JAR, so it can be used in applications which work with S3 to see how they handle inconsistent directory listings.

Notes

1. If the s3guard profile is not set, then the S3Guard properties are those of the test configuration set in contract-test-options.xml or auth-keys.xml

If the s3guard profile is set, 1. The S3Guard options from maven (the dynamo and authoritative flags) overwrite any previously set in the configuration files. 1. DynamoDB will be configured to create any missing tables.

Scale Testing MetadataStore Directly

There are some scale tests that exercise Metadata Store implementations directly. These ensure that S3Guard is are robust to things like DynamoDB throttling, and compare performance for different implementations. These are included in the scale tests executed when -Dscale is passed to the maven command line.

The two S3Guard scale testse are ITestDynamoDBMetadataStoreScale and ITestLocalMetadataStoreScale. To run the DynamoDB test, you will need to define your table name and region in your test configuration. For example, the following settings allow us to run ITestDynamoDBMetadataStoreScale with artificially low read and write capacity provisioned, so we can judge the effects of being throttled by the DynamoDB service:

<property>
  <name>scale.test.operation.count</name>
  <value>10</value>
</property>
<property>
  <name>scale.test.directory.count</name>
  <value>3</value>
</property>
<property>
  <name>fs.s3a.scale.test.enabled</name>
  <value>true</value>
</property>
<property>
  <name>fs.s3a.s3guard.ddb.table</name>
  <value>my-scale-test</value>
</property>
<property>
  <name>fs.s3a.s3guard.ddb.region</name>
  <value>us-west-2</value>
</property>
<property>
  <name>fs.s3a.s3guard.ddb.table.create</name>
  <value>true</value>
</property>
<property>
  <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
  <value>10</value>
</property>
<property>
  <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
  <value>10</value>
</property>

Testing only: Local Metadata Store

There is an in-memory Metadata Store for testing.

<property>
  <name>fs.s3a.metadatastore.impl</name>
  <value>org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore</value>
</property>

This is not for use in production.