The hadoop-azure module provides support for integration with Azure Blob Storage. The built jar file, named hadoop-azure.jar, also declares transitive dependencies on the additional artifacts it requires, notably the Azure Storage SDK for Java.
The Azure Blob Storage data model presents 3 core concepts:
Usage of Azure Blob Storage requires configuration of credentials. Typically this is set in core-site.xml. The configuration property name is of the form fs.azure.account.key.<account name>.blob.core.windows.net and the value is the access key. The access key is a secret that protects access to your storage account. Do not share the access key (or the core-site.xml file) with an untrusted party.
<property> <name>fs.azure.account.key.youraccount.blob.core.windows.net</name> <value>YOUR ACCESS KEY</value> </property>
In many Hadoop clusters, the core-site.xml file is world-readable. If it’s undesirable for the access key to be visible in core-site.xml, then it’s also possible to configure it in encrypted form. An additional configuration property specifies an external program to be invoked by Hadoop processes to decrypt the key. The encrypted key value is passed to this external program as a command line argument:
<property> <name>fs.azure.account.keyprovider.youraccount</name> <value>org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider</value> </property> <property> <name>fs.azure.account.key.youraccount.blob.core.windows.net</name> <value>YOUR ENCRYPTED ACCESS KEY</value> </property> <property> <name>fs.azure.shellkeyprovider.script</name> <value>PATH TO DECRYPTION PROGRAM</value> </property>
The Azure Blob Storage interface for Hadoop supports two kinds of blobs, block blobs and page blobs. Block blobs are the default kind of blob and are good for most big-data use cases, like input data for Hive, Pig, analytical map-reduce jobs etc. Page blob handling in hadoop-azure was introduced to support HBase log files. Page blobs can be written any number of times, whereas block blobs can only be appended to 50,000 times before you run out of blocks and your writes will fail. That won’t work for HBase logs, so page blob support was introduced to overcome this limitation.
Page blobs can be used for other purposes beyond just HBase log files though. Page blobs can be up to 1TB in size, larger than the maximum 200GB size for block blobs.
In order to have the files you create be page blobs, you must set the configuration variable fs.azure.page.blob.dir to a comma-separated list of folder names.
<property> <name>fs.azure.page.blob.dir</name> <value>/hbase/WALs,/hbase/oldWALs,/data/mypageblobfiles</value> </property>
You can set this to simply / to make all files page blobs.
The configuration option fs.azure.page.blob.size is the default initial size for a page blob. It must be 128MB or greater, and no more than 1TB, specified as an integer number of bytes.
The configuration option fs.azure.page.blob.extension.size is the page blob extension size. This defines the amount to extend a page blob if it starts to get full. It must be 128MB or greater, specified as an integer number of bytes.
Azure storage stores files as a flat key/value store without formal support for folders. The hadoop-azure file system layer simulates folders on top of Azure storage. By default, folder rename in the hadoop-azure file system layer is not atomic. That means that a failure during a folder rename could, for example, leave some folders in the original directory and some in the new one.
HBase depends on atomic folder rename. Hence, a configuration setting was introduced called fs.azure.atomic.rename.dir that allows you to specify a comma-separated list of directories to receive special treatment so that folder rename is made atomic. The default value of this setting is just /hbase. Redo will be applied to finish a folder rename that fails. A file <folderName>-renamePending.json may appear temporarily and is the record of the intention of the rename operation, to allow redo in event of a failure.
<property> <name>fs.azure.atomic.rename.dir</name> <value>/hbase,/data</value> </property>
After credentials are configured in core-site.xml, any Hadoop component may reference files in that Azure Blob Storage account by using URLs of the following format:
The schemes wasb and wasbs identify a URL on a file system backed by Azure Blob Storage. wasb utilizes unencrypted HTTP access for all interaction with the Azure Blob Storage API. wasbs utilizes SSL encrypted HTTPS access.
For example, the following FileSystem Shell commands demonstrate access to a storage account named youraccount and a container named yourcontainer.
> hadoop fs -mkdir wasb://email@example.com/testDir > hadoop fs -put testFile wasb://firstname.lastname@example.org/testDir/testFile > hadoop fs -cat wasbs://email@example.com/testDir/testFile test file content
It’s also possible to configure fs.defaultFS to use a wasb or wasbs URL. This causes all bare paths, such as /testDir/testFile to resolve automatically to that file system.
The hadoop-azure module includes a full suite of unit tests. Most of the tests will run without additional configuration by running mvn test. This includes tests against mocked storage, which is an in-memory emulation of Azure Storage.
A selection of tests can run against the Azure Storage Emulator which is a high-fidelity emulation of live Azure Storage. The emulator is sufficient for high-confidence testing. The emulator is a Windows executable that runs on a local machine.
To use the emulator, install Azure SDK 2.3 and start the storage emulator. Then, edit src/test/resources/azure-test.xml and add the following property:
<property> <name>fs.azure.test.emulator</name> <value>true</value> </property>
There is a known issue when running tests with the emulator. You may see the following failure message:
com.microsoft.windowsazure.storage.StorageException: The value for one of the HTTP headers is not in the correct format.
To resolve this, restart the Azure Emulator. Ensure it v3.2 or later.
It’s also possible to run tests against a live Azure Storage account by adding credentials to src/test/resources/azure-test.xml and setting fs.azure.test.account.name to the name of the storage account.
<property> <name>fs.azure.account.key.youraccount.blob.core.windows.net</name> <value>YOUR ACCESS KEY</value> </property> <property> <name>fs.azure.test.account.name</name> <value>youraccount</value> </property>