Hadoop HDFS over HTTP - Server Setup

This page explains how to quickly setup HttpFS with Pseudo authentication against a Hadoop cluster with Pseudo authentication.

Install HttpFS

~ $ tar xzf  httpfs-2.8.4.tar.gz

Configure HttpFS

By default, HttpFS assumes that Hadoop configuration files (core-site.xml & hdfs-site.xml) are in the HttpFS configuration directory.

If this is not the case, add to the httpfs-site.xml file the httpfs.hadoop.config.dir property set to the location of the Hadoop configuration directory.

Configure Hadoop

Edit Hadoop core-site.xml and defined the Unix user that will run the HttpFS server as a proxyuser. For example:

  <property>
    <name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>
    <value>httpfs-host.foo.com</value>
  </property>
  <property>
    <name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>
    <value>*</value>
  </property>

IMPORTANT: Replace #HTTPFSUSER# with the Unix user that will start the HttpFS server.

Restart Hadoop

You need to restart Hadoop for the proxyuser configuration to become active.

Start/Stop HttpFS

To start/stop HttpFS use HttpFS’s sbin/httpfs.sh script. For example:

$ sbin/httpfs.sh start

NOTE: Invoking the script without any parameters list all possible parameters (start, stop, run, etc.). The httpfs.sh script is a wrapper for Tomcat’s catalina.sh script that sets the environment variables and Java System properties required to run HttpFS server.

Test HttpFS is working

$ curl -sS 'http://<HTTPFSHOSTNAME>:14000/webhdfs/v1?op=gethomedirectory&user.name=hdfs'
{"Path":"\/user\/hdfs"}

Embedded Tomcat Configuration

To configure the embedded Tomcat go to the tomcat/conf.

HttpFS preconfigures the HTTP and Admin ports in Tomcat’s server.xml to 14000 and 14001.

Tomcat logs are also preconfigured to go to HttpFS’s logs/ directory.

The following environment variables (which can be set in HttpFS’s etc/hadoop/httpfs-env.sh script) can be used to alter those values:

  • HTTPFS_HTTP_PORT

  • HTTPFS_ADMIN_PORT

  • HADOOP_LOG_DIR

HttpFS Configuration

HttpFS supports the following configuration properties in the HttpFS’s etc/hadoop/httpfs-site.xml configuration file.

HttpFS over HTTPS (SSL)

To configure HttpFS to work over SSL edit the httpfs-env.sh script in the configuration directory setting the HTTPFS_SSL_ENABLED to true.

In addition, the following 2 properties may be defined (shown with default values):

  • HTTPFS_SSL_KEYSTORE_FILE=$HOME/.keystore

  • HTTPFS_SSL_KEYSTORE_PASS=password

In the HttpFS tomcat/conf directory, replace the server.xml file with the ssl-server.xml file.

You need to create an SSL certificate for the HttpFS server. As the httpfs Unix user, using the Java keytool command to create the SSL certificate:

$ keytool -genkey -alias tomcat -keyalg RSA

You will be asked a series of questions in an interactive prompt. It will create the keystore file, which will be named .keystore and located in the httpfs user home directory.

The password you enter for “keystore password” must match the value of the HTTPFS_SSL_KEYSTORE_PASS environment variable set in the httpfs-env.sh script in the configuration directory.

The answer to “What is your first and last name?” (i.e. “CN”) must be the hostname of the machine where the HttpFS Server will be running.

Start HttpFS. It should work over HTTPS.

Using the Hadoop FileSystem API or the Hadoop FS shell, use the swebhdfs:// scheme. Make sure the JVM is picking up the truststore containing the public key of the SSL certificate if using a self-signed certificate.

NOTE: Some old SSL clients may use weak ciphers that are not supported by the HttpFS server. It is recommended to upgrade the SSL client.