Operations

• HTTP GET
  • OPEN (see FileSystem.open)
  • GETFILESTATUS (see FileSystem.getFileStatus)
  • LISTSTATUS (see FileSystem.listStatus)
  • GETCONTENTSUMMARY (see FileSystem.getContentSummary)
  • GETFILECHECKSUM (see FileSystem.getFileChecksum)
  • GETHOMEDIRECTORY (see FileSystem.getHomeDirectory)
  • GETDELEGATIONTOKEN (see FileSystem.getDelegationToken)
  • GETDELEGATIONTOKENS (see FileSystem.getDelegationTokens)
• HTTP PUT
  • CREATE (see FileSystem.create)
  • MKDIRS (see FileSystem.mkdirs)
  • CREATESYMLINK (see FileContext.createSymlink)
  • RENAME (see FileSystem.rename)
  • SETREPLICATION (see FileSystem.setReplication)
  • SETOWNER (see FileSystem.setOwner)
  • SETPERMISSION (see FileSystem.setPermission)
  • SETTIMES (see FileSystem.setTimes)
  • RENEWDELEGATIONTOKEN (see FileSystem.renewDelegationToken)
  • CANCELDELEGATIONTOKEN (see FileSystem.cancelDelegationToken)
• HTTP POST
  • APPEND (see FileSystem.append)
• HTTP DELETE
  • DELETE (see FileSystem.delete)

FileSystem URIs vs HTTP URLs

The FileSystem scheme of WebHDFS is "webhdfs://". A WebHDFS FileSystem URI has the following format.

  webhdfs://<HOST>:<HTTP_PORT>/<PATH>

The above WebHDFS URI corresponds to the below HDFS URI.

  hdfs://<HOST>:<RPC_PORT>/<PATH>

In the REST API, the prefix "/webhdfs/v1" is inserted in the path and a query is appended at the end. Therefore, the corresponding HTTP URL has the following format.

  http://<HOST>:<HTTP_PORT>/webhdfs/v1/<PATH>?op=...

HDFS Configuration Options

Below are the HDFS configuration options for WebHDFS.

Create and Write to a File

• Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE
                      [&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>]
                      [&permission=<OCTAL>][&buffersize=<INT>]"

  The request is redirected to a datanode where the file data is to be written:

  HTTP/1.1 307 TEMPORARY_REDIRECT
  Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE...
  Content-Length: 0

• Step 2: Submit another HTTP PUT request using the URL in the Location header with the file data to be written.

  curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE..."

  The client receives a 201 Created response with zero content length and the WebHDFS URI of the file in the Location header:

  HTTP/1.1 201 Created
  Location: webhdfs://<HOST>:<PORT>/<PATH>
  Content-Length: 0

Note that the reason of having two-step create/append is for preventing clients to send out data before the redirect. This issue is addressed by the "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3. Unfortunately, there are software library bugs (e.g. Jetty 6 HTTP server and Java 6 HTTP client), which do not correctly implement "Expect: 100-continue". The two-step create/append is a temporary workaround for the software library bugs.

See also: overwrite, blocksize, replication, permission, buffersize, FileSystem.create

Append to a File

• Step 1: Submit a HTTP POST request without automatically following redirects and without sending the file data.

  curl -i -X POST "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=APPEND[&buffersize=<INT>]"

  The request is redirected to a datanode where the file data is to be appended:

  HTTP/1.1 307 TEMPORARY_REDIRECT
  Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND...
  Content-Length: 0

• Step 2: Submit another HTTP POST request using the URL in the Location header with the file data to be appended.

  curl -i -X POST -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND..."

  The client receives a response with zero content length:

  HTTP/1.1 200 OK
  Content-Length: 0

See the note in the previous section for the description of why this operation requires two steps.

See also: buffersize, FileSystem.append

Open and Read a File

• Submit a HTTP GET request with automatically following redirects.

  curl -i -L "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN
                      [&offset=<LONG>][&length=<LONG>][&buffersize=<INT>]"

  The request is redirected to a datanode where the file data can be read:

  HTTP/1.1 307 TEMPORARY_REDIRECT
  Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN...
  Content-Length: 0

  The client follows the redirect to the datanode and receives the file data:

  HTTP/1.1 200 OK
  Content-Type: application/octet-stream
  Content-Length: 22
  
  Hello, webhdfs user!

See also: offset, length, buffersize, FileSystem.open

Make a Directory

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/<PATH>?op=MKDIRS[&permission=<OCTAL>]"

  The client receives a response with a boolean JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {"boolean": true}

See also: permission, FileSystem.mkdirs

Create a Symbolic Link

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/<PATH>?op=CREATESYMLINK
                                &destination=<PATH>[&createParent=<true|false>]"

  The client receives a response with zero content length:

  HTTP/1.1 200 OK
  Content-Length: 0

See also: destination, createParent, FileSystem.createSymlink

Rename a File/Directory

• Submit a HTTP PUT request.

  curl -i -X PUT "<HOST>:<PORT>/webhdfs/v1/<PATH>?op=RENAME&destination=<PATH>"

  The client receives a response with a boolean JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {"boolean": true}

See also: destination, FileSystem.rename

Delete a File/Directory

• Submit a HTTP DELETE request.

  curl -i -X DELETE "http://<host>:<port>/webhdfs/v1/<path>?op=DELETE
                                [&recursive=<true|false>]"

  The client receives a response with a boolean JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {"boolean": true}

See also: recursive, FileSystem.delete

Status of a File/Directory

• Submit a HTTP GET request.

  curl -i  "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETFILESTATUS"

  The client receives a response with a FileStatus JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {
    "FileStatus":
    {
      "accessTime"      : 0,
      "blockSize"       : 0,
      "group"           : "supergroup",
      "length"          : 0,             //in bytes, zero for directories
      "modificationTime": 1320173277227,
      "owner"           : "webuser",
      "pathSuffix"      : "",
      "permission"      : "777",
      "replication"     : 0,
      "type"            : "DIRECTORY"    //enum {FILE, DIRECTORY, SYMLINK}
    }
  }

See also: FileSystem.getFileStatus

List a Directory

• Submit a HTTP GET request.

  curl -i  "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=LISTSTATUS"

  The client receives a response with a FileStatuses JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Content-Length: 427
  
  {
    "FileStatuses":
    {
      "FileStatus":
      [
        {
          "accessTime"      : 1320171722771,
          "blockSize"       : 33554432,
          "group"           : "supergroup",
          "length"          : 24930,
          "modificationTime": 1320171722771,
          "owner"           : "webuser",
          "pathSuffix"      : "a.patch",
          "permission"      : "644",
          "replication"     : 1,
          "type"            : "FILE"
        },
        {
          "accessTime"      : 0,
          "blockSize"       : 0,
          "group"           : "supergroup",
          "length"          : 0,
          "modificationTime": 1320895981256,
          "owner"           : "szetszwo",
          "pathSuffix"      : "bar",
          "permission"      : "711",
          "replication"     : 0,
          "type"            : "DIRECTORY"
        },
        ...
      ]
    }
  }

See also: FileSystem.listStatus

Get Content Summary of a Directory

• Submit a HTTP GET request.

  curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETCONTENTSUMMARY"

  The client receives a response with a ContentSummary JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {
    "ContentSummary":
    {
      "directoryCount": 2,
      "fileCount"     : 1,
      "length"        : 24930,
      "quota"         : -1,
      "spaceConsumed" : 24930,
      "spaceQuota"    : -1
    }
  }

See also: FileSystem.getContentSummary

Get File Checksum

• Submit a HTTP GET request.

  curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETFILECHECKSUM"

  The request is redirected to a datanode:

  HTTP/1.1 307 TEMPORARY_REDIRECT
  Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=GETFILECHECKSUM...
  Content-Length: 0

  The client follows the redirect to the datanode and receives a FileChecksum JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {
    "FileChecksum":
    {
      "algorithm": "MD5-of-1MD5-of-512CRC32",
      "bytes"    : "eadb10de24aa315748930df6e185c0d ...",
      "length"   : 28
    }
  }

See also: FileSystem.getFileChecksum

Get Home Directory

• Submit a HTTP GET request.

  curl -i "http://<HOST>:<PORT>/webhdfs/v1/?op=GETHOMEDIRECTORY"

  The client receives a response with a Path JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {"Path": "/user/szetszwo"}

See also: FileSystem.getHomeDirectory

Set Permission

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETPERMISSION
                                [&permission=<OCTAL>]"

  The client receives a response with zero content length:

  HTTP/1.1 200 OK
  Content-Length: 0

See also: permission, FileSystem.setPermission

Set Owner

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETOWNER
                                [&owner=<USER>][&group=<GROUP>]"

  The client receives a response with zero content length:

  HTTP/1.1 200 OK
  Content-Length: 0

See also: owner, group, FileSystem.setOwner

Set Replication Factor

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETREPLICATION
                                [&replication=<SHORT>]"

  The client receives a response with a boolean JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {"boolean": true}

See also: replication, FileSystem.setReplication

Set Access or Modification Time

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETTIMES
                                [&modificationtime=<TIME>][&accesstime=<TIME>]"

  The client receives a response with zero content length:

  HTTP/1.1 200 OK
  Content-Length: 0

See also: modificationtime, accesstime, FileSystem.setTimes

Get Delegation Token

• Submit a HTTP GET request.

  curl -i "http://<HOST>:<PORT>/webhdfs/v1/?op=GETDELEGATIONTOKEN&renewer=<USER>"

  The client receives a response with a Token JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {
    "Token":
    {
      "urlString": "JQAIaG9y..."
    }
  }

See also: renewer, FileSystem.getDelegationToken

Get Delegation Tokens

• Submit a HTTP GET request.

  curl -i "http://<HOST>:<PORT>/webhdfs/v1/?op=GETDELEGATIONTOKENS&renewer=<USER>"

  The client receives a response with a Tokens JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {
    "Tokens":
    {
      "Token":
      [
        {
          "urlString":"KAAKSm9i ..."
        }
      ]
    }
  }

See also: renewer, FileSystem.getDelegationTokens

Renew Delegation Token

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/?op=RENEWDELEGATIONTOKEN&token=<TOKEN>"

  The client receives a response with a long JSON object:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  
  {"long": 1320962673997}           //the new expiration time

See also: token, FileSystem.renewDelegationToken

Cancel Delegation Token

• Submit a HTTP PUT request.

  curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/?op=CANCELDELEGATIONTOKEN&token=<TOKEN>"

  The client receives a response with zero content length:

  HTTP/1.1 200 OK
  Content-Length: 0

See also: token, FileSystem.cancelDelegationToken

HTTP Response Codes

Below are examples of exception responses.

HTTP/1.1 400 Bad Request
Content-Type: application/json
Transfer-Encoding: chunked

{
  "RemoteException":
  {
    "exception"    : "IllegalArgumentException",
    "javaClassName": "java.lang.IllegalArgumentException",
    "message"      : "Invalid value for webhdfs parameter \"permission\": ..."
  }
}

HTTP/1.1 401 Unauthorized
Content-Type: application/json
Transfer-Encoding: chunked

{
  "RemoteException":
  {
    "exception"    : "SecurityException",
    "javaClassName": "java.lang.SecurityException",
    "message"      : "Failed to obtain user group information: ..."
  }
}

HTTP/1.1 403 Forbidden
Content-Type: application/json
Transfer-Encoding: chunked

{
  "RemoteException":
  {
    "exception"    : "AccessControlException",
    "javaClassName": "org.apache.hadoop.security.AccessControlException",
    "message"      : "Permission denied: ..."
  }
}

HTTP/1.1 404 Not Found
Content-Type: application/json
Transfer-Encoding: chunked

{
  "RemoteException":
  {
    "exception"    : "FileNotFoundException",
    "javaClassName": "java.io.FileNotFoundException",
    "message"      : "File does not exist: /foo/a.patch"
  }
}

Boolean JSON Schema

{
  "name"      : "boolean",
  "properties":
  {
    "boolean":
    {
      "description": "A boolean value",
      "type"       : "boolean",
      "required"   : true
    }
  }
}

See also: MKDIRS, RENAME, DELETE, SETREPLICATION

ContentSummary JSON Schema

{
  "name"      : "ContentSummary",
  "properties":
  {
    "ContentSummary":
    {
      "type"      : "object",
      "properties":
      {
        "directoryCount":
        {
          "description": "The number of directories.",
          "type"       : "integer",
          "required"   : true
        },
        "fileCount":
        {
          "description": "The number of files.",
          "type"       : "integer",
          "required"   : true
        },
        "length":
        {
          "description": "The number of bytes used by the content.",
          "type"       : "integer",
          "required"   : true
        },
        "quota":
        {
          "description": "The namespace quota of this directory.",
          "type"       : "integer",
          "required"   : true
        },
        "spaceConsumed":
        {
          "description": "The disk space consumed by the content.",
          "type"       : "integer",
          "required"   : true
        },
        "spaceQuota":
        {
          "description": "The disk space quota.",
          "type"       : "integer",
          "required"   : true
        }
      }
    }
  }
}

See also: GETCONTENTSUMMARY

FileChecksum JSON Schema

{
  "name"      : "FileChecksum",
  "properties":
  {
    "FileChecksum":
    {
      "type"      : "object",
      "properties":
      {
        "algorithm":
        {
          "description": "The name of the checksum algorithm.",
          "type"       : "string",
          "required"   : true
        },
        "bytes":
        {
          "description": "The byte sequence of the checksum in hexadecimal.",
          "type"       : "string",
          "required"   : true
        },
        "length":
        {
          "description": "The length of the bytes (not the length of the string).",
          "type"       : "integer",
          "required"   : true
        }
      }
    }
  }
}

See also: GETFILECHECKSUM

FileStatus JSON Schema

{
  "name"      : "FileStatus",
  "properties":
  {
    "FileStatus": fileStatusProperties      //See FileStatus Properties
  }
}

See also: FileStatus Properties, GETFILESTATUS, FileStatus

var fileStatusProperties =
{
  "type"      : "object",
  "properties":
  {
    "accessTime":
    {
      "description": "The access time.",
      "type"       : "integer",
      "required"   : true
    },
    "blockSize":
    {
      "description": "The block size of a file.",
      "type"       : "integer",
      "required"   : true
    },
    "group":
    {
      "description": "The group owner.",
      "type"       : "string",
      "required"   : true
    },
    "length":
    {
      "description": "The number of bytes in a file.",
      "type"       : "integer",
      "required"   : true
    },
    "modificationTime":
    {
      "description": "The modification time.",
      "type"       : "integer",
      "required"   : true
    },
    "owner":
    {
      "description": "The user who is the owner.",
      "type"       : "string",
      "required"   : true
    },
    "pathSuffix":
    {
      "description": "The path suffix.",
      "type"       : "string",
      "required"   : true
    },
    "permission":
    {
      "description": "The permission represented as a octal string.",
      "type"       : "string",
      "required"   : true
    },
    "replication":
    {
      "description": "The number of replication of a file.",
      "type"       : "integer",
      "required"   : true
    },
   "symlink":                                         //an optional property
    {
      "description": "The link target of a symlink.",
      "type"       : "string"
    },
   "type":
    {
      "description": "The type of the path object.",
      "enum"       : ["FILE", "DIRECTORY", "SYMLINK"],
      "required"   : true
    }
  }
};

FileStatuses JSON Schema

A FileStatuses JSON object represents an array of FileStatus JSON objects.

{
  "name"      : "FileStatuses",
  "properties":
  {
    "FileStatuses":
    {
      "type"      : "object",
      "properties":
      {
        "FileStatus":
        {
          "description": "An array of FileStatus",
          "type"       : "array",
          "items"      : fileStatusProperties      //See FileStatus Properties
        }
      }
    }
  }
}

See also: FileStatus Properties, LISTSTATUS, FileStatus

Long JSON Schema

{
  "name"      : "long",
  "properties":
  {
    "long":
    {
      "description": "A long integer value",
      "type"       : "integer",
      "required"   : true
    }
  }
}

See also: RENEWDELEGATIONTOKEN,

Path JSON Schema

{
  "name"      : "Path",
  "properties":
  {
    "Path":
    {
      "description": "The string representation a Path.",
      "type"       : "string",
      "required"   : true
    }
  }
}

See also: GETHOMEDIRECTORY, Path

RemoteException JSON Schema

{
  "name"      : "RemoteException",
  "properties":
  {
    "RemoteException":
    {
      "type"      : "object",
      "properties":
      {
        "exception":
        {
          "description": "Name of the exception",
          "type"       : "string",
          "required"   : true
        },
        "message":
        {
          "description": "Exception message",
          "type"       : "string",
          "required"   : true
        },
        "javaClassName":                                     //an optional property
        {
          "description": "Java class name of the exception",
          "type"       : "string",
        }
      }
    }
  }
}

See also: Error Responses

Token JSON Schema

{
  "name"      : "Token",
  "properties":
  {
    "Token": tokenProperties      //See Token Properties
  }
}

See also: Token Properties, GETDELEGATIONTOKEN, the note in Delegation.

var tokenProperties =
{
  "type"      : "object",
  "properties":
  {
    "urlString":
    {
      "description": "A delegation token encoded as a URL safe string.",
      "type"       : "string",
      "required"   : true
    }
  }
}

Tokens JSON Schema

A Tokens JSON object represents an array of Token JSON objects.

{
  "name"      : "Tokens",
  "properties":
  {
    "Tokens":
    {
      "type"      : "object",
      "properties":
      {
        "Token":
        {
          "description": "An array of Token",
          "type"       : "array",
          "items"      : "Token": tokenProperties      //See Token Properties
        }
      }
    }
  }
}

See also: Token Properties, GETDELEGATIONTOKENS, the note in Delegation.

Access Time

See also: SETTIMES

Block Size

See also: CREATE

Buffer Size

See also: CREATE, APPEND, OPEN

Create Parent

See also: CREATESYMLINK

Delegation

Note that delegation tokens are encoded as a URL safe string; see encodeToUrlString() and decodeFromUrlString(String) in org.apache.hadoop.security.token.Token for the details of the encoding.

See also: Authentication

Destination

See also: CREATESYMLINK, RENAME

Do As

See also: Proxy Users

Group

See also: SETOWNER

Length

See also: OPEN

Modification Time

See also: SETTIMES

Offset

See also: OPEN

Op

See also: Operations

Overwrite

See also: CREATE

Owner

See also: SETOWNER

Permission

See also: CREATE, MKDIRS, SETPERMISSION

Recursive

See also: RENAME

Renewer

See also: GETDELEGATIONTOKEN, GETDELEGATIONTOKENS

Replication

See also: CREATE, SETREPLICATION

Token

See also: RENEWDELEGATIONTOKEN, CANCELDELEGATIONTOKEN

Username

See also: Authentication