S3

Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.

Getting started

S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components.

You can start a docker based cluster, including the S3 gateway from the release package.

Go to the compose/ozones3 directory, and start the server:

docker-compose up -d

You can access the S3 gateway at http://localhost:9878

URL Schema

Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname)

By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your ozone-site.xml:

<property>
   <name>ozone.s3g.domain.name</name>
   <value>s3g.internal</value>
</property>

Bucket browser

Bucket’s could be browsed from the browser with adding ?browser=true to the bucket URL.

For example the content of the ‘testbucket’ could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true

Implemented REST endpoints

Operations on S3Gateway service:

Endpoint Status
GET service implemented

Operations on Bucket:

Endpoint Status Notes
GET Bucket (List Objects) Version 2 implemented
HEAD Bucket implemented
DELETE Bucket implemented
PUT Bucket (Create bucket) implemented
Delete Multiple Objects (POST) implemented

Operation on Objects:

Endpoint Status Notes
PUT Object implemented
GET Object implemented Range headers are not supported
Multipart Uplad not implemented
DELETE Object implemented
HEAD Object implemented

Security

Security is not yet implemented, you can use any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

Note: Ozone has a notion for ‘volumes’ which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are maped to Ozone ‘volume/bucket’ locations (depending from the given authentication information).

To show the storage location of a S3 bucket, use the ozone sh bucket path <bucketname> command.

aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1

ozone sh bucket path bucket1
Volume name for S3Bucket is : s3thisisakey
Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey

Clients

AWS Cli

aws CLI could be used with specifying the custom REST endpoint.

aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest

Or

aws s3 ls --endpoint http://localhost:9878 s3://buckettest

S3 Fuse driver (goofys)

Goofys is a S3 FUSE driver. It could be used to any mount any Ozone bucket as posix file system:

goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1