Package org.apache.hadoop.fs.kfs Description
A client for the Kosmos filesystem (KFS)
Introduction
This pages describes how to use Kosmos Filesystem
(
KFS ) as a backing
store with Hadoop. This page assumes that you have downloaded the
KFS software and installed necessary binaries as outlined in the KFS
documentation.
Steps
- In the Hadoop conf directory edit core-site.xml,
add the following:
<property>
<name>fs.kfs.impl</name>
<value>org.apache.hadoop.fs.kfs.KosmosFileSystem</value>
<description>The FileSystem for kfs: uris.</description>
</property>
- In the Hadoop conf directory edit core-site.xml,
adding the following (with appropriate values for
<server> and <port>):
<property>
<name>fs.default.name</name>
<value>kfs://<server:port></value>
</property>
<property>
<name>fs.kfs.metaServerHost</name>
<value><server></value>
<description>The location of the KFS meta server.</description>
</property>
<property>
<name>fs.kfs.metaServerPort</name>
<value><port></value>
<description>The location of the meta server's port.</description>
</property>
- Copy KFS's kfs-0.1.jar to Hadoop's lib directory. This step
enables Hadoop's to load the KFS specific modules. Note
that, kfs-0.1.jar was built when you compiled KFS source
code. This jar file contains code that calls KFS's client
library code via JNI; the native code is in KFS's
libkfsClient.so library.
- When the Hadoop map/reduce trackers start up, those
processes (on local as well as remote nodes) will now need to load
KFS's libkfsClient.so library. To simplify this process, it is advisable to
store libkfsClient.so in an NFS accessible directory (similar to where
Hadoop binaries/scripts are stored); then, modify Hadoop's
conf/hadoop-env.sh adding the following line and providing suitable
value for <path>:
export LD_LIBRARY_PATH=<path>
- Start only the map/reduce trackers
example: execute Hadoop's bin/start-mapred.sh
If the map/reduce job trackers start up, all file-I/O is done to KFS.