org.apache.hadoop.filecache

Class DistributedCache

java.lang.Object

org.apache.hadoop.mapreduce.filecache.DistributedCache

org.apache.hadoop.filecache.DistributedCache

Deprecated.

@InterfaceAudience.Public
 @InterfaceStability.Stable
 @Deprecated
public class DistributedCache
extends org.apache.hadoop.mapreduce.filecache.DistributedCache

Distribute application-specific large, read-only files efficiently.

DistributedCache is a facility provided by the Map-Reduce framework to cache files (text, archives, jars etc.) needed by applications.

Applications specify the files, via urls (hdfs:// or http://) to be cached via the JobConf. The DistributedCache assumes that the files specified via urls are already present on the FileSystem at the path specified by the url and are accessible by every machine in the cluster.

The framework will copy the necessary files on to the worker node before any tasks for the job are executed on that node. Its efficiency stems from the fact that the files are only copied once per job and the ability to cache archives which are un-archived on the workers.

DistributedCache can be used to distribute simple, read-only data/text files and/or more complex types such as archives, jars etc. Archives (zip, tar and tgz/tar.gz files) are un-archived at the worker nodes. Jars may be optionally added to the classpath of the tasks, a rudimentary software distribution mechanism. Files have execution permissions. In older version of Hadoop Map/Reduce users could optionally ask for symlinks to be created in the working directory of the child task. In the current version symlinks are always created. If the URL does not have a fragment the name of the file or directory will be used. If multiple files or directories map to the same link name, the last one added, will be used. All others will not even be downloaded.

DistributedCache tracks modification timestamps of the cache files. Clearly the cache files should not be modified by the application or externally while the job is executing.

Here is an illustrative example on how to use the DistributedCache:

     // Setting up the cache for the application

     1. Copy the requisite files to the FileSystem:

     $ bin/hadoop fs -copyFromLocal lookup.dat /myapp/lookup.dat
     $ bin/hadoop fs -copyFromLocal map.zip /myapp/map.zip
     $ bin/hadoop fs -copyFromLocal mylib.jar /myapp/mylib.jar
     $ bin/hadoop fs -copyFromLocal mytar.tar /myapp/mytar.tar
     $ bin/hadoop fs -copyFromLocal mytgz.tgz /myapp/mytgz.tgz
     $ bin/hadoop fs -copyFromLocal mytargz.tar.gz /myapp/mytargz.tar.gz

     2. Setup the application's JobConf:

     JobConf job = new JobConf();
     DistributedCache.addCacheFile(new URI("/myapp/lookup.dat#lookup.dat"),
                                   job);
     DistributedCache.addCacheArchive(new URI("/myapp/map.zip"), job);
     DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar"), job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz"), job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz"), job);

     3. Use the cached files in the Mapper
     or Reducer:

     public static class MapClass extends MapReduceBase
     implements Mapper<K, V, K, V> {

       private Path[] localArchives;
       private Path[] localFiles;

       public void configure(JobConf job) {
         // Get the cached archives/files
         File f = new File("./map.zip/some/file/in/zip.txt");
       }

       public void map(K key, V value,
                       OutputCollector<K, V> output, Reporter reporter)
       throws IOException {
         // Use data from the cached archives/files here
         // ...
         // ...
         output.collect(k, v);
       }
     }

 

It is also very common to use the DistributedCache by using GenericOptionsParser. This class includes methods that should be used by users (specifically those mentioned in the example above, as well as DistributedCache.addArchiveToClassPath(Path, Configuration)), as well as methods intended for use by the MapReduce framework (e.g., JobClient).

See Also:

JobConf, JobClient, Job

Field Summary

Fields

Modifier and Type	Field and Description
static String	CACHE_ARCHIVES Deprecated.
static String	CACHE_ARCHIVES_SIZES Deprecated.
static String	CACHE_ARCHIVES_TIMESTAMPS Deprecated.
static String	CACHE_FILES Deprecated.
static String	CACHE_FILES_SIZES Deprecated.
static String	CACHE_FILES_TIMESTAMPS Deprecated.
static String	CACHE_LOCALARCHIVES Deprecated.
static String	CACHE_LOCALFILES Deprecated.
static String	CACHE_SYMLINK Deprecated.

Fields inherited from class org.apache.hadoop.mapreduce.filecache.DistributedCache

WILDCARD

Constructor Summary

Constructors

Constructor and Description
DistributedCache() Deprecated.

Method Summary

Modifier and Type	Method and Description
static void	addLocalArchives(Configuration conf, String str) Deprecated.
static void	addLocalFiles(Configuration conf, String str) Deprecated.
static void	createAllSymlink(Configuration conf, File jobCacheDir, File workDir) Deprecated. Internal to MapReduce framework. Use DistributedCacheManager instead.
static FileStatus	getFileStatus(Configuration conf, URI cache) Deprecated.
static long	getTimestamp(Configuration conf, URI cache) Deprecated.
static void	setArchiveTimestamps(Configuration conf, String timestamps) Deprecated.
static void	setFileTimestamps(Configuration conf, String timestamps) Deprecated.
static void	setLocalArchives(Configuration conf, String str) Deprecated.
static void	setLocalFiles(Configuration conf, String str) Deprecated.

Methods inherited from class org.apache.hadoop.mapreduce.filecache.DistributedCache

addArchiveToClassPath, addArchiveToClassPath, addCacheArchive, addCacheFile, addFileToClassPath, addFileToClassPath, addFileToClassPath, checkURIs, createSymlink, getArchiveClassPaths, getArchiveTimestamps, getArchiveVisibilities, getCacheArchives, getCacheFiles, getFileClassPaths, getFileTimestamps, getFileVisibilities, getLocalCacheArchives, getLocalCacheFiles, getSymlink, setCacheArchives, setCacheFiles

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CACHE_FILES_SIZES

@Deprecated
public static final String CACHE_FILES_SIZES

Deprecated.

Warning: CACHE_FILES_SIZES is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_FILES_SIZES

See Also:

Constant Field Values

CACHE_ARCHIVES_SIZES

@Deprecated
public static final String CACHE_ARCHIVES_SIZES

Deprecated.

Warning: CACHE_ARCHIVES_SIZES is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_ARCHIVES_SIZES

See Also:

Constant Field Values

CACHE_ARCHIVES_TIMESTAMPS

@Deprecated
public static final String CACHE_ARCHIVES_TIMESTAMPS

Deprecated.

Warning: CACHE_ARCHIVES_TIMESTAMPS is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_ARCHIVES_TIMESTAMPS

See Also:

Constant Field Values

CACHE_FILES_TIMESTAMPS

@Deprecated
public static final String CACHE_FILES_TIMESTAMPS

Deprecated.

Warning: CACHE_FILES_TIMESTAMPS is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_FILE_TIMESTAMPS

See Also:

Constant Field Values

CACHE_ARCHIVES

@Deprecated
public static final String CACHE_ARCHIVES

Deprecated.

Warning: CACHE_ARCHIVES is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_ARCHIVES

See Also:

Constant Field Values

CACHE_FILES

@Deprecated
public static final String CACHE_FILES

Deprecated.

Warning: CACHE_FILES is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_FILES

See Also:

Constant Field Values

CACHE_LOCALARCHIVES

@Deprecated
public static final String CACHE_LOCALARCHIVES

Deprecated.

Warning: CACHE_LOCALARCHIVES is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_LOCALARCHIVES

See Also:

Constant Field Values

CACHE_LOCALFILES

@Deprecated
public static final String CACHE_LOCALFILES

Deprecated.

Warning: CACHE_LOCALFILES is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_LOCALFILES

See Also:

Constant Field Values

CACHE_SYMLINK

@Deprecated
public static final String CACHE_SYMLINK

Deprecated.

Warning: CACHE_SYMLINK is not a *public* constant. The variable is kept for M/R 1.x applications, M/R 2.x applications should use MRJobConfig.CACHE_SYMLINK

See Also:

Constant Field Values

Constructor Detail

DistributedCache

public DistributedCache()

Deprecated.

Method Detail

addLocalArchives

@Deprecated
public static void addLocalArchives(Configuration conf,
                                                String str)

Deprecated.

Add a archive that has been localized to the conf. Used by internal DistributedCache code.

Parameters:

conf - The conf to modify to contain the localized caches

str - a comma separated list of local archives

addLocalFiles

@Deprecated
public static void addLocalFiles(Configuration conf,
                                             String str)

Deprecated.

Add a file that has been localized to the conf.. Used by internal DistributedCache code.

Parameters:

conf - The conf to modify to contain the localized caches

str - a comma separated list of local files

createAllSymlink

@Deprecated
public static void createAllSymlink(Configuration conf,
                                                File jobCacheDir,
                                                File workDir)
                                         throws IOException

Deprecated. Internal to MapReduce framework. Use DistributedCacheManager instead.

This method create symlinks for all files in a given dir in another directory. Currently symlinks cannot be disabled. This is a NO-OP.

Parameters:

conf - the configuration

jobCacheDir - the target directory for creating symlinks

workDir - the directory in which the symlinks are created

Throws:

IOException

getFileStatus

@Deprecated
public static FileStatus getFileStatus(Configuration conf,
                                                   URI cache)
                                            throws IOException

Deprecated.

Returns FileStatus of a given cache file on hdfs. Internal to MapReduce.

Parameters:

conf - configuration

cache - cache file

Returns:

FileStatus of a given cache file on hdfs

Throws:

IOException

getTimestamp

@Deprecated
public static long getTimestamp(Configuration conf,
                                            URI cache)
                                     throws IOException

Deprecated.

Returns mtime of a given cache file on hdfs. Internal to MapReduce.

Parameters:

conf - configuration

cache - cache file

Returns:

mtime of a given cache file on hdfs

Throws:

IOException

setArchiveTimestamps

@Deprecated
public static void setArchiveTimestamps(Configuration conf,
                                                    String timestamps)

Deprecated.

This is to check the timestamp of the archives to be localized. Used by internal MapReduce code.

Parameters:

conf - Configuration which stores the timestamp's

timestamps - comma separated list of timestamps of archives. The order should be the same as the order in which the archives are added.

setFileTimestamps

@Deprecated
public static void setFileTimestamps(Configuration conf,
                                                 String timestamps)

Deprecated.

This is to check the timestamp of the files to be localized. Used by internal MapReduce code.

Parameters:

conf - Configuration which stores the timestamp's

timestamps - comma separated list of timestamps of files. The order should be the same as the order in which the files are added.

setLocalArchives

@Deprecated
public static void setLocalArchives(Configuration conf,
                                                String str)

Deprecated.

Set the conf to contain the location for localized archives. Used by internal DistributedCache code.

Parameters:

conf - The conf to modify to contain the localized caches

str - a comma separated list of local archives

setLocalFiles

@Deprecated
public static void setLocalFiles(Configuration conf,
                                             String str)

Deprecated.

Set the conf to contain the location for localized files. Used by internal DistributedCache code.

Parameters:

conf - The conf to modify to contain the localized caches

str - a comma separated list of local files