|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.hadoop.filecache.TrackerDistributedCacheManager
public class TrackerDistributedCacheManager
Manages a single machine's instance of a cross-job cache. This class would typically be instantiated by a TaskTracker (or something that emulates it, like LocalJobRunner). This class is internal to Hadoop, and should not be treated as a public interface.
Nested Class Summary | |
---|---|
protected class |
TrackerDistributedCacheManager.BaseDirManager
This class holds properties of each base directories and is responsible for clean up unused cache files in base directories. |
protected class |
TrackerDistributedCacheManager.CleanupThread
A thread to check and cleanup the unused files periodically |
Field Summary | |
---|---|
protected TrackerDistributedCacheManager.BaseDirManager |
baseDirManager
|
protected TrackerDistributedCacheManager.CleanupThread |
cleanupThread
|
Constructor Summary | |
---|---|
TrackerDistributedCacheManager(Configuration conf,
TaskController controller)
|
Method Summary | |
---|---|
static void |
createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
This method create symlinks for all files in a given dir in another directory. |
static void |
determineCacheVisibilities(Configuration job)
Determines the visibilities of the distributed cache files and archives. |
static void |
determineTimestamps(Configuration job)
Determines timestamps of files to be cached, and stores those in the configuration. |
static long |
downloadCacheObject(Configuration conf,
URI source,
Path destination,
long desiredTimestamp,
boolean isArchive,
FsPermission permission)
Download a given path to the local file system. |
static boolean[] |
getArchiveVisibilities(Configuration conf)
Get the booleans on whether the archives are public or not. |
static void |
getDelegationTokens(Configuration job,
Credentials credentials)
For each archive or cache file - get the corresponding delegation token |
static boolean[] |
getFileVisibilities(Configuration conf)
Get the booleans on whether the files are public or not. |
protected TaskDistributedCacheManager |
getTaskDistributedCacheManager(JobID jobId)
|
TaskDistributedCacheManager |
newTaskDistributedCacheManager(JobID jobId,
Configuration taskConf)
|
void |
purgeCache()
Clear the entire contents of the cache and delete the backing files. |
void |
removeTaskDistributedCacheManager(JobID jobId)
|
void |
setArchiveSizes(JobID jobId,
long[] sizes)
Set the sizes for any archives, files, or directories in the private distributed cache. |
void |
startCleanupThread()
Start the background thread |
void |
stopCleanupThread()
Stop the background thread |
static void |
validate(Configuration conf)
This is part of the framework API. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
protected TrackerDistributedCacheManager.BaseDirManager baseDirManager
protected TrackerDistributedCacheManager.CleanupThread cleanupThread
Constructor Detail |
---|
public TrackerDistributedCacheManager(Configuration conf, TaskController controller) throws IOException
IOException
Method Detail |
---|
public static long downloadCacheObject(Configuration conf, URI source, Path destination, long desiredTimestamp, boolean isArchive, FsPermission permission) throws IOException
conf
- the job's configurationsource
- the source to copy fromdestination
- where to copy the file. must be local fsdesiredTimestamp
- the required modification timestamp of the sourceisArchive
- is this an archive that should be expandedpermission
- the desired permissions of the file.
IOException
public static void createAllSymlink(Configuration conf, File jobCacheDir, File workDir) throws IOException
conf
- the configurationjobCacheDir
- the target directory for creating symlinksworkDir
- the directory in which the symlinks are created
IOException
public void purgeCache()
public TaskDistributedCacheManager newTaskDistributedCacheManager(JobID jobId, Configuration taskConf) throws IOException
IOException
public void setArchiveSizes(JobID jobId, long[] sizes) throws IOException
IOException
public void removeTaskDistributedCacheManager(JobID jobId)
protected TaskDistributedCacheManager getTaskDistributedCacheManager(JobID jobId)
public static void determineTimestamps(Configuration job) throws IOException
job
- Configuration of a job.
IOException
public static void determineCacheVisibilities(Configuration job) throws IOException
job
-
IOException
public static boolean[] getFileVisibilities(Configuration conf)
conf
- The configuration which stored the timestamps
IOException
public static boolean[] getArchiveVisibilities(Configuration conf)
conf
- The configuration which stored the timestamps
public static void getDelegationTokens(Configuration job, Credentials credentials) throws IOException
job
- credentials
-
IOException
public static void validate(Configuration conf) throws InvalidJobConfException
conf
- a Configuration
to be cheked for duplication
in cached URIs
InvalidJobConfException
public void startCleanupThread()
public void stopCleanupThread()
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |