org.apache.hadoop.yarn.server.nodemanager

Class LocalDirsHandlerService

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

public class LocalDirsHandlerService
extends org.apache.hadoop.service.AbstractService

The class which provides functionality of checking the health of the local directories of a node. This specifically manages nodemanager-local-dirs and nodemanager-log-dirs by periodically checking their health.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Constructor Summary

Constructors

Constructor and Description
LocalDirsHandlerService()
LocalDirsHandlerService(NodeManagerMetrics nodeManagerMetrics)

Method Summary

Modifier and Type	Method and Description
boolean	areDisksHealthy() The minimum fraction of number of disks needed to be healthy for a node to be considered healthy in terms of disks is configured using YarnConfiguration.NM_MIN_HEALTHY_DISKS_FRACTION, with a default value of YarnConfiguration.DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION.
void	deregisterLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
void	deregisterLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
List<String>	getDiskFullLocalDirs()
List<String>	getDiskFullLogDirs()
String	getDisksHealthReport(boolean listGoodDirs) Function to generate a report on the state of the disks.
long	getLastDisksCheckTime()
List<String>	getLocalDirs()
List<String>	getLocalDirsForCleanup() Function to get the local dirs which should be considered when cleaning up resources.
List<String>	getLocalDirsForRead() Function to get the local dirs which should be considered for reading existing files on disk.
org.apache.hadoop.fs.Path	getLocalPathForRead(String pathStr)
org.apache.hadoop.fs.Path	getLocalPathForWrite(String pathStr)
org.apache.hadoop.fs.Path	getLocalPathForWrite(String pathStr, long size, boolean checkWrite)
List<String>	getLogDirs()
List<String>	getLogDirsForCleanup() Function to get the log dirs which should be considered when cleaning up resources.
List<String>	getLogDirsForRead() Function to get the log dirs which should be considered for reading existing files on disk.
org.apache.hadoop.fs.Path	getLogPathForWrite(String pathStr, boolean checkWrite)
org.apache.hadoop.fs.Path	getLogPathToRead(String pathStr)
boolean	isGoodLocalDir(String path)
boolean	isGoodLogDir(String path)
void	registerLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
void	registerLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
protected void	serviceInit(org.apache.hadoop.conf.Configuration config) Method which initializes the timertask and its interval time.
protected void	serviceStart() Method used to start the disk health monitoring, if enabled.
protected void	serviceStop() Method used to terminate the disk health monitoring service.
protected void	updateMetrics()
static String[]	validatePaths(String[] paths)

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

LocalDirsHandlerService

public LocalDirsHandlerService()

LocalDirsHandlerService

public LocalDirsHandlerService(NodeManagerMetrics nodeManagerMetrics)

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration config)
                    throws Exception

Method which initializes the timertask and its interval time.

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

Method used to start the disk health monitoring, if enabled.

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Method used to terminate the disk health monitoring service.

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

registerLocalDirsChangeListener

public void registerLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

registerLogDirsChangeListener

public void registerLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

deregisterLocalDirsChangeListener

public void deregisterLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

deregisterLogDirsChangeListener

public void deregisterLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

getLocalDirs

public List<String> getLocalDirs()

Returns:

the good/valid local directories based on disks' health

getLogDirs

public List<String> getLogDirs()

Returns:

the good/valid log directories based on disks' health

getDiskFullLocalDirs

public List<String> getDiskFullLocalDirs()

Returns:

the local directories which have no disk space

getDiskFullLogDirs

public List<String> getDiskFullLogDirs()

Returns:

the log directories that have no disk space

getLocalDirsForRead

public List<String> getLocalDirsForRead()

Function to get the local dirs which should be considered for reading existing files on disk. Contains the good local dirs and the local dirs that have reached the disk space limit

Returns:

the local dirs which should be considered for reading

getLocalDirsForCleanup

public List<String> getLocalDirsForCleanup()

Function to get the local dirs which should be considered when cleaning up resources. Contains the good local dirs and the local dirs that have reached the disk space limit

Returns:

the local dirs which should be considered for cleaning up

getLogDirsForRead

public List<String> getLogDirsForRead()

Function to get the log dirs which should be considered for reading existing files on disk. Contains the good log dirs and the log dirs that have reached the disk space limit

Returns:

the log dirs which should be considered for reading

getLogDirsForCleanup

public List<String> getLogDirsForCleanup()

Function to get the log dirs which should be considered when cleaning up resources. Contains the good log dirs and the log dirs that have reached the disk space limit

Returns:

the log dirs which should be considered for cleaning up

getDisksHealthReport

public String getDisksHealthReport(boolean listGoodDirs)

Function to generate a report on the state of the disks.

Parameters:

listGoodDirs - flag to determine whether the report should report the state of good dirs or failed dirs

Returns:

the health report of nm-local-dirs and nm-log-dirs

areDisksHealthy

public boolean areDisksHealthy()

The minimum fraction of number of disks needed to be healthy for a node to be considered healthy in terms of disks is configured using YarnConfiguration.NM_MIN_HEALTHY_DISKS_FRACTION, with a default value of YarnConfiguration.DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION.

Returns:

false if either (a) more than the allowed percentage of nm-local-dirs failed or (b) more than the allowed percentage of nm-log-dirs failed.

getLastDisksCheckTime

public long getLastDisksCheckTime()

isGoodLocalDir

public boolean isGoodLocalDir(String path)

isGoodLogDir

public boolean isGoodLogDir(String path)

getLocalPathForWrite

public org.apache.hadoop.fs.Path getLocalPathForWrite(String pathStr)
                                               throws IOException

Throws:

IOException

getLocalPathForWrite

public org.apache.hadoop.fs.Path getLocalPathForWrite(String pathStr,
                                                      long size,
                                                      boolean checkWrite)
                                               throws IOException

Throws:

IOException

getLocalPathForRead

public org.apache.hadoop.fs.Path getLocalPathForRead(String pathStr)
                                              throws IOException

Throws:

IOException

getLogPathForWrite

public org.apache.hadoop.fs.Path getLogPathForWrite(String pathStr,
                                                    boolean checkWrite)
                                             throws IOException

Throws:

IOException

getLogPathToRead

public org.apache.hadoop.fs.Path getLogPathToRead(String pathStr)
                                           throws IOException

Throws:

IOException

validatePaths

public static String[] validatePaths(String[] paths)

updateMetrics

protected void updateMetrics()