org.apache.hadoop.yarn.server.nodemanager

Class LocalDirsHandlerService

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, HealthReporter

public class LocalDirsHandlerService
extends org.apache.hadoop.service.AbstractService
implements HealthReporter

The class which provides functionality of checking the health of the local directories of a node. This specifically manages nodemanager-local-dirs and nodemanager-log-dirs by periodically checking their health.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Constructor Summary

Constructors

Constructor and Description
LocalDirsHandlerService()
LocalDirsHandlerService(NodeManagerMetrics nodeManagerMetrics)

Method Summary

Modifier and Type	Method and Description
boolean	areDisksHealthy() The minimum fraction of number of disks needed to be healthy for a node to be considered healthy in terms of disks is configured using YarnConfiguration.NM_MIN_HEALTHY_DISKS_FRACTION, with a default value of YarnConfiguration.DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION.
void	checkDirs()
void	deregisterLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
void	deregisterLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
Iterable<org.apache.hadoop.fs.Path>	getAllLocalPathsForRead(String pathStr)
List<String>	getDiskFullLocalDirs()
List<String>	getDiskFullLogDirs()
String	getDisksHealthReport(boolean listGoodDirs) Function to generate a report on the state of the disks.
String	getHealthReport() Returns output from health check.
long	getLastDisksCheckTime()
long	getLastHealthReportTime() Returns time stamp when node health check was last run.
List<String>	getLocalDirs()
List<String>	getLocalDirsForCleanup() Function to get the local dirs which should be considered when cleaning up resources.
List<String>	getLocalDirsForRead() Function to get the local dirs which should be considered for reading existing files on disk.
org.apache.hadoop.fs.Path	getLocalPathForRead(String pathStr)
org.apache.hadoop.fs.Path	getLocalPathForWrite(String pathStr)
org.apache.hadoop.fs.Path	getLocalPathForWrite(String pathStr, long size, boolean checkWrite)
List<String>	getLogDirs()
List<String>	getLogDirsForCleanup() Function to get the log dirs which should be considered when cleaning up resources.
List<String>	getLogDirsForRead() Function to get the log dirs which should be considered for reading existing files on disk.
org.apache.hadoop.fs.Path	getLogPathForWrite(String pathStr, boolean checkWrite)
org.apache.hadoop.fs.Path	getLogPathToRead(String pathStr)
boolean	isGoodLocalDir(String path)
boolean	isGoodLogDir(String path)
boolean	isHealthy() Gets whether the node is healthy or not.
void	registerLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
void	registerLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)
protected void	serviceInit(org.apache.hadoop.conf.Configuration config) Method which initializes the timertask and its interval time.
protected void	serviceStart() Method used to start the disk health monitoring, if enabled.
protected void	serviceStop() Method used to terminate the disk health monitoring service.
protected void	updateMetrics()
static String[]	validatePaths(String[] paths)

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

LocalDirsHandlerService

public LocalDirsHandlerService()

LocalDirsHandlerService

public LocalDirsHandlerService(NodeManagerMetrics nodeManagerMetrics)

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration config)
                    throws Exception

Method which initializes the timertask and its interval time.

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

Method used to start the disk health monitoring, if enabled.

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Method used to terminate the disk health monitoring service.

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

registerLocalDirsChangeListener

public void registerLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

registerLogDirsChangeListener

public void registerLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

deregisterLocalDirsChangeListener

public void deregisterLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

deregisterLogDirsChangeListener

public void deregisterLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

getLocalDirs

public List<String> getLocalDirs()

Returns:

the good/valid local directories based on disks' health

getLogDirs

public List<String> getLogDirs()

Returns:

the good/valid log directories based on disks' health

getDiskFullLocalDirs

public List<String> getDiskFullLocalDirs()

Returns:

the local directories which have no disk space

getDiskFullLogDirs

public List<String> getDiskFullLogDirs()

Returns:

the log directories that have no disk space

getLocalDirsForRead

public List<String> getLocalDirsForRead()

Function to get the local dirs which should be considered for reading existing files on disk. Contains the good local dirs and the local dirs that have reached the disk space limit

Returns:

the local dirs which should be considered for reading

getLocalDirsForCleanup

public List<String> getLocalDirsForCleanup()

Function to get the local dirs which should be considered when cleaning up resources. Contains the good local dirs and the local dirs that have reached the disk space limit

Returns:

the local dirs which should be considered for cleaning up

getLogDirsForRead

public List<String> getLogDirsForRead()

Function to get the log dirs which should be considered for reading existing files on disk. Contains the good log dirs and the log dirs that have reached the disk space limit

Returns:

the log dirs which should be considered for reading

getLogDirsForCleanup

public List<String> getLogDirsForCleanup()

Function to get the log dirs which should be considered when cleaning up resources. Contains the good log dirs and the log dirs that have reached the disk space limit

Returns:

the log dirs which should be considered for cleaning up

getDisksHealthReport

public String getDisksHealthReport(boolean listGoodDirs)

Function to generate a report on the state of the disks.

Parameters:

listGoodDirs - flag to determine whether the report should report the state of good dirs or failed dirs

Returns:

the health report of nm-local-dirs and nm-log-dirs

getHealthReport

public String getHealthReport()

Description copied from interface: HealthReporter

Returns output from health check. If node is healthy then an empty string is returned.

Specified by:

getHealthReport in interface HealthReporter

Returns:

output from health check

areDisksHealthy

public boolean areDisksHealthy()

The minimum fraction of number of disks needed to be healthy for a node to be considered healthy in terms of disks is configured using YarnConfiguration.NM_MIN_HEALTHY_DISKS_FRACTION, with a default value of YarnConfiguration.DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION.

Returns:

false if either (a) more than the allowed percentage of nm-local-dirs failed or (b) more than the allowed percentage of nm-log-dirs failed.

isHealthy

public boolean isHealthy()

Description copied from interface: HealthReporter

Gets whether the node is healthy or not.

Specified by:

isHealthy in interface HealthReporter

Returns:

true if node is healthy

getLastDisksCheckTime

public long getLastDisksCheckTime()

getLastHealthReportTime

public long getLastHealthReportTime()

Description copied from interface: HealthReporter

Returns time stamp when node health check was last run.

Specified by:

getLastHealthReportTime in interface HealthReporter

Returns:

timestamp when node health script was last run

isGoodLocalDir

public boolean isGoodLocalDir(String path)

isGoodLogDir

public boolean isGoodLogDir(String path)

checkDirs

@VisibleForTesting
public void checkDirs()

getLocalPathForWrite

public org.apache.hadoop.fs.Path getLocalPathForWrite(String pathStr)
                                               throws IOException

Throws:

IOException

getLocalPathForWrite

public org.apache.hadoop.fs.Path getLocalPathForWrite(String pathStr,
                                                      long size,
                                                      boolean checkWrite)
                                               throws IOException

Throws:

IOException

getLocalPathForRead

public org.apache.hadoop.fs.Path getLocalPathForRead(String pathStr)
                                              throws IOException

Throws:

IOException

getAllLocalPathsForRead

public Iterable<org.apache.hadoop.fs.Path> getAllLocalPathsForRead(String pathStr)
                                                            throws IOException

Throws:

IOException

getLogPathForWrite

public org.apache.hadoop.fs.Path getLogPathForWrite(String pathStr,
                                                    boolean checkWrite)
                                             throws IOException

Throws:

IOException

getLogPathToRead

public org.apache.hadoop.fs.Path getLogPathToRead(String pathStr)
                                           throws IOException

Throws:

IOException

validatePaths

public static String[] validatePaths(String[] paths)

updateMetrics

protected void updateMetrics()