Package org.apache.hadoop.yarn.server.nodemanager

Class LocalDirsHandlerService

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service, HealthReporter
public class LocalDirsHandlerService extends org.apache.hadoop.service.AbstractService implements HealthReporter
The class which provides functionality of checking the health of the local directories of a node. This specifically manages nodemanager-local-dirs and nodemanager-log-dirs by periodically checking their health.

  • Constructor Details

    • LocalDirsHandlerService

      public LocalDirsHandlerService()

    • LocalDirsHandlerService

      public LocalDirsHandlerService(NodeManagerMetrics nodeManagerMetrics)

  • Method Details

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration config) throws Exception
      Method which initializes the timertask and its interval time.
      Overrides:
      serviceInit in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStart

      protected void serviceStart() throws Exception
      Method used to start the disk health monitoring, if enabled.
      Overrides:
      serviceStart in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStop

      protected void serviceStop() throws Exception
      Method used to terminate the disk health monitoring service.
      Overrides:
      serviceStop in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • registerLocalDirsChangeListener

      public void registerLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

    • registerLogDirsChangeListener

      public void registerLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

    • deregisterLocalDirsChangeListener

      public void deregisterLocalDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

    • deregisterLogDirsChangeListener

      public void deregisterLogDirsChangeListener(DirectoryCollection.DirsChangeListener listener)

    • getLocalDirs

      public List<String> getLocalDirs()
      Returns:
      the good/valid local directories based on disks' health

    • getLogDirs

      public List<String> getLogDirs()
      Returns:
      the good/valid log directories based on disks' health

    • getDiskFullLocalDirs

      public List<String> getDiskFullLocalDirs()
      Returns:
      the local directories which have no disk space

    • getDiskFullLogDirs

      public List<String> getDiskFullLogDirs()
      Returns:
      the log directories that have no disk space

    • getLocalDirsForRead

      public List<String> getLocalDirsForRead()
      Function to get the local dirs which should be considered for reading existing files on disk. Contains the good local dirs and the local dirs that have reached the disk space limit
      Returns:
      the local dirs which should be considered for reading

    • getLocalDirsForCleanup

      public List<String> getLocalDirsForCleanup()
      Function to get the local dirs which should be considered when cleaning up resources. Contains the good local dirs and the local dirs that have reached the disk space limit
      Returns:
      the local dirs which should be considered for cleaning up

    • getLogDirsForRead

      public List<String> getLogDirsForRead()
      Function to get the log dirs which should be considered for reading existing files on disk. Contains the good log dirs and the log dirs that have reached the disk space limit
      Returns:
      the log dirs which should be considered for reading

    • getLogDirsForCleanup

      public List<String> getLogDirsForCleanup()
      Function to get the log dirs which should be considered when cleaning up resources. Contains the good log dirs and the log dirs that have reached the disk space limit
      Returns:
      the log dirs which should be considered for cleaning up

    • getDisksHealthReport

      public String getDisksHealthReport(boolean listGoodDirs)
      Function to generate a report on the state of the disks.
      Parameters:
      listGoodDirs - flag to determine whether the report should report the state of good dirs or failed dirs
      Returns:
      the health report of nm-local-dirs and nm-log-dirs

    • getHealthReport

      public String getHealthReport()
      Description copied from interface: HealthReporter
      Returns output from health check. If node is healthy then an empty string is returned.
      Specified by:
      getHealthReport in interface HealthReporter
      Returns:
      output from health check

    • areDisksHealthy

      public boolean areDisksHealthy()
      The minimum fraction of number of disks needed to be healthy for a node to be considered healthy in terms of disks is configured using YarnConfiguration.NM_MIN_HEALTHY_DISKS_FRACTION, with a default value of YarnConfiguration.DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION.
      Returns:
      false if either (a) more than the allowed percentage of nm-local-dirs failed or (b) more than the allowed percentage of nm-log-dirs failed.

    • isHealthy

      public boolean isHealthy()
      Description copied from interface: HealthReporter
      Gets whether the node is healthy or not.
      Specified by:
      isHealthy in interface HealthReporter
      Returns:
      true if node is healthy

    • getLastDisksCheckTime

      public long getLastDisksCheckTime()

    • getLastHealthReportTime

      public long getLastHealthReportTime()
      Description copied from interface: HealthReporter
      Returns time stamp when node health check was last run.
      Specified by:
      getLastHealthReportTime in interface HealthReporter
      Returns:
      timestamp when node health script was last run

    • isGoodLocalDir

      public boolean isGoodLocalDir(String path)

    • isGoodLogDir

      public boolean isGoodLogDir(String path)

    • checkDirs

      @VisibleForTesting public void checkDirs()

    • getLocalPathForWrite

      public org.apache.hadoop.fs.Path getLocalPathForWrite(String pathStr) throws IOException
      Throws:
      IOException

    • getLocalPathForWrite

      public org.apache.hadoop.fs.Path getLocalPathForWrite(String pathStr, long size, boolean checkWrite) throws IOException
      Throws:
      IOException

    • getLocalPathForRead

      public org.apache.hadoop.fs.Path getLocalPathForRead(String pathStr) throws IOException
      Throws:
      IOException

    • getAllLocalPathsForRead

      public Iterable<org.apache.hadoop.fs.Path> getAllLocalPathsForRead(String pathStr) throws IOException
      Throws:
      IOException

    • getLogPathForWrite

      public org.apache.hadoop.fs.Path getLogPathForWrite(String pathStr, boolean checkWrite) throws IOException
      Throws:
      IOException

    • getLogPathToRead

      public org.apache.hadoop.fs.Path getLogPathToRead(String pathStr) throws IOException
      Throws:
      IOException

    • validatePaths

      public static String[] validatePaths(String[] paths)

    • updateMetrics

      protected void updateMetrics()