Package org.apache.hadoop.yarn.logaggregation.filecontroller

Class LogAggregationFileController

java.lang.Object
org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController
Direct Known Subclasses:
LogAggregationIndexedFileController, LogAggregationTFileController
@Public @Unstable public abstract class LogAggregationFileController extends Object
Base class to implement Log Aggregation File Controller.

  • Field Details

    • TLDIR_PERMISSIONS

      protected static final org.apache.hadoop.fs.permission.FsPermission TLDIR_PERMISSIONS
      Permissions for the top level directory under which app directories will be created.

    • APP_DIR_PERMISSIONS

      protected static final org.apache.hadoop.fs.permission.FsPermission APP_DIR_PERMISSIONS
      Permissions for the Application directory.

    • APP_LOG_FILE_UMASK

      protected static final org.apache.hadoop.fs.permission.FsPermission APP_LOG_FILE_UMASK
      Umask for the log file.

    • conf

      protected org.apache.hadoop.conf.Configuration conf

    • remoteRootLogDir

      protected org.apache.hadoop.fs.Path remoteRootLogDir

    • remoteRootLogDirSuffix

      protected String remoteRootLogDirSuffix

    • retentionSize

      protected int retentionSize

    • fileControllerName

      protected String fileControllerName

    • fsSupportsChmod

      protected boolean fsSupportsChmod

  • Constructor Details

    • LogAggregationFileController

      public LogAggregationFileController()

  • Method Details

    • initialize

      public void initialize(org.apache.hadoop.conf.Configuration conf, String controllerName)
      Initialize the log file controller.
      Parameters:
      conf - the Configuration
      controllerName - the log controller class name

    • initInternal

      protected abstract void initInternal(org.apache.hadoop.conf.Configuration conf)
      Derived classes initialize themselves using this method.
      Parameters:
      conf - the Configuration

    • getRemoteRootLogDir

      public org.apache.hadoop.fs.Path getRemoteRootLogDir()
      Get the remote root log directory.
      Returns:
      the remote root log directory path

    • getRemoteRootLogDirSuffix

      public String getRemoteRootLogDirSuffix()
      Get the log aggregation directory suffix.
      Returns:
      the log aggregation directory suffix

    • getFileControllerName

      public String getFileControllerName()
      Get the name of the file controller.
      Returns:
      name of the file controller.

    • initializeWriter

      public abstract void initializeWriter(LogAggregationFileControllerContext context) throws IOException
      Initialize the writer.
      Parameters:
      context - the LogAggregationFileControllerContext
      Throws:
      IOException - if fails to initialize the writer

    • closeWriter

      public abstract void closeWriter() throws LogAggregationDFSException
      Close the writer.
      Throws:
      LogAggregationDFSException - if the closing of the writer fails (for example due to HDFS quota being exceeded)

    • write

      public abstract void write(AggregatedLogFormat.LogKey logKey, AggregatedLogFormat.LogValue logValue) throws IOException
      Write the log content.
      Parameters:
      logKey - the log key
      logValue - the log content
      Throws:
      IOException - if fails to write the logs

    • postWrite

      public abstract void postWrite(LogAggregationFileControllerContext record) throws Exception
      Operations needed after write the log content.
      Parameters:
      record - the LogAggregationFileControllerContext
      Throws:
      Exception - if anything fails

    • closePrintStream

      protected void closePrintStream(OutputStream out)

    • readAggregatedLogs

      public abstract boolean readAggregatedLogs(ContainerLogsRequest logRequest, OutputStream os) throws IOException
      Output container log.
      Parameters:
      logRequest - ContainerLogsRequest
      os - the output stream
      Returns:
      true if we can read the aggregated logs successfully
      Throws:
      IOException - if we can not access the log file.

    • readAggregatedLogsMeta

      public abstract List<ContainerLogMeta> readAggregatedLogsMeta(ContainerLogsRequest logRequest) throws IOException
      Return a list of ContainerLogMeta for an application from Remote FileSystem.
      Parameters:
      logRequest - ContainerLogsRequest
      Returns:
      a list of ContainerLogMeta
      Throws:
      IOException - if there is no available log file

    • getLogMetaFilesOfNode

      public Map<String,List<ContainerLogFileInfo>> getLogMetaFilesOfNode(ExtendedLogMetaRequest logRequest, org.apache.hadoop.fs.FileStatus currentNodeFile, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
      Returns log file metadata for a node grouped by containers.
      Parameters:
      logRequest - extended query information holder
      currentNodeFile - file status of a node in an application directory
      appId - id of the application, which is the same as in node path
      Returns:
      log file metadata
      Throws:
      IOException - if there is no node file

    • getApplicationDirectoriesOfUser

      public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.FileStatus> getApplicationDirectoriesOfUser(String user) throws IOException
      Gets all application directories of a user.
      Parameters:
      user - name of the user
      Returns:
      a lazy iterator of directories
      Throws:
      IOException - if user directory does not exist

    • getNodeFilesOfApplicationDirectory

      public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.FileStatus> getNodeFilesOfApplicationDirectory(org.apache.hadoop.fs.FileStatus appDir) throws IOException
      Gets all node files in an application directory.
      Parameters:
      appDir - application directory
      Returns:
      a lazy iterator of files
      Throws:
      IOException - if file context is not reachable

    • renderAggregatedLogsBlock

      public abstract void renderAggregatedLogsBlock(HtmlBlock.Block html, View.ViewContext context)
      Render Aggregated Logs block.
      Parameters:
      html - the html
      context - the ViewContext

    • getApplicationOwner

      public abstract String getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
      Returns the owner of the application.
      Parameters:
      aggregatedLogPath - the aggregatedLog path
      appId - the ApplicationId
      Returns:
      the application owner
      Throws:
      IOException - if we can not get the application owner

    • getApplicationAcls

      public abstract Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String> getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
      Returns ACLs for the application. An empty map is returned if no ACLs are found.
      Parameters:
      aggregatedLogPath - the aggregatedLog path.
      appId - the ApplicationId
      Returns:
      a map of the Application ACLs.
      Throws:
      IOException - if we can not get the application acls

    • verifyAndCreateRemoteLogDir

      public void verifyAndCreateRemoteLogDir()
      Verify and create the remote log directory.

    • createAppDir

      public void createAppDir(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.security.UserGroupInformation userUgi)
      Create remote Application directory for log aggregation.
      Parameters:
      user - the user
      appId - the application ID
      userUgi - the UGI

    • getFileSystem

      @VisibleForTesting protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • createDir

      protected void createDir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission fsPerm) throws IOException
      Throws:
      IOException

    • checkExists

      protected boolean checkExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission fsPerm) throws IOException
      Throws:
      IOException

    • getRemoteNodeLogFileForApp

      public org.apache.hadoop.fs.Path getRemoteNodeLogFileForApp(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user, org.apache.hadoop.yarn.api.records.NodeId nodeId)
      Get the remote aggregated log path.
      Parameters:
      appId - the ApplicationId
      user - the Application Owner
      nodeId - the NodeManager Id
      Returns:
      the remote aggregated log path

    • getRemoteAppLogDir

      public org.apache.hadoop.fs.Path getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String appOwner) throws IOException
      Get the remote application directory for log aggregation.
      Parameters:
      appId - the Application ID
      appOwner - the Application Owner
      Returns:
      the remote application directory
      Throws:
      IOException - if can not find the remote application directory

    • getOlderRemoteAppLogDir

      public org.apache.hadoop.fs.Path getOlderRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String appOwner) throws IOException
      Get the older remote application directory for log aggregation.
      Parameters:
      appId - the Application ID
      appOwner - the Application Owner
      Returns:
      the older remote application directory
      Throws:
      IOException - if can not find the remote application directory

    • cleanOldLogs

      protected void cleanOldLogs(org.apache.hadoop.fs.Path remoteNodeLogFileForApp, org.apache.hadoop.yarn.api.records.NodeId nodeId, org.apache.hadoop.security.UserGroupInformation userUgi)

    • aggregatedLogSuffix

      protected String aggregatedLogSuffix(String fileName)
      Create the aggregated log suffix. The LogAggregationFileController should call this to get the suffix and append the suffix to the end of each log. This would keep the aggregated log format consistent.
      Parameters:
      fileName - the File Name
      Returns:
      the aggregated log suffix String

    • isFsSupportsChmod

      public boolean isFsSupportsChmod()

    • belongsToAppAttempt

      protected boolean belongsToAppAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId, String containerIdStr)