Package org.apache.hadoop.yarn.logaggregation.filecontroller.ifile

Class LogAggregationIndexedFileController

java.lang.Object

org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController

@Private
@Unstable
public class LogAggregationIndexedFileController
extends LogAggregationFileController

The Indexed Log Aggregation File Format implementation.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	LogAggregationIndexedFileController.IndexedFileLogMeta	This IndexedFileLogMeta includes the meta information for a single file which would be aggregated in one Log aggregation cycle.
static class	LogAggregationIndexedFileController.IndexedLogsMeta	This IndexedLogsMeta includes all the meta information for the aggregated log file.
static class	LogAggregationIndexedFileController.IndexedPerAggregationLogMeta	This IndexedPerAggregationLogMeta includes the meta information for all files which would be aggregated in one Log aggregation cycle.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	CHECK_SUM_FILE_SUFFIX

Fields inherited from class org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

APP_DIR_PERMISSIONS, APP_LOG_FILE_UMASK, conf, fileControllerName, fsSupportsChmod, remoteRootLogDir, remoteRootLogDirSuffix, retentionSize, TLDIR_PERMISSIONS

Constructor Summary

Constructors

Constructor	Description
LogAggregationIndexedFileController()

Method Summary

Modifier and Type	Method	Description
void	closeWriter()	Close the writer.
org.apache.hadoop.fs.FileStatus	getAllChecksumFiles(Map<String,org.apache.hadoop.fs.FileStatus> fileMap, String fileName)
Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String>	getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId)	Returns ACLs for the application.
String	getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId)	Returns the owner of the application.
static int	getFSInputBufferSize(org.apache.hadoop.conf.Configuration conf)
static int	getFSOutputBufferSize(org.apache.hadoop.conf.Configuration conf)
Map<String,List<ContainerLogFileInfo>>	getLogMetaFilesOfNode(ExtendedLogMetaRequest logRequest, org.apache.hadoop.fs.FileStatus currentNodeFile, org.apache.hadoop.yarn.api.records.ApplicationId appId)	Returns log file metadata for a node grouped by containers.
List<org.apache.hadoop.fs.FileStatus>	getNodeLogFileToRead(List<org.apache.hadoop.fs.FileStatus> nodeFiles, String nodeId, org.apache.hadoop.yarn.api.records.ApplicationId appId)
org.apache.hadoop.fs.Path	getOlderRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user)	Get the older remote application directory for log aggregation.
org.apache.hadoop.fs.Path	getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user)	Get the remote application directory for log aggregation.
long	getRollOverLogMaxSize(org.apache.hadoop.conf.Configuration conf)
Clock	getSystemClock()
void	initializeWriter(LogAggregationFileControllerContext context)	Initialize the writer.
void	initInternal(org.apache.hadoop.conf.Configuration conf)	Derived classes initialize themselves using this method.
boolean	isRollover(org.apache.hadoop.fs.FileContext fc, org.apache.hadoop.fs.Path candidate)
LogAggregationIndexedFileController.IndexedLogsMeta	loadIndexedLogsMeta(org.apache.hadoop.fs.Path remoteLogPath, long end, org.apache.hadoop.yarn.api.records.ApplicationId appId)
Map<String,Long>	parseCheckSumFiles(List<org.apache.hadoop.fs.FileStatus> fileList)
void	postWrite(LogAggregationFileControllerContext record)	Operations needed after write the log content.
boolean	readAggregatedLogs(ContainerLogsRequest logRequest, OutputStream os)	Output container log.
List<ContainerLogMeta>	readAggregatedLogsMeta(ContainerLogsRequest logRequest)	Return a list of ContainerLogMeta for an application from Remote FileSystem.
void	renderAggregatedLogsBlock(HtmlBlock.Block html, View.ViewContext context)	Render Aggregated Logs block.
void	write(AggregatedLogFormat.LogKey logKey, AggregatedLogFormat.LogValue logValue)	Write the log content.

Methods inherited from class org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

aggregatedLogSuffix, belongsToAppAttempt, checkExists, cleanOldLogs, closePrintStream, createAppDir, createDir, getApplicationDirectoriesOfUser, getFileControllerName, getFileSystem, getNodeFilesOfApplicationDirectory, getRemoteNodeLogFileForApp, getRemoteRootLogDir, getRemoteRootLogDirSuffix, initialize, isFsSupportsChmod, verifyAndCreateRemoteLogDir

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

CHECK_SUM_FILE_SUFFIX

@VisibleForTesting
public static final String CHECK_SUM_FILE_SUFFIX

See Also:

• Constant Field Values

Constructor Details

LogAggregationIndexedFileController

public LogAggregationIndexedFileController()

Method Details

initInternal

public void initInternal(org.apache.hadoop.conf.Configuration conf)

Description copied from class: LogAggregationFileController

Derived classes initialize themselves using this method.

Specified by:

initInternal in class LogAggregationFileController

Parameters:

conf - the Configuration

initializeWriter

public void initializeWriter(LogAggregationFileControllerContext context)
                      throws IOException

Description copied from class: LogAggregationFileController

Initialize the writer.

Specified by:

initializeWriter in class LogAggregationFileController

Parameters:

context - the LogAggregationFileControllerContext

Throws:

IOException - if fails to initialize the writer

closeWriter

public void closeWriter()

Description copied from class: LogAggregationFileController

Close the writer.

Specified by:

closeWriter in class LogAggregationFileController

write

public void write(AggregatedLogFormat.LogKey logKey,
 AggregatedLogFormat.LogValue logValue)
           throws IOException

Description copied from class: LogAggregationFileController

Write the log content.

Specified by:

write in class LogAggregationFileController

Parameters:

logKey - the log key

logValue - the log content

Throws:

IOException - if fails to write the logs

postWrite

public void postWrite(LogAggregationFileControllerContext record)
               throws Exception

Description copied from class: LogAggregationFileController

Operations needed after write the log content.

Specified by:

postWrite in class LogAggregationFileController

Parameters:

record - the LogAggregationFileControllerContext

Throws:

Exception - if anything fails

readAggregatedLogs

public boolean readAggregatedLogs(ContainerLogsRequest logRequest,
 OutputStream os)
                           throws IOException

Description copied from class: LogAggregationFileController

Output container log.

Specified by:

readAggregatedLogs in class LogAggregationFileController

Parameters:

logRequest - ContainerLogsRequest

os - the output stream

Returns:

true if we can read the aggregated logs successfully

Throws:

IOException - if we can not access the log file.

getLogMetaFilesOfNode

public Map<String,List<ContainerLogFileInfo>> getLogMetaFilesOfNode(ExtendedLogMetaRequest logRequest,
 org.apache.hadoop.fs.FileStatus currentNodeFile,
 org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                             throws IOException

Description copied from class: LogAggregationFileController

Returns log file metadata for a node grouped by containers.

Overrides:

getLogMetaFilesOfNode in class LogAggregationFileController

Parameters:

logRequest - extended query information holder

currentNodeFile - file status of a node in an application directory

appId - id of the application, which is the same as in node path

Returns:

log file metadata

Throws:

IOException - if there is no node file

readAggregatedLogsMeta

public List<ContainerLogMeta> readAggregatedLogsMeta(ContainerLogsRequest logRequest)
                                              throws IOException

Description copied from class: LogAggregationFileController

Return a list of ContainerLogMeta for an application from Remote FileSystem.

Specified by:

readAggregatedLogsMeta in class LogAggregationFileController

Parameters:

logRequest - ContainerLogsRequest

Returns:

a list of ContainerLogMeta

Throws:

IOException - if there is no available log file

parseCheckSumFiles

@Private
public Map<String,Long> parseCheckSumFiles(List<org.apache.hadoop.fs.FileStatus> fileList)
                                    throws IOException

Throws:

IOException

getNodeLogFileToRead

@Private
public List<org.apache.hadoop.fs.FileStatus> getNodeLogFileToRead(List<org.apache.hadoop.fs.FileStatus> nodeFiles,
 String nodeId,
 org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                           throws IOException

Throws:

IOException

getAllChecksumFiles

@Private
public org.apache.hadoop.fs.FileStatus getAllChecksumFiles(Map<String,org.apache.hadoop.fs.FileStatus> fileMap,
 String fileName)

renderAggregatedLogsBlock

public void renderAggregatedLogsBlock(HtmlBlock.Block html,
 View.ViewContext context)

Description copied from class: LogAggregationFileController

Render Aggregated Logs block.

Specified by:

renderAggregatedLogsBlock in class LogAggregationFileController

Parameters:

html - the html

context - the ViewContext

getApplicationOwner

public String getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLogPath,
 org.apache.hadoop.yarn.api.records.ApplicationId appId)
                           throws IOException

Description copied from class: LogAggregationFileController

Returns the owner of the application.

Specified by:

getApplicationOwner in class LogAggregationFileController

Parameters:

aggregatedLogPath - the aggregatedLog path

appId - the ApplicationId

Returns:

the application owner

Throws:

IOException - if we can not get the application owner

getApplicationAcls

public Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String> getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLogPath,
 org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                                                        throws IOException

Description copied from class: LogAggregationFileController

Returns ACLs for the application. An empty map is returned if no ACLs are found.

Specified by:

getApplicationAcls in class LogAggregationFileController

Parameters:

aggregatedLogPath - the aggregatedLog path.

appId - the ApplicationId

Returns:

a map of the Application ACLs.

Throws:

IOException - if we can not get the application acls

getRemoteAppLogDir

public org.apache.hadoop.fs.Path getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId,
 String user)
                                             throws IOException

Description copied from class: LogAggregationFileController

Get the remote application directory for log aggregation.

Overrides:

getRemoteAppLogDir in class LogAggregationFileController

Parameters:

appId - the Application ID

user - the Application Owner

Returns:

the remote application directory

Throws:

IOException - if can not find the remote application directory

getOlderRemoteAppLogDir

public org.apache.hadoop.fs.Path getOlderRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId,
 String user)
                                                  throws IOException

Description copied from class: LogAggregationFileController

Get the older remote application directory for log aggregation.

Overrides:

getOlderRemoteAppLogDir in class LogAggregationFileController

Parameters:

appId - the Application ID

user - the Application Owner

Returns:

the older remote application directory

Throws:

IOException - if can not find the remote application directory

loadIndexedLogsMeta

@Private
public LogAggregationIndexedFileController.IndexedLogsMeta loadIndexedLogsMeta(org.apache.hadoop.fs.Path remoteLogPath,
 long end,
 org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                                        throws IOException

Throws:

IOException

getFSOutputBufferSize

@Private
public static int getFSOutputBufferSize(org.apache.hadoop.conf.Configuration conf)

getFSInputBufferSize

@Private
public static int getFSInputBufferSize(org.apache.hadoop.conf.Configuration conf)

getRollOverLogMaxSize

@Private
@VisibleForTesting
public long getRollOverLogMaxSize(org.apache.hadoop.conf.Configuration conf)

isRollover

@Private
@VisibleForTesting
public boolean isRollover(org.apache.hadoop.fs.FileContext fc,
 org.apache.hadoop.fs.Path candidate)
                   throws IOException

Throws:

IOException

getSystemClock

@Private
@VisibleForTesting
public Clock getSystemClock()