org.apache.hadoop.yarn.logaggregation.filecontroller

Class LogAggregationFileController

java.lang.Object

org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

Direct Known Subclasses:

LogAggregationIndexedFileController, LogAggregationTFileController

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract class LogAggregationFileController
extends Object

Base class to implement Log Aggregation File Controller.

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.apache.hadoop.fs.permission.FsPermission	APP_DIR_PERMISSIONS Permissions for the Application directory.
protected static org.apache.hadoop.fs.permission.FsPermission	APP_LOG_FILE_UMASK Umask for the log file.
protected org.apache.hadoop.conf.Configuration	conf
protected String	fileControllerName
static String	LOG_AGGREGATION_FS_SUPPORT_APPEND
protected org.apache.hadoop.fs.Path	remoteRootLogDir
protected String	remoteRootLogDirSuffix
protected int	retentionSize
protected static org.apache.hadoop.fs.permission.FsPermission	TLDIR_PERMISSIONS Permissions for the top level directory under which app directories will be created.

Constructor Summary

Constructors

Constructor and Description
LogAggregationFileController()

Method Summary

Modifier and Type	Method and Description
protected String	aggregatedLogSuffix(String fileName) Create the aggregated log suffix.
protected boolean	checkExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission fsPerm)
protected void	cleanOldLogs(org.apache.hadoop.fs.Path remoteNodeLogFileForApp, org.apache.hadoop.yarn.api.records.NodeId nodeId, org.apache.hadoop.security.UserGroupInformation userUgi)
protected void	closePrintStream(OutputStream out)
abstract void	closeWriter() Close the writer.
void	createAppDir(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.security.UserGroupInformation userUgi) Create remote Application directory for log aggregation.
protected void	createDir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission fsPerm)
abstract Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String>	getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId) Returns ACLs for the application.
abstract String	getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId) Returns the owner of the application.
protected org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.fs.Path	getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String appOwner) Get the remote application directory for log aggregation.
org.apache.hadoop.fs.Path	getRemoteNodeLogFileForApp(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user, org.apache.hadoop.yarn.api.records.NodeId nodeId) Get the remote aggregated log path.
org.apache.hadoop.fs.Path	getRemoteRootLogDir() Get the remote root log directory.
String	getRemoteRootLogDirSuffix() Get the log aggregation directory suffix.
void	initialize(org.apache.hadoop.conf.Configuration conf, String controllerName) Initialize the log file controller.
abstract void	initializeWriter(LogAggregationFileControllerContext context) Initialize the writer.
protected abstract void	initInternal(org.apache.hadoop.conf.Configuration conf) Derived classes initialize themselves using this method.
abstract void	postWrite(LogAggregationFileControllerContext record) Operations needed after write the log content.
abstract boolean	readAggregatedLogs(ContainerLogsRequest logRequest, OutputStream os) Output container log.
abstract List<ContainerLogMeta>	readAggregatedLogsMeta(ContainerLogsRequest logRequest) Return a list of ContainerLogMeta for an application from Remote FileSystem.
abstract void	renderAggregatedLogsBlock(HtmlBlock.Block html, View.ViewContext context) Render Aggregated Logs block.
void	verifyAndCreateRemoteLogDir() Verify and create the remote log directory.
abstract void	write(AggregatedLogFormat.LogKey logKey, AggregatedLogFormat.LogValue logValue) Write the log content.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TLDIR_PERMISSIONS

protected static final org.apache.hadoop.fs.permission.FsPermission TLDIR_PERMISSIONS

Permissions for the top level directory under which app directories will be created.

APP_DIR_PERMISSIONS

protected static final org.apache.hadoop.fs.permission.FsPermission APP_DIR_PERMISSIONS

Permissions for the Application directory.

APP_LOG_FILE_UMASK

protected static final org.apache.hadoop.fs.permission.FsPermission APP_LOG_FILE_UMASK

Umask for the log file.

LOG_AGGREGATION_FS_SUPPORT_APPEND

public static final String LOG_AGGREGATION_FS_SUPPORT_APPEND

See Also:

Constant Field Values

conf

protected org.apache.hadoop.conf.Configuration conf

remoteRootLogDir

protected org.apache.hadoop.fs.Path remoteRootLogDir

remoteRootLogDirSuffix

protected String remoteRootLogDirSuffix

retentionSize

protected int retentionSize

fileControllerName

protected String fileControllerName

Constructor Detail

LogAggregationFileController

public LogAggregationFileController()

Method Detail

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf,
                       String controllerName)

Initialize the log file controller.

Parameters:

conf - the Configuration

controllerName - the log controller class name

initInternal

protected abstract void initInternal(org.apache.hadoop.conf.Configuration conf)

Derived classes initialize themselves using this method.

Parameters:

conf - the Configuration

getRemoteRootLogDir

public org.apache.hadoop.fs.Path getRemoteRootLogDir()

Get the remote root log directory.

Returns:

the remote root log directory path

getRemoteRootLogDirSuffix

public String getRemoteRootLogDirSuffix()

Get the log aggregation directory suffix.

Returns:

the log aggregation directory suffix

initializeWriter

public abstract void initializeWriter(LogAggregationFileControllerContext context)
                               throws IOException

Initialize the writer.

Parameters:

context - the LogAggregationFileControllerContext

Throws:

IOException - if fails to initialize the writer

closeWriter

public abstract void closeWriter()
                          throws LogAggregationDFSException

Close the writer.

Throws:

LogAggregationDFSException - if the closing of the writer fails (for example due to HDFS quota being exceeded)

write

public abstract void write(AggregatedLogFormat.LogKey logKey,
                           AggregatedLogFormat.LogValue logValue)
                    throws IOException

Write the log content.

Parameters:

logKey - the log key

logValue - the log content

Throws:

IOException - if fails to write the logs

postWrite

public abstract void postWrite(LogAggregationFileControllerContext record)
                        throws Exception

Operations needed after write the log content.

Parameters:

record - the LogAggregationFileControllerContext

Throws:

Exception - if anything fails

closePrintStream

protected void closePrintStream(OutputStream out)

readAggregatedLogs

public abstract boolean readAggregatedLogs(ContainerLogsRequest logRequest,
                                           OutputStream os)
                                    throws IOException

Output container log.

Parameters:

logRequest - ContainerLogsRequest

os - the output stream

Returns:

true if we can read the aggregated logs successfully

Throws:

IOException - if we can not access the log file.

readAggregatedLogsMeta

public abstract List<ContainerLogMeta> readAggregatedLogsMeta(ContainerLogsRequest logRequest)
                                                       throws IOException

Return a list of ContainerLogMeta for an application from Remote FileSystem.

Parameters:

logRequest - ContainerLogsRequest

Returns:

a list of ContainerLogMeta

Throws:

IOException - if there is no available log file

renderAggregatedLogsBlock

public abstract void renderAggregatedLogsBlock(HtmlBlock.Block html,
                                               View.ViewContext context)

Render Aggregated Logs block.

Parameters:

html - the html

context - the ViewContext

getApplicationOwner

public abstract String getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLogPath,
                                           org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                    throws IOException

Returns the owner of the application.

Parameters:

aggregatedLogPath - the aggregatedLog path

appId - the ApplicationId

Returns:

the application owner

Throws:

IOException - if we can not get the application owner

getApplicationAcls

public abstract Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String> getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLogPath,
                                                                                                        org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                                                                 throws IOException

Returns ACLs for the application. An empty map is returned if no ACLs are found.

Parameters:

aggregatedLogPath - the aggregatedLog path.

appId - the ApplicationId

Returns:

a map of the Application ACLs.

Throws:

IOException - if we can not get the application acls

verifyAndCreateRemoteLogDir

public void verifyAndCreateRemoteLogDir()

Verify and create the remote log directory.

createAppDir

public void createAppDir(String user,
                         org.apache.hadoop.yarn.api.records.ApplicationId appId,
                         org.apache.hadoop.security.UserGroupInformation userUgi)

Create remote Application directory for log aggregation.

Parameters:

user - the user

appId - the application ID

userUgi - the UGI

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException

Throws:

IOException

createDir

protected void createDir(org.apache.hadoop.fs.FileSystem fs,
                         org.apache.hadoop.fs.Path path,
                         org.apache.hadoop.fs.permission.FsPermission fsPerm)
                  throws IOException

Throws:

IOException

checkExists

protected boolean checkExists(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path path,
                              org.apache.hadoop.fs.permission.FsPermission fsPerm)
                       throws IOException

Throws:

IOException

getRemoteNodeLogFileForApp

public org.apache.hadoop.fs.Path getRemoteNodeLogFileForApp(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                            String user,
                                                            org.apache.hadoop.yarn.api.records.NodeId nodeId)

Get the remote aggregated log path.

Parameters:

appId - the ApplicationId

user - the Application Owner

nodeId - the NodeManager Id

Returns:

the remote aggregated log path

getRemoteAppLogDir

public org.apache.hadoop.fs.Path getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                    String appOwner)
                                             throws IOException

Get the remote application directory for log aggregation.

Parameters:

appId - the Application ID

appOwner - the Application Owner

Returns:

the remote application directory

Throws:

IOException - if can not find the remote application directory

cleanOldLogs

protected void cleanOldLogs(org.apache.hadoop.fs.Path remoteNodeLogFileForApp,
                            org.apache.hadoop.yarn.api.records.NodeId nodeId,
                            org.apache.hadoop.security.UserGroupInformation userUgi)

aggregatedLogSuffix

protected String aggregatedLogSuffix(String fileName)

Create the aggregated log suffix. The LogAggregationFileController should call this to get the suffix and append the suffix to the end of each log. This would keep the aggregated log format consistent.

Parameters:

fileName - the File Name

Returns:

the aggregated log suffix String