org.apache.hadoop.yarn.api.records

Class LogAggregationContext

java.lang.Object

org.apache.hadoop.yarn.api.records.LogAggregationContext

@InterfaceStability.Evolving
 @InterfaceAudience.Public
public abstract class LogAggregationContext
extends Object

LogAggregationContext represents all of the information needed by the NodeManager to handle the logs for an application.

It includes details such as:

• includePattern. It uses Java Regex to filter the log files which match the defined include pattern and those log files will be uploaded when the application finishes.
• excludePattern. It uses Java Regex to filter the log files which match the defined exclude pattern and those log files will not be uploaded when application finishes. If the log file name matches both the include and the exclude pattern, this file will be excluded eventually.
• rolledLogsIncludePattern. It uses Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion.
• rolledLogsExcludePattern. It uses Java Regex to filter the log files which match the defined exclude pattern and those log files will not be aggregated in a rolling fashion. If the log file name matches both the include and the exclude pattern, this file will be excluded eventually.
• policyClassName. The policy class name that implements ContainerLogAggregationPolicy. At runtime, nodemanager will the policy if a given container's log should be aggregated based on the ContainerType and other runtime state such as exit code by calling ContainerLogAggregationPolicy#shouldDoLogAggregation. This is useful when the app only wants to aggregate logs of a subset of containers. Here are the available policies. Please make sure to specify the canonical name by prefixing org.apache.hadoop.yarn.server. nodemanager.containermanager.logaggregation. to the class simple name below. NoneContainerLogAggregationPolicy: skip aggregation for all containers. AllContainerLogAggregationPolicy: aggregate all containers. AMOrFailedContainerLogAggregationPolicy: aggregate application master or failed containers. FailedOrKilledContainerLogAggregationPolicy: aggregate failed or killed containers FailedContainerLogAggregationPolicy: aggregate failed containers AMOnlyLogAggregationPolicy: aggregate application master containers SampleContainerLogAggregationPolicy: sample logs of successful worker containers, in addition to application master and failed/killed containers. LimitSizeContainerLogAggregationPolicy: skip aggregation for killed containers whose log size exceeds the limit of container log size. If it isn't specified, it will use the cluster-wide default policy defined by configuration yarn.nodemanager.log-aggregation.policy.class. The default value of yarn.nodemanager.log-aggregation.policy.class is AllContainerLogAggregationPolicy.
• policyParameters. The parameters passed to the policy class via ContainerLogAggregationPolicy#parseParameters during the policy object initialization. This is optional. Some policy class might use parameters to adjust its settings. It is up to policy class to define the scheme of parameters. For example, SampleContainerLogAggregationPolicy supports the format of "SR:0.5,MIN:50", which means sample rate of 50% beyond the first 50 successful worker containers.

See Also:

ApplicationSubmissionContext

Constructor Summary

Constructors

Constructor and Description
LogAggregationContext()

Method Summary

Modifier and Type	Method and Description
abstract String	getExcludePattern() Get exclude pattern.
abstract String	getIncludePattern() Get include pattern.
abstract String	getLogAggregationPolicyClassName() Get the log aggregation policy class.
abstract String	getLogAggregationPolicyParameters() Get the log aggregation policy parameters.
abstract String	getRolledLogsExcludePattern() Get exclude pattern for aggregation in a rolling fashion.
abstract String	getRolledLogsIncludePattern() Get include pattern in a rolling fashion.
static LogAggregationContext	newInstance(String includePattern, String excludePattern)
static LogAggregationContext	newInstance(String includePattern, String excludePattern, String rolledLogsIncludePattern, String rolledLogsExcludePattern)
static LogAggregationContext	newInstance(String includePattern, String excludePattern, String rolledLogsIncludePattern, String rolledLogsExcludePattern, String policyClassName, String policyParameters)
abstract void	setExcludePattern(String excludePattern) Set exclude pattern.
abstract void	setIncludePattern(String includePattern) Set include pattern.
abstract void	setLogAggregationPolicyClassName(String className) Set the log aggregation policy class.
abstract void	setLogAggregationPolicyParameters(String parameters) Set the log aggregation policy parameters.
abstract void	setRolledLogsExcludePattern(String rolledLogsExcludePattern) Set exclude pattern for in a rolling fashion.
abstract void	setRolledLogsIncludePattern(String rolledLogsIncludePattern) Set include pattern in a rolling fashion.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LogAggregationContext

public LogAggregationContext()

Method Detail

newInstance

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public static LogAggregationContext newInstance(String includePattern,
                                                                                                        String excludePattern)

newInstance

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public static LogAggregationContext newInstance(String includePattern,
                                                                                                        String excludePattern,
                                                                                                        String rolledLogsIncludePattern,
                                                                                                        String rolledLogsExcludePattern)

newInstance

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public static LogAggregationContext newInstance(String includePattern,
                                                                                                        String excludePattern,
                                                                                                        String rolledLogsIncludePattern,
                                                                                                        String rolledLogsExcludePattern,
                                                                                                        String policyClassName,
                                                                                                        String policyParameters)

getIncludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract String getIncludePattern()

Get include pattern. This includePattern only takes affect on logs that exist at the time of application finish.

Returns:

include pattern

setIncludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract void setIncludePattern(String includePattern)

Set include pattern. This includePattern only takes affect on logs that exist at the time of application finish.

Parameters:

includePattern - include pattern.

getExcludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract String getExcludePattern()

Get exclude pattern. This excludePattern only takes affect on logs that exist at the time of application finish.

Returns:

exclude pattern

setExcludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract void setExcludePattern(String excludePattern)

Set exclude pattern. This excludePattern only takes affect on logs that exist at the time of application finish.

Parameters:

excludePattern - exclude pattern.

getRolledLogsIncludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract String getRolledLogsIncludePattern()

Get include pattern in a rolling fashion.

Returns:

include pattern

setRolledLogsIncludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract void setRolledLogsIncludePattern(String rolledLogsIncludePattern)

Set include pattern in a rolling fashion.

Parameters:

rolledLogsIncludePattern - It uses Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion.

getRolledLogsExcludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract String getRolledLogsExcludePattern()

Get exclude pattern for aggregation in a rolling fashion.

Returns:

exclude pattern

setRolledLogsExcludePattern

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract void setRolledLogsExcludePattern(String rolledLogsExcludePattern)

Set exclude pattern for in a rolling fashion.

Parameters:

rolledLogsExcludePattern - rolled logs exclude pattern.

getLogAggregationPolicyClassName

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract String getLogAggregationPolicyClassName()

Get the log aggregation policy class.

Returns:

log aggregation policy class

setLogAggregationPolicyClassName

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract void setLogAggregationPolicyClassName(String className)

Set the log aggregation policy class.

Parameters:

className - log aggregation policy class name.

getLogAggregationPolicyParameters

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract String getLogAggregationPolicyParameters()

Get the log aggregation policy parameters.

Returns:

log aggregation policy parameters

setLogAggregationPolicyParameters

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract void setLogAggregationPolicyParameters(String parameters)

Set the log aggregation policy parameters. There is no schema defined for the parameters string. It is up to the log aggregation policy class to decide how to parse the parameters string.

Parameters:

parameters - log aggregation policy parameters.