org.apache.hadoop.yarn.server.nodemanager

Class LinuxContainerExecutor

java.lang.Object

org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class LinuxContainerExecutor
extends ContainerExecutor

This class provides Container execution using a native container-executor binary. By using a helper written it native code, this class is able to do several things that the DefaultContainerExecutor cannot, such as execution of applications as the applications' owners, provide localization that takes advantage of mapping the application owner to a UID on the execution host, resource management through Linux CGROUPS, and Docker support.

If hadoop.security.authetication is set to simple, then the yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users property will determine whether the LinuxContainerExecutor runs processes as the application owner or as the default user, as set in the yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user property.

The LinuxContainerExecutor will manage applications through an appropriate LinuxContainerRuntime instance. This class uses a DelegatingLinuxContainerRuntime instance, which will delegate calls to either a DefaultLinuxContainerRuntime instance or a DockerLinuxContainerRuntime instance, depending on the job's configuration.

See Also:

LinuxContainerRuntime, DelegatingLinuxContainerRuntime, DefaultLinuxContainerRuntime, DockerLinuxContainerRuntime, DockerLinuxContainerRuntime.isDockerContainerRequested(java.util.Map<java.lang.String, java.lang.String>)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LinuxContainerExecutor.ExitCode The container exit code.

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

ContainerExecutor.DelayedProcessKiller, ContainerExecutor.Signal

Field Summary

Fields inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

DIRECTORY_CONTENTS, TASK_LAUNCH_SCRIPT_PERMISSION, WILDCARD

Constructor Summary

Constructors

Constructor and Description
LinuxContainerExecutor() Default constructor to allow for creation through reflection.
LinuxContainerExecutor(LinuxContainerRuntime linuxContainerRuntime) Create a LinuxContainerExecutor with a provided LinuxContainerRuntime.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addSchedPriorityCommand(List<String> command) Add a niceness level to the process that will be executed.
void	buildMainArgs(List<String> command, String user, String appId, String locId, InetSocketAddress nmAddr, List<String> localDirs) Set up the ContainerLocalizer.
void	deleteAsUser(DeletionAsUserContext ctx) Delete specified directories as a given user.
protected String	getContainerExecutorExecutablePath(org.apache.hadoop.conf.Configuration conf) Get the path to the container-executor binary.
String[]	getIpAndHost(Container container)
protected PrivilegedOperationExecutor	getPrivilegedOperationExecutor()
void	init() Run the executor initialization steps.
boolean	isContainerAlive(ContainerLivenessContext ctx) Check if a container is alive.
int	launchContainer(ContainerStartContext ctx) Launch the container on the node.
void	mountCgroups(List<String> cgroupKVs, String hierarchy) Mount a CGROUPS controller at the requested mount point and create a hierarchy for the NodeManager to manage.
int	reacquireContainer(ContainerReacquisitionContext ctx) Recover an already existing container.
protected File[]	readDirAsUser(String user, org.apache.hadoop.fs.Path dir) Return the files in the target directory.
void	setConf(org.apache.hadoop.conf.Configuration conf)
boolean	signalContainer(ContainerSignalContext ctx) Signal container with the specified signal.
void	startLocalizer(LocalizerStartContext ctx) Prepare the environment for containers in this application to execute.
void	symLink(String target, String symlink) Create a symlink file which points to the target.
protected void	updateEnvForWhitelistVars(Map<String,String> env) Propagate variables from the nodemanager's environment into the container's environment if unspecified by the container.

Methods inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

activateContainer, deactivateContainer, getConf, getLocalIpAndHost, getNMEnvVar, getPidFilePath, getProcessId, getRunCommand, getRunCommand, getRunCommandForOther, getRunCommandForWindows, isContainerActive, localizeClasspathJar, logOutput, pauseContainer, resumeContainer, writeLaunchEnv, writeLaunchEnv

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LinuxContainerExecutor

public LinuxContainerExecutor()

Default constructor to allow for creation through reflection.

LinuxContainerExecutor

public LinuxContainerExecutor(LinuxContainerRuntime linuxContainerRuntime)

Create a LinuxContainerExecutor with a provided LinuxContainerRuntime. Used primarily for testing.

Parameters:

linuxContainerRuntime - the runtime to use

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class ContainerExecutor

getContainerExecutorExecutablePath

protected String getContainerExecutorExecutablePath(org.apache.hadoop.conf.Configuration conf)

Get the path to the container-executor binary. The path will be absolute.

Parameters:

conf - the Configuration

Returns:

the path to the container-executor binary

addSchedPriorityCommand

protected void addSchedPriorityCommand(List<String> command)

Add a niceness level to the process that will be executed. Adds -n <nice> to the given command. The niceness level will be taken from the yarn.nodemanager.container-executer.os.sched.prioity property.

Parameters:

command - the command to which to add the niceness setting.

getPrivilegedOperationExecutor

protected PrivilegedOperationExecutor getPrivilegedOperationExecutor()

init

public void init()
          throws IOException

Description copied from class: ContainerExecutor

Run the executor initialization steps. Verify that the necessary configs and permissions are in place.

Specified by:

init in class ContainerExecutor

Throws:

IOException - if initialization fails

startLocalizer

public void startLocalizer(LocalizerStartContext ctx)
                    throws IOException,
                           InterruptedException

Description copied from class: ContainerExecutor

Prepare the environment for containers in this application to execute.

 For $x in local.dirs
   create $x/$user/$appId
 Copy $nmLocal/appTokens -> $N/$user/$appId
 For $rsrc in private resources
   Copy $rsrc -> $N/$user/filecache/[idef]
 For $rsrc in job resources
   Copy $rsrc -> $N/$user/$appId/filecache/idef
 

Specified by:

startLocalizer in class ContainerExecutor

Parameters:

ctx - LocalizerStartContext that encapsulates necessary information for starting a localizer.

Throws:

IOException - for most application init failures

InterruptedException - if application init thread is halted by NM

buildMainArgs

public void buildMainArgs(List<String> command,
                 String user,
                 String appId,
                 String locId,
                 InetSocketAddress nmAddr,
                 List<String> localDirs)

Set up the ContainerLocalizer.

Parameters:

command - the current ShellCommandExecutor command line

user - localization user

appId - localized app id

locId - localizer id

nmAddr - nodemanager address

localDirs - list of local dirs

See Also:

ContainerLocalizer.buildMainArgs(java.util.List<java.lang.String>, java.lang.String, java.lang.String, java.lang.String, java.net.InetSocketAddress, java.util.List<java.lang.String>)

updateEnvForWhitelistVars

protected void updateEnvForWhitelistVars(Map<String,String> env)

Description copied from class: ContainerExecutor

Propagate variables from the nodemanager's environment into the container's environment if unspecified by the container.

Overrides:

updateEnvForWhitelistVars in class ContainerExecutor

Parameters:

env - the environment to update

See Also:

YarnConfiguration.NM_ENV_WHITELIST

launchContainer

public int launchContainer(ContainerStartContext ctx)
                    throws IOException,
                           org.apache.hadoop.yarn.exceptions.ConfigurationException

Description copied from class: ContainerExecutor

Launch the container on the node. This is a blocking call and returns only when the container exits.

Specified by:

launchContainer in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for launching containers.

Returns:

the return status of the launch

Throws:

IOException - if the container launch fails

org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

getIpAndHost

public String[] getIpAndHost(Container container)
                      throws ContainerExecutionException

Overrides:

getIpAndHost in class ContainerExecutor

Throws:

ContainerExecutionException

reacquireContainer

public int reacquireContainer(ContainerReacquisitionContext ctx)
                       throws IOException,
                              InterruptedException

Description copied from class: ContainerExecutor

Recover an already existing container. This is a blocking call and returns only when the container exits. Note that the container must have been activated prior to this call.

Overrides:

reacquireContainer in class ContainerExecutor

Parameters:

ctx - encapsulates information necessary to reacquire container

Returns:

The exit code of the pre-existing container

Throws:

IOException - if there is a failure while reacquiring the container

InterruptedException - if interrupted while waiting to reacquire the container

signalContainer

public boolean signalContainer(ContainerSignalContext ctx)
                        throws IOException

Description copied from class: ContainerExecutor

Signal container with the specified signal.

Specified by:

signalContainer in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for signaling containers.

Returns:

returns true if the operation succeeded

Throws:

IOException - if signaling the container fails

deleteAsUser

public void deleteAsUser(DeletionAsUserContext ctx)

Description copied from class: ContainerExecutor

Delete specified directories as a given user.

Specified by:

deleteAsUser in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for deletion.

readDirAsUser

protected File[] readDirAsUser(String user,
                   org.apache.hadoop.fs.Path dir)

Description copied from class: ContainerExecutor

Return the files in the target directory. If retrieving the list of files requires specific access rights, that access will happen as the specified user. The list will not include entries for "." or "..".

Overrides:

readDirAsUser in class ContainerExecutor

Parameters:

user - the user as whom to access the target directory

dir - the target directory

Returns:

a list of files in the target directory

symLink

public void symLink(String target,
           String symlink)

Description copied from class: ContainerExecutor

Create a symlink file which points to the target.

Specified by:

symLink in class ContainerExecutor

Parameters:

target - The target for symlink

symlink - the symlink file

isContainerAlive

public boolean isContainerAlive(ContainerLivenessContext ctx)
                         throws IOException

Description copied from class: ContainerExecutor

Check if a container is alive.

Specified by:

isContainerAlive in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for container liveness check.

Returns:

true if container is still alive

Throws:

IOException - if there is a failure while checking the container status

mountCgroups

public void mountCgroups(List<String> cgroupKVs,
                String hierarchy)
                  throws IOException

Mount a CGROUPS controller at the requested mount point and create a hierarchy for the NodeManager to manage.

Parameters:

cgroupKVs - a key-value pair of the form controller=mount-path

hierarchy - the top directory of the hierarchy for the NodeManager

Throws:

IOException - if there is a problem mounting the CGROUPS