public class LinuxContainerExecutor extends ContainerExecutor
This class provides Container
execution using a native
container-executor
binary. By using a helper written it native code,
this class is able to do several things that the
DefaultContainerExecutor
cannot, such as execution of applications
as the applications' owners, provide localization that takes advantage of
mapping the application owner to a UID on the execution host, resource
management through Linux CGROUPS, and Docker support.
If hadoop.security.authentication
is set to simple
,
then the
yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users
property will determine whether the LinuxContainerExecutor
runs
processes as the application owner or as the default user, as set in the
yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user
property.
The LinuxContainerExecutor
will manage applications through an
appropriate LinuxContainerRuntime
instance. This class uses a
DelegatingLinuxContainerRuntime
instance, which will delegate calls
to either a DefaultLinuxContainerRuntime
instance or a
OCIContainerRuntime
instance, depending on the job's
configuration.
Modifier and Type | Class and Description |
---|---|
static class |
LinuxContainerExecutor.ExitCode
The container exit code.
|
ContainerExecutor.DelayedProcessKiller, ContainerExecutor.Signal
DIRECTORY_CONTENTS, TASK_LAUNCH_SCRIPT_PERMISSION, TOKEN_FILE_NAME_FMT, WILDCARD
Constructor and Description |
---|
LinuxContainerExecutor()
Default constructor to allow for creation through reflection.
|
LinuxContainerExecutor(LinuxContainerRuntime linuxContainerRuntime)
Create a LinuxContainerExecutor with a provided
LinuxContainerRuntime . |
Modifier and Type | Method and Description |
---|---|
protected void |
addSchedPriorityCommand(List<String> command)
Add a niceness level to the process that will be executed.
|
void |
buildMainArgs(List<String> command,
String user,
String appId,
String locId,
InetSocketAddress nmAddr,
String tokenFileName,
List<String> localDirs)
Set up the
ContainerLocalizer . |
void |
deleteAsUser(DeletionAsUserContext ctx)
Delete specified directories as a given user.
|
org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair |
execContainer(ContainerExecContext ctx)
Performs container exec.
|
protected String |
getContainerExecutorExecutablePath(org.apache.hadoop.conf.Configuration conf)
Get the path to the
container-executor binary. |
String |
getExposedPorts(Container container) |
String[] |
getIpAndHost(Container container) |
Map<String,org.apache.hadoop.yarn.api.records.LocalResource> |
getLocalResources(Container container) |
protected PrivilegedOperationExecutor |
getPrivilegedOperationExecutor() |
ResourceHandler |
getResourceHandler() |
void |
init(Context context)
Run the executor initialization steps.
|
boolean |
isContainerAlive(ContainerLivenessContext ctx)
Check if a container is alive.
|
int |
launchContainer(ContainerStartContext ctx)
Launch the container on the node.
|
void |
mountCgroups(List<String> cgroupKVs,
String hierarchy)
Mount a CGROUPS controller at the requested mount point and create
a hierarchy for the NodeManager to manage.
|
void |
prepareContainer(ContainerPrepareContext ctx)
Prepare the container prior to the launch environment being written.
|
int |
reacquireContainer(ContainerReacquisitionContext ctx)
Recover an already existing container.
|
protected File[] |
readDirAsUser(String user,
org.apache.hadoop.fs.Path dir)
Return the files in the target directory.
|
boolean |
reapContainer(ContainerReapContext ctx)
Performs the tasks necessary to reap the container.
|
int |
relaunchContainer(ContainerStartContext ctx)
Relaunch the container on the node.
|
void |
removeDockerContainer(String containerId)
Remove the docker container referenced in the context.
|
void |
setConf(org.apache.hadoop.conf.Configuration conf) |
boolean |
signalContainer(ContainerSignalContext ctx)
Signal container with the specified signal.
|
void |
start() |
void |
startLocalizer(LocalizerStartContext ctx)
Prepare the environment for containers in this application to execute.
|
void |
stop() |
void |
symLink(String target,
String symlink)
Create a symlink file which points to the target.
|
void |
updateYarnSysFS(Context ctx,
String user,
String appId,
String spec)
Update cluster information inside container.
|
activateContainer, cleanupBeforeRelaunch, deactivateContainer, getConf, getLocalIpAndHost, getNMEnvVar, getPidFilePath, getProcessId, getRunCommand, getRunCommand, getRunCommandForOther, getRunCommandForWindows, isContainerActive, localizeClasspathJar, logOutput, pauseContainer, resumeContainer, writeLaunchEnv, writeLaunchEnv
public LinuxContainerExecutor()
public LinuxContainerExecutor(LinuxContainerRuntime linuxContainerRuntime)
LinuxContainerRuntime
. Used primarily for testing.linuxContainerRuntime
- the runtime to usepublic void setConf(org.apache.hadoop.conf.Configuration conf)
setConf
in interface org.apache.hadoop.conf.Configurable
setConf
in class ContainerExecutor
protected String getContainerExecutorExecutablePath(org.apache.hadoop.conf.Configuration conf)
container-executor
binary. The path will
be absolute.conf
- the Configuration
container-executor
binaryprotected void addSchedPriorityCommand(List<String> command)
-n <nice>
to the given command. The niceness level will be
taken from the
YarnConfiguration.NM_CONTAINER_EXECUTOR_SCHED_PRIORITY
property.command
- the command to which to add the niceness setting.protected PrivilegedOperationExecutor getPrivilegedOperationExecutor()
public void init(Context context) throws IOException
ContainerExecutor
init
in class ContainerExecutor
context
- Context of NMIOException
- if initialization failspublic void start()
start
in class ContainerExecutor
public void stop()
stop
in class ContainerExecutor
public void startLocalizer(LocalizerStartContext ctx) throws IOException, InterruptedException
ContainerExecutor
For $x in local.dirs create $x/$user/$appId Copy $nmLocal/appTokens -> $N/$user/$appId For $rsrc in private resources Copy $rsrc -> $N/$user/filecache/[idef] For $rsrc in job resources Copy $rsrc -> $N/$user/$appId/filecache/idef
startLocalizer
in class ContainerExecutor
ctx
- LocalizerStartContext that encapsulates necessary information
for starting a localizer.IOException
- for most application init failuresInterruptedException
- if application init thread is halted by NMpublic void buildMainArgs(List<String> command, String user, String appId, String locId, InetSocketAddress nmAddr, String tokenFileName, List<String> localDirs)
ContainerLocalizer
.command
- the current ShellCommandExecutor command lineuser
- localization userappId
- localized app idlocId
- localizer idnmAddr
- nodemanager addresslocalDirs
- list of local dirsContainerLocalizer.buildMainArgs(java.util.List<java.lang.String>, java.lang.String, java.lang.String, java.lang.String, java.net.InetSocketAddress, java.lang.String, java.util.List<java.lang.String>, org.apache.hadoop.conf.Configuration)
public void prepareContainer(ContainerPrepareContext ctx) throws IOException
ContainerExecutor
prepareContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for launching containers.IOException
- if errors occur during container preparationpublic int launchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
ContainerExecutor
launchContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for launching containers.IOException
- if the container launch failsorg.apache.hadoop.yarn.exceptions.ConfigurationException
- if config error was foundpublic int relaunchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
ContainerExecutor
relaunchContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for relaunching containers.IOException
- if the container relaunch failsorg.apache.hadoop.yarn.exceptions.ConfigurationException
- if config error was foundpublic String[] getIpAndHost(Container container) throws ContainerExecutionException
getIpAndHost
in class ContainerExecutor
ContainerExecutionException
public int reacquireContainer(ContainerReacquisitionContext ctx) throws IOException, InterruptedException
ContainerExecutor
reacquireContainer
in class ContainerExecutor
ctx
- encapsulates information necessary to reacquire containerIOException
- if there is a failure while reacquiring the containerInterruptedException
- if interrupted while waiting to reacquire
the containerpublic boolean signalContainer(ContainerSignalContext ctx) throws IOException
ContainerExecutor
signalContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for signaling containers.IOException
- if signaling the container failspublic boolean reapContainer(ContainerReapContext ctx) throws IOException
reapContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for reaping containers.IOException
- if an error occurs while reaping the container.public org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair execContainer(ContainerExecContext ctx) throws ContainerExecutionException
execContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for exec container.ContainerExecutionException
- if container exec fails.public void deleteAsUser(DeletionAsUserContext ctx)
ContainerExecutor
deleteAsUser
in class ContainerExecutor
ctx
- Encapsulates information necessary for deletion.protected File[] readDirAsUser(String user, org.apache.hadoop.fs.Path dir)
ContainerExecutor
readDirAsUser
in class ContainerExecutor
user
- the user as whom to access the target directorydir
- the target directorypublic void symLink(String target, String symlink)
ContainerExecutor
symLink
in class ContainerExecutor
target
- The target for symlinksymlink
- the symlink filepublic boolean isContainerAlive(ContainerLivenessContext ctx) throws IOException
ContainerExecutor
isContainerAlive
in class ContainerExecutor
ctx
- Encapsulates information necessary for container liveness check.IOException
- if there is a failure while checking the container
statuspublic void mountCgroups(List<String> cgroupKVs, String hierarchy) throws IOException
cgroupKVs
- a key-value pair of the form
controller=mount-path
hierarchy
- the top directory of the hierarchy for the NodeManagerIOException
- if there is a problem mounting the CGROUPSpublic ResourceHandler getResourceHandler()
public void removeDockerContainer(String containerId)
containerId
- the containerId for the container.public void updateYarnSysFS(Context ctx, String user, String appId, String spec) throws IOException
ContainerExecutor
updateYarnSysFS
in class ContainerExecutor
ctx
- ContainerRuntimeContextuser
- Owner of applicationappId
- YARN application IDspec
- Service SpecificationIOException
- if there is a failure while writing spec to diskpublic String getExposedPorts(Container container) throws ContainerExecutionException
getExposedPorts
in class ContainerExecutor
ContainerExecutionException
public Map<String,org.apache.hadoop.yarn.api.records.LocalResource> getLocalResources(Container container) throws IOException
getLocalResources
in class ContainerExecutor
IOException
Copyright © 2008–2023 Apache Software Foundation. All rights reserved.