@InterfaceAudience.Private @InterfaceStability.Unstable public class DockerLinuxContainerRuntime extends OCIContainerRuntime
This class is an extension of OCIContainerRuntime
that uses the
native container-executor
binary via a
PrivilegedOperationExecutor
instance to launch processes inside
Docker containers.
The following environment variables are used to configure the Docker engine:
YARN_CONTAINER_RUNTIME_TYPE
ultimately determines whether a
Docker container will be used. If the value is docker
, a Docker
container will be used. Otherwise a regular process tree container will
be used. This environment variable is checked by the
isDockerContainerRequested(org.apache.hadoop.conf.Configuration, java.util.Map<java.lang.String, java.lang.String>)
method, which is called by the
DelegatingLinuxContainerRuntime
.
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE
names which image
will be used to launch the Docker container.
YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE
controls
whether the Docker container's default command is overridden. When set
to true
, the Docker container's command will be
bash <path_to_launch_script>
. When unset or set to false
the Docker container's default command is used.
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK
sets the
network type to be used by the Docker container. It must be a valid
value as determined by the
yarn.nodemanager.runtime.linux.docker.allowed-container-networks
property.
YARN_CONTAINER_RUNTIME_DOCKER_PORTS_MAPPING
allows users to
specify ports mapping for the bridge network Docker container. The value
of the environment variable should be a comma-separated list of ports
mapping. It's the same to "-p" option for the Docker run command. If the
value is empty, "-P" will be added.
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_PID_NAMESPACE
controls which PID namespace will be used by the Docker container. By
default, each Docker container has its own PID namespace. To share the
namespace of the host, the
yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed
property must be set to true
. If the host PID namespace is
allowed and this environment variable is set to host
, the
Docker container will share the host's PID namespace. No other value is
allowed.
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_HOSTNAME
sets the
hostname to be used by the Docker container. If not specified, a
hostname will be derived from the container ID and set as default
hostname for networks other than 'host'.
YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER
controls whether the Docker container is a privileged container. In order
to use privileged containers, the
yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed
property must be set to true
, and the application owner must
appear in the value of the
yarn.nodemanager.runtime.linux.docker.privileged-containers.acl
property. If this environment variable is set to true
, a
privileged Docker container will be used if allowed. No other value is
allowed, so the environment variable should be left unset rather than
setting it to false.
YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS
allows users to specify
additional volume mounts for the Docker container. The value of the
environment variable should be a comma-separated list of mounts.
All such mounts must be given as source:dest[:mode]
and the mode
must be "ro" (read-only) or "rw" (read-write) to specify the type of
access being requested. If neither is specified, read-write will be
assumed. The mode may include a bind propagation option. In that case,
the mode should either be of the form [option], rw+[option], or
ro+[option]. Valid bind propagation options are shared, rshared, slave,
rslave, private, and rprivate. The requested mounts will be validated by
container-executor based on the values set in container-executor.cfg for
docker.allowed.ro-mounts
and docker.allowed.rw-mounts
.
YARN_CONTAINER_RUNTIME_DOCKER_TMPFS_MOUNTS
allows users to
specify additional tmpfs mounts for the Docker container. The value of
the environment variable should be a comma-separated list of mounts.
YARN_CONTAINER_RUNTIME_DOCKER_DELAYED_REMOVAL
allows a user
to request delayed deletion of the Docker containers on a per
container basis. If true, Docker containers will not be removed until
the duration defined by yarn.nodemanager.delete.debug-delay-sec
has elapsed. Administrators can disable this feature through the
yarn-site property
yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed
.
This feature is disabled by default. When this feature is disabled or set
to false, the container will be removed as soon as it exits.
YARN_CONTAINER_RUNTIME_YARN_SYSFS_ENABLE
allows export yarn
service json to docker container. This feature is disabled by default.
When this feature is set, app.json will be available in
/hadoop/yarn/sysfs/app.json.
Modifier and Type | Field and Description |
---|---|
static String |
DOCKER_IMAGE_PATTERN |
static String |
ENV_DOCKER_CONTAINER_DELAYED_REMOVAL |
static String |
ENV_DOCKER_CONTAINER_DOCKER_RUNTIME |
static String |
ENV_DOCKER_CONTAINER_DOCKER_SERVICE_MODE |
static String |
ENV_DOCKER_CONTAINER_HOSTNAME |
static String |
ENV_DOCKER_CONTAINER_IMAGE |
static String |
ENV_DOCKER_CONTAINER_MOUNTS |
static String |
ENV_DOCKER_CONTAINER_NETWORK |
static String |
ENV_DOCKER_CONTAINER_PORTS_MAPPING |
static String |
ENV_DOCKER_CONTAINER_TMPFS_MOUNTS |
static String |
ENV_DOCKER_CONTAINER_YARN_SYSFS |
static String |
ENV_OCI_CONTAINER_PID_NAMESPACE |
static String |
ENV_OCI_CONTAINER_RUN_PRIVILEGED_CONTAINER |
CONTAINER_PID_NAMESPACE_SUFFIX, RUN_PRIVILEGED_CONTAINER_SUFFIX, RUNTIME_PREFIX
Constructor and Description |
---|
DockerLinuxContainerRuntime(PrivilegedOperationExecutor privilegedOperationExecutor)
Create an instance using the given
PrivilegedOperationExecutor
instance for performing operations. |
DockerLinuxContainerRuntime(PrivilegedOperationExecutor privilegedOperationExecutor,
CGroupsHandler cGroupsHandler)
Create an instance using the given
PrivilegedOperationExecutor
instance for performing operations and the given CGroupsHandler
instance. |
Modifier and Type | Method and Description |
---|---|
protected void |
addCGroupParentIfRequired(String resourcesOptions,
String containerIdStr,
DockerRunCommand runCommand)
If CGROUPS in enabled and not set to none, then set the CGROUP parent for
the command instance.
|
org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair |
execContainer(ContainerExecContext ctx)
Perform docker exec command into running container.
|
Set<String> |
getCapabilities() |
String |
getExposedPorts(Container container)
Return the exposed ports of the container.
|
String[] |
getIpAndHost(Container container)
Return the host and ip of the container.
|
void |
initialize(org.apache.hadoop.conf.Configuration conf,
Context nmContext)
Initialize the runtime.
|
static boolean |
isDockerContainerRequested(org.apache.hadoop.conf.Configuration daemonConf,
Map<String,String> env)
Return whether the given environment variables indicate that the operation
is requesting a Docker container.
|
boolean |
isRuntimeRequested(Map<String,String> env)
Return whether the given environment variables indicate that the operation
is requesting this runtime.
|
void |
launchContainer(ContainerRuntimeContext ctx)
Launch a container.
|
void |
pullImageFromRemote(String containerIdStr,
String imageName) |
void |
reapContainer(ContainerRuntimeContext ctx)
Reap the docker container.
|
void |
relaunchContainer(ContainerRuntimeContext ctx)
Relaunch a container.
|
void |
signalContainer(ContainerRuntimeContext ctx)
Signal the docker container.
|
static void |
validateImageName(String imageName) |
allowHostPidNamespace, allowPrivilegedContainerExecution, formatOciEnvKey, getCsiClients, getGroupIdInfo, getUserIdInfo, initiateCsiClients, isContainerRequestedAsPrivileged, isOCICompliantContainerRequested, mountReadOnlyPath, prepareContainer, validateContainerNetworkType, validateContainerRuntimeType, validateHostname
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getLocalResources, start, stop
public static final String DOCKER_IMAGE_PATTERN
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_IMAGE
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_NETWORK
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_HOSTNAME
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_MOUNTS
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_TMPFS_MOUNTS
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_DELAYED_REMOVAL
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_PORTS_MAPPING
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_YARN_SYSFS
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_DOCKER_RUNTIME
@InterfaceAudience.Private public static final String ENV_DOCKER_CONTAINER_DOCKER_SERVICE_MODE
@InterfaceAudience.Private public static final String ENV_OCI_CONTAINER_PID_NAMESPACE
@InterfaceAudience.Private public static final String ENV_OCI_CONTAINER_RUN_PRIVILEGED_CONTAINER
public DockerLinuxContainerRuntime(PrivilegedOperationExecutor privilegedOperationExecutor)
PrivilegedOperationExecutor
instance for performing operations.privilegedOperationExecutor
- the PrivilegedOperationExecutor
instancepublic DockerLinuxContainerRuntime(PrivilegedOperationExecutor privilegedOperationExecutor, CGroupsHandler cGroupsHandler)
PrivilegedOperationExecutor
instance for performing operations and the given CGroupsHandler
instance. This constructor is intended for use in testing.privilegedOperationExecutor
- the PrivilegedOperationExecutor
instancecGroupsHandler
- the CGroupsHandler
instancepublic static boolean isDockerContainerRequested(org.apache.hadoop.conf.Configuration daemonConf, Map<String,String> env)
YARN_CONTAINER_RUNTIME_TYPE
whose value is docker
,
this method will return true. Otherwise it will return false.daemonConf
- the NodeManager daemon configurationenv
- the environment variable settings for the operationpublic void initialize(org.apache.hadoop.conf.Configuration conf, Context nmContext) throws ContainerExecutionException
LinuxContainerRuntime
initialize
in interface LinuxContainerRuntime
initialize
in class OCIContainerRuntime
conf
- the Configuration
to usenmContext
- NMContextContainerExecutionException
- if an error occurs while initializing
the runtimepublic boolean isRuntimeRequested(Map<String,String> env)
LinuxContainerRuntime
env
- the environment variable settings for the operationprotected void addCGroupParentIfRequired(String resourcesOptions, String containerIdStr, DockerRunCommand runCommand)
resourcesOptions
- the resource options to check for "cgroups=none"containerIdStr
- the container IDrunCommand
- the command to set with the CGROUP parentpublic void launchContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException
ContainerRuntime
ctx
- the ContainerRuntimeContext
ContainerExecutionException
- if an error occurs while launching
the containerpublic void relaunchContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException
ContainerRuntime
ctx
- the ContainerRuntimeContext
ContainerExecutionException
- if an error occurs while relaunching
the containerpublic void signalContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException
ctx
- the ContainerRuntimeContext
.ContainerExecutionException
- if the signaling fails.public void reapContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException
ctx
- the ContainerRuntimeContext
.ContainerExecutionException
- if the removal fails.public org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair execContainer(ContainerExecContext ctx) throws ContainerExecutionException
ctx
- container exec contextContainerExecutionException
public String[] getIpAndHost(Container container)
ContainerRuntime
container
- the Container
public String getExposedPorts(Container container)
ContainerRuntime
container
- the Container
public static void validateImageName(String imageName) throws ContainerExecutionException
ContainerExecutionException
public void pullImageFromRemote(String containerIdStr, String imageName) throws ContainerExecutionException
ContainerExecutionException
Copyright © 2008–2023 Apache Software Foundation. All rights reserved.