public class DefaultContainerExecutor extends ContainerExecutor
DefaultContainerExecuter
class offers generic container
execution services. Process execution is handled in a platform-independent
way via ProcessBuilder
.Modifier and Type | Class and Description |
---|---|
protected class |
DefaultContainerExecutor.LocalWrapperScriptBuilder
This class is a utility to create a wrapper script that is platform
appropriate.
|
ContainerExecutor.DelayedProcessKiller, ContainerExecutor.ExitCode, ContainerExecutor.Signal
Modifier and Type | Field and Description |
---|---|
protected org.apache.hadoop.fs.FileContext |
lfs
A
FileContext for the local file system. |
DIRECTORY_CONTENTS, TASK_LAUNCH_SCRIPT_PERMISSION, TOKEN_FILE_NAME_FMT, WILDCARD
Constructor and Description |
---|
DefaultContainerExecutor()
Default constructor for use in testing.
|
Modifier and Type | Method and Description |
---|---|
protected org.apache.hadoop.util.Shell.CommandExecutor |
buildCommandExecutor(String wrapperScriptPath,
String containerIdStr,
String user,
org.apache.hadoop.fs.Path pidFile,
org.apache.hadoop.yarn.api.records.Resource resource,
File workDir,
Map<String,String> environment)
Create a new
Shell.ShellCommandExecutor using the parameters. |
void |
clearLogDirPermissions()
Clear the internal variable for repeatable testing.
|
static boolean |
containerIsAlive(String pid)
Returns true if the process with the specified pid is alive.
|
protected void |
copyFile(org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst,
String owner)
Copy a file using the
lfs FileContext . |
protected ContainerLocalizer |
createContainerLocalizer(String user,
String appId,
String locId,
String tokenFileName,
List<String> localDirs,
org.apache.hadoop.fs.FileContext localizerFc)
Create a new
ContainerLocalizer instance. |
protected void |
createDir(org.apache.hadoop.fs.Path dirPath,
org.apache.hadoop.fs.permission.FsPermission perms,
boolean createParent,
String user)
Use the
lfs FileContext to create the target directory. |
void |
deleteAsUser(DeletionAsUserContext ctx)
Delete specified directories as a given user.
|
org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair |
execContainer(ContainerExecContext ctx)
Perform interactive docker command into running container.
|
protected DefaultContainerExecutor.LocalWrapperScriptBuilder |
getLocalWrapperScriptBuilder(String containerIdStr,
org.apache.hadoop.fs.Path containerWorkDir)
Create a
DefaultContainerExecutor.LocalWrapperScriptBuilder for the given container ID
and path that is appropriate to the current platform. |
String |
getLogDirPermissions()
Return the default container log directory permissions.
|
protected org.apache.hadoop.fs.Path |
getWorkingDir(List<String> localDirs,
String user,
String appId)
Return a randomly chosen application directory from a list of local storage
directories.
|
void |
init(Context nmContext)
Run the executor initialization steps.
|
boolean |
isContainerAlive(ContainerLivenessContext ctx)
Check if a container is alive.
|
protected void |
killContainer(String pid,
ContainerExecutor.Signal signal)
Send a specified signal to the specified pid
|
int |
launchContainer(ContainerStartContext ctx)
Launch the container on the node.
|
boolean |
reapContainer(ContainerReapContext ctx)
No-op for reaping containers within the DefaultContainerExecutor.
|
int |
relaunchContainer(ContainerStartContext ctx)
Relaunch the container on the node.
|
protected void |
setScriptExecutable(org.apache.hadoop.fs.Path script,
String owner)
Make a file executable using the
lfs FileContext . |
boolean |
signalContainer(ContainerSignalContext ctx)
Signal container with the specified signal.
|
void |
startLocalizer(LocalizerStartContext ctx)
Prepare the environment for containers in this application to execute.
|
void |
symLink(String target,
String symlink)
Create a symlink file which points to the target.
|
void |
updateYarnSysFS(Context ctx,
String user,
String appId,
String spec)
Update cluster information inside container.
|
activateContainer, cleanupBeforeRelaunch, deactivateContainer, getConf, getExposedPorts, getIpAndHost, getLocalIpAndHost, getLocalResources, getNMEnvVar, getPidFilePath, getProcessId, getRunCommand, getRunCommand, getRunCommandForOther, getRunCommandForWindows, isContainerActive, localizeClasspathJar, logOutput, pauseContainer, prepareContainer, reacquireContainer, readDirAsUser, resumeContainer, setConf, start, stop, writeLaunchEnv, writeLaunchEnv
protected final org.apache.hadoop.fs.FileContext lfs
FileContext
for the local file system.public DefaultContainerExecutor()
protected void copyFile(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, String owner) throws IOException
lfs
FileContext
.src
- the file to copydst
- where to copy the fileowner
- the owner of the new copy. Used only in secure Windows
clustersIOException
- when the copy failsWindowsSecureContainerExecutor
protected void setScriptExecutable(org.apache.hadoop.fs.Path script, String owner) throws IOException
lfs
FileContext
.script
- the path to make executableowner
- the new owner for the file. Used only in secure Windows
clustersIOException
- when the change mode operation failsWindowsSecureContainerExecutor
public void init(Context nmContext) throws IOException
ContainerExecutor
init
in class ContainerExecutor
nmContext
- Context of NMIOException
- if initialization failspublic void startLocalizer(LocalizerStartContext ctx) throws IOException, InterruptedException
ContainerExecutor
For $x in local.dirs create $x/$user/$appId Copy $nmLocal/appTokens -> $N/$user/$appId For $rsrc in private resources Copy $rsrc -> $N/$user/filecache/[idef] For $rsrc in job resources Copy $rsrc -> $N/$user/$appId/filecache/idef
startLocalizer
in class ContainerExecutor
ctx
- LocalizerStartContext that encapsulates necessary information
for starting a localizer.IOException
- for most application init failuresInterruptedException
- if application init thread is halted by NM@InterfaceAudience.Private protected ContainerLocalizer createContainerLocalizer(String user, String appId, String locId, String tokenFileName, List<String> localDirs, org.apache.hadoop.fs.FileContext localizerFc) throws IOException
ContainerLocalizer
instance.user
- the user who owns the job for which the localization is being
runappId
- the ID of the application for which the localization is being
runlocId
- the ID of the container for which the localization is being
runlocalDirs
- a list of directories to use as destinations for the
localizationlocalizerFc
- the FileContext
to use when localizing filesContainerLocalizer
instanceIOException
- if user
or locId
is null
or if
the container localizer has an initialization failurepublic int launchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
ContainerExecutor
launchContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for launching containers.IOException
- if the container launch failsorg.apache.hadoop.yarn.exceptions.ConfigurationException
- if config error was foundpublic int relaunchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
ContainerExecutor
relaunchContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for relaunching containers.IOException
- if the container relaunch failsorg.apache.hadoop.yarn.exceptions.ConfigurationException
- if config error was foundprotected org.apache.hadoop.util.Shell.CommandExecutor buildCommandExecutor(String wrapperScriptPath, String containerIdStr, String user, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.yarn.api.records.Resource resource, File workDir, Map<String,String> environment)
Shell.ShellCommandExecutor
using the parameters.wrapperScriptPath
- the path to the script to executecontainerIdStr
- the container IDuser
- the application owner's usernamepidFile
- the path to the container's PID fileresource
- this parameter controls memory and CPU limits.workDir
- If not-null, specifies the directory which should be set
as the current working directory for the command. If null,
the current working directory is not modified.environment
- the container environmentShell.ShellCommandExecutor
Shell.ShellCommandExecutor
protected DefaultContainerExecutor.LocalWrapperScriptBuilder getLocalWrapperScriptBuilder(String containerIdStr, org.apache.hadoop.fs.Path containerWorkDir)
DefaultContainerExecutor.LocalWrapperScriptBuilder
for the given container ID
and path that is appropriate to the current platform.containerIdStr
- the container IDcontainerWorkDir
- the container's working directoryDefaultContainerExecutor.LocalWrapperScriptBuilder
public boolean signalContainer(ContainerSignalContext ctx) throws IOException
ContainerExecutor
signalContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for signaling containers.IOException
- if signaling the container failspublic boolean reapContainer(ContainerReapContext ctx)
reapContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for reaping containers.public boolean isContainerAlive(ContainerLivenessContext ctx) throws IOException
ContainerExecutor
isContainerAlive
in class ContainerExecutor
ctx
- Encapsulates information necessary for container liveness check.IOException
- if there is a failure while checking the container
statuspublic static boolean containerIsAlive(String pid) throws IOException
pid
- String pidIOException
- if the command to test process liveliness failsprotected void killContainer(String pid, ContainerExecutor.Signal signal) throws IOException
pid
- the pid of the process [group] to signal.signal
- signal to sendIOException
- if the command to kill the process failspublic void deleteAsUser(DeletionAsUserContext ctx) throws IOException, InterruptedException
ContainerExecutor
deleteAsUser
in class ContainerExecutor
ctx
- Encapsulates information necessary for deletion.IOException
- if delete failsInterruptedException
- if interrupted while waiting for the deletion
operation to completepublic void symLink(String target, String symlink) throws IOException
ContainerExecutor
symLink
in class ContainerExecutor
target
- The target for symlinksymlink
- the symlink fileIOException
- Error when creating symlinksprotected org.apache.hadoop.fs.Path getWorkingDir(List<String> localDirs, String user, String appId) throws IOException
localDirs
- the target directories from which to selectuser
- the user who owns the applicationappId
- the application IDIOException
- if no application directories for the user can be
foundprotected void createDir(org.apache.hadoop.fs.Path dirPath, org.apache.hadoop.fs.permission.FsPermission perms, boolean createParent, String user) throws IOException
lfs
FileContext
to create the target directory.dirPath
- the target directoryperms
- the target permissions for the target directorycreateParent
- whether the parent directories should also be createduser
- the user as whom the target directory should be created.
Used only on secure Windows hosts.IOException
- if there's a failure performing a file operationWindowsSecureContainerExecutor
public String getLogDirPermissions()
public void clearLogDirPermissions()
public org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair execContainer(ContainerExecContext ctx) throws ContainerExecutionException
ContainerExecutor
execContainer
in class ContainerExecutor
ctx
- Encapsulates information necessary for exec containers.ContainerExecutionException
public void updateYarnSysFS(Context ctx, String user, String appId, String spec) throws IOException
ContainerExecutor
updateYarnSysFS
in class ContainerExecutor
ctx
- ContainerRuntimeContextuser
- Owner of applicationappId
- YARN application IDspec
- Service SpecificationIOException
- if there is a failure while writing spec to diskCopyright © 2008–2023 Apache Software Foundation. All rights reserved.