Shell (Apache Hadoop Main 2.9.2 API)

java.lang.Object
- org.apache.hadoop.util.Shell

```
@InterfaceAudience.Public
@InterfaceStability.Evolving
public abstract class Shell
extends Object
```
A base class for running a Shell command. Shell can be used to run shell commands like du or df. It also offers facilities to gate commands by time-intervals.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`ENV_HADOOP_HOME` Environment variable for Hadoop's home dir: "HADOOP_HOME".
`static String`	`ENV_NAME_REGEX` Regular expression for environment variables: "[A-Za-z_][A-Za-z0-9_]*".
`static boolean`	`FREEBSD`
`protected boolean`	`inheritParentEnv` Indicates if the parent env vars should be inherited or not
`static boolean`	`isSetsidAvailable` Flag which is true if setsid exists.
`static String`	`LINK_COMMAND` a Unix command to create a link: "ln".
`static boolean`	`LINUX`
`static org.slf4j.Logger`	`LOG`
`static boolean`	`MAC`
`static org.apache.hadoop.util.Shell.OSType`	`osType` Get the type of the operating system, as determined from parsing the `os.name` property.
`static boolean`	`OTHER`
`static boolean`	`PPC_64`
`static String`	`READ_LINK_COMMAND` a Unix command to get a link target: "readlink".
`static String`	`SET_GROUP_COMMAND` a Unix command to set the change user's groups list: "chgrp".
`static String`	`SET_OWNER_COMMAND` a Unix command to set owner: "chown".
`static String`	`SET_PERMISSION_COMMAND` a Unix command to set permission: "chmod".
`static boolean`	`SOLARIS`
`static String`	`SYSPROP_HADOOP_HOME_DIR` System property for the Hadoop home directory: "hadoop.home.dir".
`protected long`	`timeOutInterval` Time after which the executing script would be timedout.
`static String`	`TOKEN_SEPARATOR_REGEX` Token separator regex used to parse Shell tool outputs.
`static String`	`USER_NAME_COMMAND` a Unix command to get the current user's name: "whoami".
`static boolean`	`WINDOWS`
`static int`	`WINDOWS_MAX_SHELL_LENGHT` Deprecated. use the correctly spelled constant.
`static int`	`WINDOWS_MAX_SHELL_LENGTH` Maximum command line length in Windows KB830473 documents this as 8191
`static Object`	`WindowsProcessLaunchLock` Windows `CreateProcess` synchronization object.
`static String`	`WINUTILS` Deprecated. use one of the exception-raising getter methods, specifically `getWinUtilsPath()` or `getWinUtilsFile()`

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`Shell()` Create an instance with no minimum interval between runs; stderr is not merged with stdout.
`protected`	`Shell(long interval)` Create an instance with a minimum interval between executions; stderr is not merged with stdout.
`protected`	`Shell(long interval, boolean redirectErrorStream)` Create a shell instance which can be re-executed when the `run()` method is invoked with a given elapsed time between calls.

Method Summary

Methods
Modifier and Type	Method and Description
`static File`	`appendScriptExtension(File parent, String basename)` Returns a File referencing a script with the given basename, inside the given parent directory.
`static String`	`appendScriptExtension(String basename)` Returns a script file name with the given basename.
`static boolean`	`checkIsBashSupported()`
`static void`	`checkWindowsCommandLineLength(String... commands)` Checks if a given command (String[]) fits in the Windows maximum command line length Note that the input is expected to already include space delimiters, no extra count will be added for delimiters.
`static void`	`destroyAllShellProcesses()` Static method to destroy all running `Shell` processes.
`static String`	`execCommand(Map<String,String> env, String... cmd)` Static method to execute a shell command.
`static String`	`execCommand(Map<String,String> env, String[] cmd, long timeout)` Static method to execute a shell command.
`static String`	`execCommand(String... cmd)` Static method to execute a shell command.
`static Set<Shell>`	`getAllShells()` Static method to return a Set of all `Shell` objects.
`static String[]`	`getCheckProcessIsAliveCommand(String pid)` Return a command for determining if process with specified pid is alive.
`String`	`getEnvironment(String env)` Get an environment variable.
`static String`	`getEnvironmentVariableRegex()` Return a regular expression string that match environment variables.
`protected abstract String[]`	`getExecString()` return an array containing the command name and its parameters.
`int`	`getExitCode()` get the exit code.
`static String[]`	`getGetPermissionCommand()` Return a command to get permission information.
`static String[]`	`getGroupsCommand()` a Unix command to get the current user's groups list.
`static String[]`	`getGroupsForUserCommand(String user)` A command to get a given user's groups list.
`static String[]`	`getGroupsIDForUserCommand(String user)` A command to get a given user's group id list.
`static String`	`getHadoopHome()` Get the Hadoop home directory.
`static Long`	`getMemlockLimit(Long ulimit)` Static method to return the memory lock limit for datanode.
`Process`	`getProcess()` get the current sub-process executing the given command.
`static File`	`getQualifiedBin(String executable)` Fully qualify the path to a binary that should be in a known hadoop bin location.
`static String`	`getQualifiedBinPath(String executable)` Fully qualify the path to a binary that should be in a known hadoop bin location.
`static String[]`	`getReadlinkCommand(String link)` Return a command to read the target of the a symbolic link.
`static String[]`	`getRunScriptCommand(File script)` Returns a command to run the given script.
`static String[]`	`getSetOwnerCommand(String owner)` Return a command to set owner.
`static String[]`	`getSetPermissionCommand(String perm, boolean recursive)` Return a command to set permission.
`static String[]`	`getSetPermissionCommand(String perm, boolean recursive, String file)` Return a command to set permission for specific file.
`static String[]`	`getSignalKillCommand(int code, String pid)` Return a command to send a signal to a given pid.
`static String[]`	`getSymlinkCommand(String target, String link)` Return a command to create symbolic links.
`static String[]`	`getUsersForNetgroupCommand(String netgroup)` A command to get a given netgroup's user list.
`Thread`	`getWaitingThread()` get the thread that is waiting on this instance of `Shell`.
`static File`	`getWinUtilsFile()` Get a file reference to winutils.
`static String`	`getWinUtilsPath()` Locate the winutils binary, or fail with a meaningful exception and stack trace as an RTE.
`static boolean`	`hasWinutilsPath()` Predicate to indicate whether or not the path to winutils is known.
`static boolean`	`isJava7OrAbove()` Deprecated. This call isn't needed any more: please remove uses of it.
`boolean`	`isTimedOut()` To check if the passed script to shell command executor timed out or not.
`protected abstract void`	`parseExecResult(BufferedReader lines)` Parse the execution result
`protected void`	`run()` Check to see if a command needs to be executed and execute if needed.
`protected void`	`setEnvironment(Map<String,String> env)` Set the environment for the command.
`protected void`	`setWorkingDirectory(File dir)` Set the working directory.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - LOG
```
public static final org.slf4j.Logger LOG
```
  - SYSPROP_HADOOP_HOME_DIR
```
public static final String SYSPROP_HADOOP_HOME_DIR
```
    System property for the Hadoop home directory: "hadoop.home.dir".
    
    See Also:
    Constant Field Values
  - ENV_HADOOP_HOME
```
public static final String ENV_HADOOP_HOME
```
    Environment variable for Hadoop's home dir: "HADOOP_HOME".
    
    See Also:
    Constant Field Values
  - WINDOWS_MAX_SHELL_LENGTH
```
public static final int WINDOWS_MAX_SHELL_LENGTH
```
    Maximum command line length in Windows KB830473 documents this as 8191
    
    See Also:
    Constant Field Values
  - WINDOWS_MAX_SHELL_LENGHT
```
@Deprecated
public static final int WINDOWS_MAX_SHELL_LENGHT
```
    Deprecated. use the correctly spelled constant.
    
    mis-spelling of WINDOWS_MAX_SHELL_LENGTH.
    
    See Also:
    Constant Field Values
  - USER_NAME_COMMAND
```
public static final String USER_NAME_COMMAND
```
    a Unix command to get the current user's name: "whoami".
    
    See Also:
    Constant Field Values
  - WindowsProcessLaunchLock
```
public static final Object WindowsProcessLaunchLock
```
    Windows CreateProcess synchronization object.
  - osType
```
public static final org.apache.hadoop.util.Shell.OSType osType
```
    Get the type of the operating system, as determined from parsing the os.name property.
  - WINDOWS
```
public static final boolean WINDOWS
```
  - SOLARIS
```
public static final boolean SOLARIS
```
  - MAC
```
public static final boolean MAC
```
  - FREEBSD
```
public static final boolean FREEBSD
```
  - LINUX
```
public static final boolean LINUX
```
  - OTHER
```
public static final boolean OTHER
```
  - PPC_64
```
public static final boolean PPC_64
```
  - ENV_NAME_REGEX
```
public static final String ENV_NAME_REGEX
```
    Regular expression for environment variables: "[A-Za-z_][A-Za-z0-9_]*".
    
    See Also:
    Constant Field Values
  - SET_PERMISSION_COMMAND
```
public static final String SET_PERMISSION_COMMAND
```
    a Unix command to set permission: "chmod".
    
    See Also:
    Constant Field Values
  - SET_OWNER_COMMAND
```
public static final String SET_OWNER_COMMAND
```
    a Unix command to set owner: "chown".
    
    See Also:
    Constant Field Values
  - SET_GROUP_COMMAND
```
public static final String SET_GROUP_COMMAND
```
    a Unix command to set the change user's groups list: "chgrp".
    
    See Also:
    Constant Field Values
  - LINK_COMMAND
```
public static final String LINK_COMMAND
```
    a Unix command to create a link: "ln".
    
    See Also:
    Constant Field Values
  - READ_LINK_COMMAND
```
public static final String READ_LINK_COMMAND
```
    a Unix command to get a link target: "readlink".
    
    See Also:
    Constant Field Values
  - timeOutInterval
```
protected long timeOutInterval
```
    Time after which the executing script would be timedout.
  - inheritParentEnv
```
protected boolean inheritParentEnv
```
    Indicates if the parent env vars should be inherited or not
  - WINUTILS
```
@Deprecated
public static final String WINUTILS
```
    Deprecated. use one of the exception-raising getter methods, specifically getWinUtilsPath() or getWinUtilsFile()
    
    Location of winutils as a string; null if not found.
    Important: caller must check for this value being null. The lack of such checks has led to many support issues being raised.
  - isSetsidAvailable
```
public static final boolean isSetsidAvailable
```
    Flag which is true if setsid exists.
  - TOKEN_SEPARATOR_REGEX
```
public static final String TOKEN_SEPARATOR_REGEX
```
    Token separator regex used to parse Shell tool outputs.
- Constructor Detail
  - Shell
```
protected Shell()
```
    Create an instance with no minimum interval between runs; stderr is not merged with stdout.
  - Shell
```
protected Shell(long interval)
```
    Create an instance with a minimum interval between executions; stderr is not merged with stdout.
    
    Parameters:
    interval - interval in milliseconds between command executions.
  - Shell
```
protected Shell(long interval,
     boolean redirectErrorStream)
```
    Create a shell instance which can be re-executed when the run() method is invoked with a given elapsed time between calls.
    
    Parameters:
    interval - the minimum duration in milliseconds to wait before re-executing the command. If set to 0, there is no minimum.
    redirectErrorStream - should the error stream be merged with the normal output stream?
- Method Detail
  - isJava7OrAbove
```
@Deprecated
public static boolean isJava7OrAbove()
```
    Deprecated. This call isn't needed any more: please remove uses of it.
    
    query to see if system is Java 7 or later. Now that Hadoop requires Java 7 or later, this always returns true.
    
    Returns:
    true, always.
  - checkWindowsCommandLineLength
```
public static void checkWindowsCommandLineLength(String... commands)
                                          throws IOException
```
    Checks if a given command (String[]) fits in the Windows maximum command line length Note that the input is expected to already include space delimiters, no extra count will be added for delimiters.
    
    Parameters:
    commands - command parts, including any space delimiters
    
    Throws:
    
    IOException
  - getGroupsCommand
```
public static String[] getGroupsCommand()
```
    a Unix command to get the current user's groups list.
  - getGroupsForUserCommand
```
public static String[] getGroupsForUserCommand(String user)
```
    A command to get a given user's groups list. If the OS is not WINDOWS, the command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice.
  - getGroupsIDForUserCommand
```
public static String[] getGroupsIDForUserCommand(String user)
```
    A command to get a given user's group id list. The command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice. This command does not support Windows and will only return group names.
  - getUsersForNetgroupCommand
```
public static String[] getUsersForNetgroupCommand(String netgroup)
```
    A command to get a given netgroup's user list.
  - getGetPermissionCommand
```
public static String[] getGetPermissionCommand()
```
    Return a command to get permission information.
  - getSetPermissionCommand
```
public static String[] getSetPermissionCommand(String perm,
                               boolean recursive)
```
    Return a command to set permission.
  - getSetPermissionCommand
```
public static String[] getSetPermissionCommand(String perm,
                               boolean recursive,
                               String file)
```
    Return a command to set permission for specific file.
    
    Parameters:
    perm - String permission to set
    recursive - boolean true to apply to all sub-directories recursively
    file - String file to set
    
    Returns:
    String[] containing command and arguments
  - getSetOwnerCommand
```
public static String[] getSetOwnerCommand(String owner)
```
    Return a command to set owner.
  - getSymlinkCommand
```
public static String[] getSymlinkCommand(String target,
                         String link)
```
    Return a command to create symbolic links.
  - getReadlinkCommand
```
public static String[] getReadlinkCommand(String link)
```
    Return a command to read the target of the a symbolic link.
  - getCheckProcessIsAliveCommand
```
public static String[] getCheckProcessIsAliveCommand(String pid)
```
    Return a command for determining if process with specified pid is alive.
    
    Parameters:
    pid - process ID
    
    Returns:
    a kill -0 command or equivalent
  - getSignalKillCommand
```
public static String[] getSignalKillCommand(int code,
                            String pid)
```
    Return a command to send a signal to a given pid.
  - getEnvironmentVariableRegex
```
public static String getEnvironmentVariableRegex()
```
    Return a regular expression string that match environment variables.
  - appendScriptExtension
```
public static File appendScriptExtension(File parent,
                         String basename)
```
    Returns a File referencing a script with the given basename, inside the given parent directory. The file extension is inferred by platform: ".cmd" on Windows, or ".sh" otherwise.
    
    Parameters:
    parent - File parent directory
    basename - String script file basename
    
    Returns:
    File referencing the script in the directory
  - appendScriptExtension
```
public static String appendScriptExtension(String basename)
```
    Returns a script file name with the given basename. The file extension is inferred by platform: ".cmd" on Windows, or ".sh" otherwise.
    
    Parameters:
    basename - String script file basename
    
    Returns:
    String script file name
  - getRunScriptCommand
```
public static String[] getRunScriptCommand(File script)
```
    Returns a command to run the given script. The script interpreter is inferred by platform: cmd on Windows or bash otherwise.
    
    Parameters:
    script - File script to run
    
    Returns:
    String[] command to run the script
  - getHadoopHome
```
public static String getHadoopHome()
                            throws IOException
```
    Get the Hadoop home directory. Raises an exception if not found
    
    Returns:
    the home dir
    
    Throws:
    
    IOException - if the home directory cannot be located.
  - getQualifiedBin
```
public static File getQualifiedBin(String executable)
                            throws FileNotFoundException
```
    Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
    
    Parameters:
    executable - executable
    
    Returns:
    executable file reference
    
    Throws:
    
    FileNotFoundException - if the path does not exist
  - getQualifiedBinPath
```
public static String getQualifiedBinPath(String executable)
                                  throws IOException
```
    Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
    
    Parameters:
    executable - executable
    
    Returns:
    executable file reference
    
    Throws:
    
    FileNotFoundException - if the path does not exist
    
    IOException - on path canonicalization failures
  - hasWinutilsPath
```
public static boolean hasWinutilsPath()
```
    Predicate to indicate whether or not the path to winutils is known. If true, then WINUTILS is non-null, and both getWinUtilsPath() and getWinUtilsFile() will successfully return this value. Always false on non-windows systems.
    
    Returns:
    true if there is a valid path to the binary
  - getWinUtilsPath
```
public static String getWinUtilsPath()
```
    Locate the winutils binary, or fail with a meaningful exception and stack trace as an RTE. This method is for use in methods which don't explicitly throw an IOException.
    
    Returns:
    the path to WINUTILS_EXE
    
    Throws:
    
    RuntimeException - if the path is not resolvable
  - getWinUtilsFile
```
public static File getWinUtilsFile()
                            throws FileNotFoundException
```
    Get a file reference to winutils. Always raises an exception if there isn't one
    
    Returns:
    the file instance referring to the winutils bin.
    
    Throws:
    
    FileNotFoundException - on any failure to locate that file.
  - checkIsBashSupported
```
public static boolean checkIsBashSupported()
                                    throws InterruptedIOException
```
    Throws:
    
    InterruptedIOException
  - setEnvironment
```
protected void setEnvironment(Map<String,String> env)
```
    Set the environment for the command.
    
    Parameters:
    env - Mapping of environment variables
  - setWorkingDirectory
```
protected void setWorkingDirectory(File dir)
```
    Set the working directory.
    
    Parameters:
    dir - The directory where the command will be executed
  - run
```
protected void run()
            throws IOException
```
    Check to see if a command needs to be executed and execute if needed.
    
    Throws:
    
    IOException
  - getExecString
```
protected abstract String[] getExecString()
```
    return an array containing the command name and its parameters.
  - parseExecResult
```
protected abstract void parseExecResult(BufferedReader lines)
                                 throws IOException
```
    Parse the execution result
    
    Throws:
    
    IOException
  - getEnvironment
```
public String getEnvironment(String env)
```
    Get an environment variable.
    
    Parameters:
    env - the environment var
    
    Returns:
    the value or null if it was unset.
  - getProcess
```
public Process getProcess()
```
    get the current sub-process executing the given command.
    
    Returns:
    process executing the command
  - getExitCode
```
public int getExitCode()
```
    get the exit code.
    
    Returns:
    the exit code of the process
  - getWaitingThread
```
public Thread getWaitingThread()
```
    get the thread that is waiting on this instance of Shell.
    
    Returns:
    the thread that ran runCommand() that spawned this shell or null if no thread is waiting for this shell to complete
  - isTimedOut
```
public boolean isTimedOut()
```
    To check if the passed script to shell command executor timed out or not.
    
    Returns:
    if the script timed out.
  - execCommand
```
public static String execCommand(String... cmd)
                          throws IOException
```
    Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the Shell interface.
    
    Parameters:
    cmd - shell command to execute.
    
    Returns:
    the output of the executed command.
    
    Throws:
    
    IOException
  - execCommand
```
public static String execCommand(Map<String,String> env,
                 String[] cmd,
                 long timeout)
                          throws IOException
```
    Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the Shell interface.
    
    Parameters:
    env - the map of environment key=value
    cmd - shell command to execute.
    timeout - time in milliseconds after which script should be marked timeout
    
    Returns:
    the output of the executed command.
    
    Throws:
    
    IOException - on any problem.
  - execCommand
```
public static String execCommand(Map<String,String> env,
                 String... cmd)
                          throws IOException
```
    Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the Shell interface.
    
    Parameters:
    env - the map of environment key=value
    cmd - shell command to execute.
    
    Returns:
    the output of the executed command.
    
    Throws:
    
    IOException - on any problem.
  - destroyAllShellProcesses
```
public static void destroyAllShellProcesses()
```
    Static method to destroy all running Shell processes. Iterates through a map of all currently running Shell processes and destroys them one by one. This method is thread safe
  - getAllShells
```
public static Set<Shell> getAllShells()
```
    Static method to return a Set of all Shell objects.
  - getMemlockLimit
```
public static Long getMemlockLimit(Long ulimit)
```
    Static method to return the memory lock limit for datanode.
    
    Parameters:
    ulimit - max value at which memory locked should be capped.
    
    Returns:
    long value specifying the memory lock limit.

Class Shell

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

SYSPROP_HADOOP_HOME_DIR

ENV_HADOOP_HOME

WINDOWS_MAX_SHELL_LENGTH

WINDOWS_MAX_SHELL_LENGHT

USER_NAME_COMMAND

WindowsProcessLaunchLock

osType

WINDOWS

SOLARIS

MAC

FREEBSD

LINUX

OTHER

PPC_64

ENV_NAME_REGEX

SET_PERMISSION_COMMAND

SET_OWNER_COMMAND

SET_GROUP_COMMAND

LINK_COMMAND

READ_LINK_COMMAND

timeOutInterval

inheritParentEnv

WINUTILS

isSetsidAvailable

TOKEN_SEPARATOR_REGEX

Constructor Detail

Shell

Shell

Shell

Method Detail

isJava7OrAbove

checkWindowsCommandLineLength

getGroupsCommand

getGroupsForUserCommand

getGroupsIDForUserCommand

getUsersForNetgroupCommand

getGetPermissionCommand

getSetPermissionCommand

getSetPermissionCommand

getSetOwnerCommand

getSymlinkCommand

getReadlinkCommand

getCheckProcessIsAliveCommand

getSignalKillCommand

getEnvironmentVariableRegex

appendScriptExtension

appendScriptExtension

getRunScriptCommand

getHadoopHome

getQualifiedBin

getQualifiedBinPath

hasWinutilsPath

getWinUtilsPath

getWinUtilsFile

checkIsBashSupported

setEnvironment

setWorkingDirectory

run

getExecString

parseExecResult

getEnvironment

getProcess

getExitCode

getWaitingThread

isTimedOut

execCommand

execCommand

execCommand

destroyAllShellProcesses

getAllShells

getMemlockLimit