Package org.apache.hadoop.util

Class Shell

java.lang.Object
org.apache.hadoop.util.Shell
@Public @Evolving public abstract class Shell extends Object
A base class for running a Shell command. Shell can be used to run shell commands like du or df. It also offers facilities to gate commands by time-intervals.

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

    • SYSPROP_HADOOP_HOME_DIR

      public static final String SYSPROP_HADOOP_HOME_DIR
      System property for the Hadoop home directory: "hadoop.home.dir".
      See Also:

    • ENV_HADOOP_HOME

      public static final String ENV_HADOOP_HOME
      Environment variable for Hadoop's home dir: "HADOOP_HOME".
      See Also:

    • WINDOWS_MAX_SHELL_LENGTH

      public static final int WINDOWS_MAX_SHELL_LENGTH
      Maximum command line length in Windows KB830473 documents this as 8191
      See Also:

    • WINDOWS_MAX_SHELL_LENGHT

      @Deprecated public static final int WINDOWS_MAX_SHELL_LENGHT
      Deprecated.
      use the correctly spelled constant.
      mis-spelling of WINDOWS_MAX_SHELL_LENGTH.
      See Also:

    • USER_NAME_COMMAND

      public static final String USER_NAME_COMMAND
      a Unix command to get the current user's name: "whoami".
      See Also:

    • WindowsProcessLaunchLock

      public static final Object WindowsProcessLaunchLock
      Windows CreateProcess synchronization object.

    • osType

      public static final Shell.OSType osType
      Get the type of the operating system, as determined from parsing the os.name property.

    • WINDOWS

      public static final boolean WINDOWS

    • SOLARIS

      public static final boolean SOLARIS

    • MAC

      public static final boolean MAC

    • FREEBSD

      public static final boolean FREEBSD

    • LINUX

      public static final boolean LINUX

    • OTHER

      public static final boolean OTHER

    • PPC_64

      public static final boolean PPC_64

    • ENV_NAME_REGEX

      public static final String ENV_NAME_REGEX
      Regular expression for environment variables: "[A-Za-z_][A-Za-z0-9_]*".
      See Also:

    • SET_PERMISSION_COMMAND

      public static final String SET_PERMISSION_COMMAND
      a Unix command to set permission: "chmod".
      See Also:

    • SET_OWNER_COMMAND

      public static final String SET_OWNER_COMMAND
      a Unix command to set owner: "chown".
      See Also:

    • SET_GROUP_COMMAND

      public static final String SET_GROUP_COMMAND
      a Unix command to set the change user's groups list: "chgrp".
      See Also:

    • timeOutInterval

      protected long timeOutInterval
      Time after which the executing script would be timedout.

    • inheritParentEnv

      protected boolean inheritParentEnv
      Indicates if the parent env vars should be inherited or not

    • WINUTILS

      @Deprecated public static final String WINUTILS
      Deprecated.
      use one of the exception-raising getter methods, specifically getWinUtilsPath() or getWinUtilsFile()
      Location of winutils as a string; null if not found.

      Important: caller must check for this value being null. The lack of such checks has led to many support issues being raised.

    • isSetsidAvailable

      public static final boolean isSetsidAvailable
      Flag which is true if setsid exists.

    • TOKEN_SEPARATOR_REGEX

      public static final String TOKEN_SEPARATOR_REGEX
      Token separator regex used to parse Shell tool outputs.

  • Constructor Details

    • Shell

      protected Shell()
      Create an instance with no minimum interval between runs; stderr is not merged with stdout.

    • Shell

      protected Shell(long interval)
      Create an instance with a minimum interval between executions; stderr is not merged with stdout.
      Parameters:
      interval - interval in milliseconds between command executions.

    • Shell

      protected Shell(long interval, boolean redirectErrorStream)
      Create a shell instance which can be re-executed when the run() method is invoked with a given elapsed time between calls.
      Parameters:
      interval - the minimum duration in milliseconds to wait before re-executing the command. If set to 0, there is no minimum.
      redirectErrorStream - should the error stream be merged with the normal output stream?

  • Method Details

    • isJava7OrAbove

      @Deprecated public static boolean isJava7OrAbove()
      Deprecated.
      This call isn't needed any more: please remove uses of it.
      query to see if system is Java 7 or later. Now that Hadoop requires Java 7 or later, this always returns true.
      Returns:
      true, always.

    • isJavaVersionAtLeast

      public static boolean isJavaVersionAtLeast(int version)
      Query to see if major version of Java specification of the system is equal or greater than the parameter.
      Parameters:
      version - 8, 9, 10 etc.
      Returns:
      comparison with system property, always true for 8

    • checkWindowsCommandLineLength

      public static void checkWindowsCommandLineLength(String... commands) throws IOException
      Checks if a given command (String[]) fits in the Windows maximum command line length Note that the input is expected to already include space delimiters, no extra count will be added for delimiters.
      Parameters:
      commands - command parts, including any space delimiters
      Throws:
      IOException - raised on errors performing I/O.

    • bashQuote

      @Private public static String bashQuote(String arg)
      Quote the given arg so that bash will interpret it as a single value. Note that this quotes it for one level of bash, if you are passing it into a badly written shell script, you need to fix your shell script.
      Parameters:
      arg - the argument to quote
      Returns:
      the quoted string

    • getGroupsCommand

      public static String[] getGroupsCommand()
      a Unix command to get the current user's groups list.
      Returns:
      group command array.

    • getGroupsForUserCommand

      public static String[] getGroupsForUserCommand(String user)
      A command to get a given user's groups list. If the OS is not WINDOWS, the command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice.
      Parameters:
      user - user.
      Returns:
      groups for user command.

    • getGroupsIDForUserCommand

      public static String[] getGroupsIDForUserCommand(String user)
      A command to get a given user's group id list. The command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice. This command does not support Windows and will only return group names.
      Parameters:
      user - user.
      Returns:
      groups id for user command.

    • getUsersForNetgroupCommand

      public static String[] getUsersForNetgroupCommand(String netgroup)
      A command to get a given netgroup's user list.
      Parameters:
      netgroup - net group.
      Returns:
      users for net group command.

    • getGetPermissionCommand

      public static String[] getGetPermissionCommand()
      Return a command to get permission information.
      Returns:
      permission command.

    • getSetPermissionCommand

      public static String[] getSetPermissionCommand(String perm, boolean recursive)
      Return a command to set permission.
      Parameters:
      perm - permission.
      recursive - recursive.
      Returns:
      set permission command.

    • getSetPermissionCommand

      public static String[] getSetPermissionCommand(String perm, boolean recursive, String file)
      Return a command to set permission for specific file.
      Parameters:
      perm - String permission to set
      recursive - boolean true to apply to all sub-directories recursively
      file - String file to set
      Returns:
      String[] containing command and arguments

    • getSetOwnerCommand

      public static String[] getSetOwnerCommand(String owner)
      Return a command to set owner.
      Parameters:
      owner - owner.
      Returns:
      set owner command.

    • getSymlinkCommand

      public static String[] getSymlinkCommand(String target, String link)
      Return a command to create symbolic links.
      Parameters:
      target - target.
      link - link.
      Returns:
      symlink command.

    • getReadlinkCommand

      public static String[] getReadlinkCommand(String link)
      Return a command to read the target of the a symbolic link.
      Parameters:
      link - link.
      Returns:
      read link command.

    • getCheckProcessIsAliveCommand

      public static String[] getCheckProcessIsAliveCommand(String pid)
      Return a command for determining if process with specified pid is alive.
      Parameters:
      pid - process ID
      Returns:
      a kill -0 command or equivalent

    • getSignalKillCommand

      public static String[] getSignalKillCommand(int code, String pid)
      Return a command to send a signal to a given pid.
      Parameters:
      code - code.
      pid - pid.
      Returns:
      signal kill command.

    • getEnvironmentVariableRegex

      public static String getEnvironmentVariableRegex()
      Return a regular expression string that match environment variables.
      Returns:
      environment variable regex.

    • appendScriptExtension

      public static File appendScriptExtension(File parent, String basename)
      Returns a File referencing a script with the given basename, inside the given parent directory. The file extension is inferred by platform: ".cmd" on Windows, or ".sh" otherwise.
      Parameters:
      parent - File parent directory
      basename - String script file basename
      Returns:
      File referencing the script in the directory

    • appendScriptExtension

      public static String appendScriptExtension(String basename)
      Returns a script file name with the given basename. The file extension is inferred by platform: ".cmd" on Windows, or ".sh" otherwise.
      Parameters:
      basename - String script file basename
      Returns:
      String script file name

    • getRunScriptCommand

      public static String[] getRunScriptCommand(File script)
      Returns a command to run the given script. The script interpreter is inferred by platform: cmd on Windows or bash otherwise.
      Parameters:
      script - File script to run
      Returns:
      String[] command to run the script

    • getHadoopHome

      public static String getHadoopHome() throws IOException
      Get the Hadoop home directory. Raises an exception if not found
      Returns:
      the home dir
      Throws:
      IOException - if the home directory cannot be located.

    • getQualifiedBin

      public static File getQualifiedBin(String executable) throws FileNotFoundException
      Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
      Parameters:
      executable - executable
      Returns:
      executable file reference
      Throws:
      FileNotFoundException - if the path does not exist

    • getQualifiedBinPath

      public static String getQualifiedBinPath(String executable) throws IOException
      Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
      Parameters:
      executable - executable
      Returns:
      executable file reference
      Throws:
      FileNotFoundException - if the path does not exist
      IOException - on path canonicalization failures

    • hasWinutilsPath

      public static boolean hasWinutilsPath()
      Predicate to indicate whether or not the path to winutils is known. If true, then WINUTILS is non-null, and both getWinUtilsPath() and getWinUtilsFile() will successfully return this value. Always false on non-windows systems.
      Returns:
      true if there is a valid path to the binary

    • getWinUtilsPath

      public static String getWinUtilsPath()
      Locate the winutils binary, or fail with a meaningful exception and stack trace as an RTE. This method is for use in methods which don't explicitly throw an IOException.
      Returns:
      the path to WINUTILS_EXE
      Throws:
      RuntimeException - if the path is not resolvable

    • getWinUtilsFile

      public static File getWinUtilsFile() throws FileNotFoundException
      Get a file reference to winutils. Always raises an exception if there isn't one
      Returns:
      the file instance referring to the winutils bin.
      Throws:
      FileNotFoundException - on any failure to locate that file.

    • checkIsBashSupported

      public static boolean checkIsBashSupported() throws InterruptedIOException
      Throws:
      InterruptedIOException

    • setEnvironment

      protected void setEnvironment(Map<String,String> env)
      Set the environment for the command.
      Parameters:
      env - Mapping of environment variables

    • setWorkingDirectory

      protected void setWorkingDirectory(File dir)
      Set the working directory.
      Parameters:
      dir - The directory where the command will be executed

    • run

      protected void run() throws IOException
      Check to see if a command needs to be executed and execute if needed.
      Throws:
      IOException - raised on errors performing I/O.

    • getExecString

      protected abstract String[] getExecString()
      return an array containing the command name and its parameters.
      Returns:
      exec string array.

    • parseExecResult

      protected abstract void parseExecResult(BufferedReader lines) throws IOException
      Parse the execution result.
      Parameters:
      lines - lines.
      Throws:
      IOException - raised on errors performing I/O.

    • getEnvironment

      public String getEnvironment(String env)
      Get an environment variable.
      Parameters:
      env - the environment var
      Returns:
      the value or null if it was unset.

    • getProcess

      public Process getProcess()
      get the current sub-process executing the given command.
      Returns:
      process executing the command

    • getExitCode

      public int getExitCode()
      get the exit code.
      Returns:
      the exit code of the process

    • getWaitingThread

      public Thread getWaitingThread()
      get the thread that is waiting on this instance of Shell.
      Returns:
      the thread that ran runCommand() that spawned this shell or null if no thread is waiting for this shell to complete

    • isTimedOut

      public boolean isTimedOut()
      To check if the passed script to shell command executor timed out or not.
      Returns:
      if the script timed out.

    • execCommand

      public static String execCommand(String... cmd) throws IOException
      Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the Shell interface.
      Parameters:
      cmd - shell command to execute.
      Returns:
      the output of the executed command.
      Throws:
      IOException - raised on errors performing I/O.

    • execCommand

      public static String execCommand(Map<String,String> env, String[] cmd, long timeout) throws IOException
      Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the Shell interface.
      Parameters:
      env - the map of environment key=value
      cmd - shell command to execute.
      timeout - time in milliseconds after which script should be marked timeout
      Returns:
      the output of the executed command.
      Throws:
      IOException - on any problem.

    • execCommand

      public static String execCommand(Map<String,String> env, String... cmd) throws IOException
      Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the Shell interface.
      Parameters:
      env - the map of environment key=value
      cmd - shell command to execute.
      Returns:
      the output of the executed command.
      Throws:
      IOException - on any problem.

    • destroyAllShellProcesses

      public static void destroyAllShellProcesses()
      Static method to destroy all running Shell processes. Iterates through a map of all currently running Shell processes and destroys them one by one. This method is thread safe

    • getAllShells

      public static Set<Shell> getAllShells()
      Static method to return a Set of all Shell objects.
      Returns:
      all shells set.

    • getMemlockLimit

      public static Long getMemlockLimit(Long ulimit)
      Static method to return the memory lock limit for datanode.
      Parameters:
      ulimit - max value at which memory locked should be capped.
      Returns:
      long value specifying the memory lock limit.