Package org.apache.hadoop.fs

Class FileUtil

java.lang.Object
org.apache.hadoop.fs.FileUtil
@Public @Evolving public class FileUtil extends Object
A collection of file-processing util methods

  • Field Details

  • Constructor Details

    • FileUtil

      public FileUtil()

  • Method Details

    • stat2Paths

      public static Path[] stat2Paths(FileStatus[] stats)
      convert an array of FileStatus to an array of Path
      Parameters:
      stats - an array of FileStatus objects
      Returns:
      an array of paths corresponding to the input

    • stat2Paths

      public static Path[] stat2Paths(FileStatus[] stats, Path path)
      convert an array of FileStatus to an array of Path. If stats if null, return path
      Parameters:
      stats - an array of FileStatus objects
      path - default path to return in stats is null
      Returns:
      an array of paths corresponding to the input

    • fullyDeleteOnExit

      public static void fullyDeleteOnExit(File file)
      Register all files recursively to be deleted on exit.
      Parameters:
      file - File/directory to be deleted

    • fullyDelete

      public static boolean fullyDelete(File dir)
      Delete a directory and all its contents. If we return false, the directory may be partially-deleted. (1) If dir is symlink to a file, the symlink is deleted. The file pointed to by the symlink is not deleted. (2) If dir is symlink to a directory, symlink is deleted. The directory pointed to by symlink is not deleted. (3) If dir is a normal file, it is deleted. (4) If dir is a normal directory, then dir and all its contents recursively are deleted.
      Parameters:
      dir - dir.
      Returns:
      fully delete status.

    • fullyDelete

      public static boolean fullyDelete(File dir, boolean tryGrantPermissions)
      Delete a directory and all its contents. If we return false, the directory may be partially-deleted. (1) If dir is symlink to a file, the symlink is deleted. The file pointed to by the symlink is not deleted. (2) If dir is symlink to a directory, symlink is deleted. The directory pointed to by symlink is not deleted. (3) If dir is a normal file, it is deleted. (4) If dir is a normal directory, then dir and all its contents recursively are deleted.
      Parameters:
      dir - the file or directory to be deleted
      tryGrantPermissions - true if permissions should be modified to delete a file.
      Returns:
      true on success false on failure.

    • readLink

      public static String readLink(File f)
      Returns the target of the given symlink. Returns the empty string if the given path does not refer to a symlink or there is an error accessing the symlink.
      Parameters:
      f - File representing the symbolic link.
      Returns:
      The target of the symbolic link, empty string on error or if not a symlink.

    • fullyDeleteContents

      public static boolean fullyDeleteContents(File dir)
      Delete the contents of a directory, not the directory itself. If we return false, the directory may be partially-deleted. If dir is a symlink to a directory, all the contents of the actual directory pointed to by dir will be deleted.
      Parameters:
      dir - dir.
      Returns:
      fullyDeleteContents Status.

    • fullyDeleteContents

      public static boolean fullyDeleteContents(File dir, boolean tryGrantPermissions)
      Delete the contents of a directory, not the directory itself. If we return false, the directory may be partially-deleted. If dir is a symlink to a directory, all the contents of the actual directory pointed to by dir will be deleted.
      Parameters:
      dir - dir.
      tryGrantPermissions - if 'true', try grant +rwx permissions to this and all the underlying directories before trying to delete their contents.
      Returns:
      fully delete contents status.

    • fullyDelete

      @Deprecated public static void fullyDelete(FileSystem fs, Path dir) throws IOException
      Deprecated.
      Use FileSystem.delete(Path, boolean)
      Recursively delete a directory.
      Parameters:
      fs - FileSystem on which the path is present
      dir - directory to recursively delete
      Throws:
      IOException - raised on errors performing I/O.

    • copy

      public static boolean copy(FileSystem srcFS, Path src, FileSystem dstFS, Path dst, boolean deleteSource, Configuration conf) throws IOException
      Copy files between FileSystems.
      Parameters:
      srcFS - src fs.
      src - src.
      dstFS - dst fs.
      dst - dst.
      deleteSource - delete source.
      conf - configuration.
      Returns:
      if copy success true, not false.
      Throws:
      IOException - raised on errors performing I/O.

    • copy

      public static boolean copy(FileSystem srcFS, Path[] srcs, FileSystem dstFS, Path dst, boolean deleteSource, boolean overwrite, Configuration conf) throws IOException
      Throws:
      IOException

    • copy

      public static boolean copy(FileSystem srcFS, Path src, FileSystem dstFS, Path dst, boolean deleteSource, boolean overwrite, Configuration conf) throws IOException
      Copy files between FileSystems.
      Parameters:
      srcFS - srcFs.
      src - src.
      dstFS - dstFs.
      dst - dst.
      deleteSource - delete source.
      overwrite - overwrite.
      conf - configuration.
      Returns:
      true if the operation succeeded.
      Throws:
      IOException - raised on errors performing I/O.

    • copy

      public static boolean copy(FileSystem srcFS, FileStatus srcStatus, FileSystem dstFS, Path dst, boolean deleteSource, boolean overwrite, Configuration conf) throws IOException
      Copy a file/directory tree within/between filesystems.

      returns true if the operation succeeded. When deleteSource is true, this means "after the copy, delete(source) returned true" If the destination is a directory, and mkdirs (dest) fails, the operation will return false rather than raise any exception.

      The overwrite flag is about overwriting files; it has no effect about handing an attempt to copy a file atop a directory (expect an IOException), or a directory over a path which contains a file (mkdir will fail, so "false").

      The operation is recursive, and the deleteSource operation takes place as each subdirectory is copied. Therefore, if an operation fails partway through, the source tree may be partially deleted.

      Parameters:
      srcFS - source filesystem
      srcStatus - status of source
      dstFS - destination filesystem
      dst - path of source
      deleteSource - delete the source?
      overwrite - overwrite files at destination?
      conf - configuration to use when opening files
      Returns:
      true if the operation succeeded.
      Throws:
      IOException - failure

    • copy

      public static boolean copy(File src, FileSystem dstFS, Path dst, boolean deleteSource, Configuration conf) throws IOException
      Copy local files to a FileSystem.
      Parameters:
      src - src.
      dstFS - dstFs.
      dst - dst.
      deleteSource - delete source.
      conf - configuration.
      Returns:
      true if the operation succeeded.
      Throws:
      IOException - raised on errors performing I/O.

    • copy

      public static boolean copy(FileSystem srcFS, Path src, File dst, boolean deleteSource, Configuration conf) throws IOException
      Copy FileSystem files to local files.
      Parameters:
      srcFS - srcFs.
      src - src.
      dst - dst.
      deleteSource - delete source.
      conf - configuration.
      Returns:
      true if the operation succeeded.
      Throws:
      IOException - raised on errors performing I/O.

    • isRegularFile

      public static boolean isRegularFile(File file)

    • isRegularFile

      public static boolean isRegularFile(File file, boolean allowLinks)
      Check if the file is regular.
      Parameters:
      file - The file being checked.
      allowLinks - Whether to allow matching links.
      Returns:
      Returns the result of checking whether the file is a regular file.

    • makeShellPath

      public static String makeShellPath(String filename) throws IOException
      Convert a os-native filename to a path that works for the shell.
      Parameters:
      filename - The filename to convert
      Returns:
      The unix pathname
      Throws:
      IOException - on windows, there can be problems with the subprocess

    • makeShellPath

      public static String makeShellPath(File file) throws IOException
      Convert a os-native filename to a path that works for the shell.
      Parameters:
      file - The filename to convert
      Returns:
      The unix pathname
      Throws:
      IOException - on windows, there can be problems with the subprocess

    • makeSecureShellPath

      public static String makeSecureShellPath(File file) throws IOException
      Convert a os-native filename to a path that works for the shell and avoids script injection attacks.
      Parameters:
      file - The filename to convert
      Returns:
      The unix pathname
      Throws:
      IOException - on windows, there can be problems with the subprocess

    • makeShellPath

      public static String makeShellPath(File file, boolean makeCanonicalPath) throws IOException
      Convert a os-native filename to a path that works for the shell.
      Parameters:
      file - The filename to convert
      makeCanonicalPath - Whether to make canonical path for the file passed
      Returns:
      The unix pathname
      Throws:
      IOException - on windows, there can be problems with the subprocess

    • getDU

      public static long getDU(File dir)
      Takes an input dir and returns the du on that local directory. Very basic implementation.
      Parameters:
      dir - The input dir to get the disk space of this local dir
      Returns:
      The total disk space of the input local directory

    • unZip

      public static void unZip(InputStream inputStream, File toDir) throws IOException
      Given a stream input it will unzip the it in the unzip directory. passed as the second parameter
      Parameters:
      inputStream - The zip file as input
      toDir - The unzip directory where to unzip the zip file.
      Throws:
      IOException - an exception occurred

    • unZip

      public static void unZip(File inFile, File unzipDir) throws IOException
      Given a File input it will unzip it in the unzip directory. passed as the second parameter
      Parameters:
      inFile - The zip file as input
      unzipDir - The unzip directory where to unzip the zip file.
      Throws:
      IOException - An I/O exception has occurred

    • unTar

      public static void unTar(InputStream inputStream, File untarDir, boolean gzipped) throws IOException, InterruptedException, ExecutionException
      Given a Tar File as input it will untar the file in a the untar directory passed as the second parameter This utility will untar ".tar" files and ".tar.gz","tgz" files.
      Parameters:
      inputStream - The tar file as input.
      untarDir - The untar directory where to untar the tar file.
      gzipped - The input stream is gzipped TODO Use magic number and PusbackInputStream to identify
      Throws:
      IOException - an exception occurred
      InterruptedException - command interrupted
      ExecutionException - task submit failed

    • unTar

      public static void unTar(File inFile, File untarDir) throws IOException
      Given a Tar File as input it will untar the file in a the untar directory passed as the second parameter This utility will untar ".tar" files and ".tar.gz","tgz" files.
      Parameters:
      inFile - The tar file as input.
      untarDir - The untar directory where to untar the tar file.
      Throws:
      IOException - an exception occurred.

    • symLink

      public static int symLink(String target, String linkname) throws IOException
      Create a soft link between a src and destination only on a local disk. HDFS does not support this. On Windows, when symlink creation fails due to security setting, we will log a warning. The return code in this case is 2.
      Parameters:
      target - the target for symlink
      linkname - the symlink
      Returns:
      0 on success
      Throws:
      IOException - raised on errors performing I/O.

    • chmod

      public static int chmod(String filename, String perm) throws IOException, InterruptedException
      Change the permissions on a filename.
      Parameters:
      filename - the name of the file to change
      perm - the permission string
      Returns:
      the exit code from the command
      Throws:
      IOException - raised on errors performing I/O.
      InterruptedException - command interrupted.

    • chmod

      public static int chmod(String filename, String perm, boolean recursive) throws IOException
      Change the permissions on a file / directory, recursively, if needed.
      Parameters:
      filename - name of the file whose permissions are to change
      perm - permission string
      recursive - true, if permissions should be changed recursively
      Returns:
      the exit code from the command.
      Throws:
      IOException - raised on errors performing I/O.

    • setOwner

      public static void setOwner(File file, String username, String groupname) throws IOException
      Set the ownership on a file / directory. User name and group name cannot both be null.
      Parameters:
      file - the file to change
      username - the new user owner name
      groupname - the new group owner name
      Throws:
      IOException - raised on errors performing I/O.

    • setReadable

      public static boolean setReadable(File f, boolean readable)
      Platform independent implementation for File.setReadable(boolean) File#setReadable does not work as expected on Windows.
      Parameters:
      f - input file
      readable - readable.
      Returns:
      true on success, false otherwise

    • setWritable

      public static boolean setWritable(File f, boolean writable)
      Platform independent implementation for File.setWritable(boolean) File#setWritable does not work as expected on Windows.
      Parameters:
      f - input file
      writable - writable.
      Returns:
      true on success, false otherwise

    • setExecutable

      public static boolean setExecutable(File f, boolean executable)
      Platform independent implementation for File.setExecutable(boolean) File#setExecutable does not work as expected on Windows. Note: revoking execute permission on folders does not have the same behavior on Windows as on Unix platforms. Creating, deleting or renaming a file within that folder will still succeed on Windows.
      Parameters:
      f - input file
      executable - executable.
      Returns:
      true on success, false otherwise

    • canRead

      public static boolean canRead(File f)
      Platform independent implementation for File.canRead()
      Parameters:
      f - input file
      Returns:
      On Unix, same as File.canRead() On Windows, true if process has read access on the path

    • canWrite

      public static boolean canWrite(File f)
      Platform independent implementation for File.canWrite()
      Parameters:
      f - input file
      Returns:
      On Unix, same as File.canWrite() On Windows, true if process has write access on the path

    • canExecute

      public static boolean canExecute(File f)
      Platform independent implementation for File.canExecute()
      Parameters:
      f - input file
      Returns:
      On Unix, same as File.canExecute() On Windows, true if process has execute access on the path

    • setPermission

      public static void setPermission(File f, FsPermission permission) throws IOException
      Set permissions to the required value. Uses the java primitives instead of forking if group == other.
      Parameters:
      f - the file to change
      permission - the new permissions
      Throws:
      IOException - raised on errors performing I/O.

    • createLocalTempFile

      public static final File createLocalTempFile(File basefile, String prefix, boolean isDeleteOnExit) throws IOException
      Create a tmp file for a base file.
      Parameters:
      basefile - the base file of the tmp
      prefix - file name prefix of tmp
      isDeleteOnExit - if true, the tmp will be deleted when the VM exits
      Returns:
      a newly created tmp file
      Throws:
      IOException - If a tmp file cannot created
      See Also:

    • replaceFile

      public static void replaceFile(File src, File target) throws IOException
      Move the src file to the name specified by target.
      Parameters:
      src - the source file
      target - the target file
      Throws:
      IOException - If this operation fails

    • listFiles

      public static File[] listFiles(File dir) throws IOException
      A wrapper for File.listFiles(). This java.io API returns null when a dir is not a directory or for any I/O error. Instead of having null check everywhere File#listFiles() is used, we will add utility API to get around this problem. For the majority of cases where we prefer an IOException to be thrown.
      Parameters:
      dir - directory for which listing should be performed
      Returns:
      list of files or empty list
      Throws:
      IOException - for invalid directory or for a bad disk.

    • list

      public static String[] list(File dir) throws IOException
      A wrapper for File.list(). This java.io API returns null when a dir is not a directory or for any I/O error. Instead of having null check everywhere File#list() is used, we will add utility API to get around this problem. For the majority of cases where we prefer an IOException to be thrown.
      Parameters:
      dir - directory for which listing should be performed
      Returns:
      list of file names or empty string list
      Throws:
      AccessDeniedException - for unreadable directory
      IOException - for invalid directory or for bad disk

    • createJarWithClassPath

      public static String[] createJarWithClassPath(String inputClassPath, Path pwd, Map<String,String> callerEnv) throws IOException
      Throws:
      IOException

    • createJarWithClassPath

      public static String[] createJarWithClassPath(String inputClassPath, Path pwd, Path targetDir, Map<String,String> callerEnv) throws IOException
      Create a jar file at the given path, containing a manifest with a classpath that references all specified entries. Some platforms may have an upper limit on command line length. For example, the maximum command line length on Windows is 8191 characters, but the length of the classpath may exceed this. To work around this limitation, use this method to create a small intermediate jar with a manifest that contains the full classpath. It returns the absolute path to the new jar, which the caller may set as the classpath for a new process. Environment variable evaluation is not supported within a jar manifest, so this method expands environment variables before inserting classpath entries to the manifest. The method parses environment variables according to platform-specific syntax (%VAR% on Windows, or $VAR otherwise). On Windows, environment variables are case-insensitive. For example, %VAR% and %var% evaluate to the same value. Specifying the classpath in a jar manifest does not support wildcards, so this method expands wildcards internally. Any classpath entry that ends with * is translated to all files at that path with extension .jar or .JAR.
      Parameters:
      inputClassPath - String input classpath to bundle into the jar manifest
      pwd - Path to working directory to save jar
      targetDir - path to where the jar execution will have its working dir
      callerEnv - Map <String, String> caller's environment variables to use for expansion
      Returns:
      String[] with absolute path to new jar in position 0 and unexpanded wild card entry path in position 1
      Throws:
      IOException - if there is an I/O error while writing the jar file

    • getJarsInDirectory

      public static List<Path> getJarsInDirectory(String path)
      Returns all jars that are in the directory. It is useful in expanding a wildcard path to return all jars from the directory to use in a classpath. It operates only on local paths.
      Parameters:
      path - the path to the directory. The path may include the wildcard.
      Returns:
      the list of jars as URLs, or an empty list if there are no jars, or the directory does not exist locally

    • getJarsInDirectory

      public static List<Path> getJarsInDirectory(String path, boolean useLocal)
      Returns all jars that are in the directory. It is useful in expanding a wildcard path to return all jars from the directory to use in a classpath.
      Parameters:
      path - the path to the directory. The path may include the wildcard.
      useLocal - use local.
      Returns:
      the list of jars as URLs, or an empty list if there are no jars, or the directory does not exist

    • compareFs

      public static boolean compareFs(FileSystem srcFs, FileSystem destFs)

    • write

      public static FileSystem write(FileSystem fs, Path path, byte[] bytes) throws IOException
      Writes bytes to a file. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file. All bytes in the byte array are written to the file.
      Parameters:
      fs - the file system with which to create the file
      path - the path to the file
      bytes - the byte array with the bytes to write
      Returns:
      the file system
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • write

      public static FileContext write(FileContext fileContext, Path path, byte[] bytes) throws IOException
      Writes bytes to a file. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file. All bytes in the byte array are written to the file.
      Parameters:
      fileContext - the file context with which to create the file
      path - the path to the file
      bytes - the byte array with the bytes to write
      Returns:
      the file context
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • write

      public static FileSystem write(FileSystem fs, Path path, Iterable<? extends CharSequence> lines, Charset cs) throws IOException
      Write lines of text to a file. Each line is a char sequence and is written to the file in sequence with each line terminated by the platform's line separator, as defined by the system property line.separator. Characters are encoded into bytes using the specified charset. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file.
      Parameters:
      fs - the file system with which to create the file
      path - the path to the file
      lines - a Collection to iterate over the char sequences
      cs - the charset to use for encoding
      Returns:
      the file system
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • write

      public static FileContext write(FileContext fileContext, Path path, Iterable<? extends CharSequence> lines, Charset cs) throws IOException
      Write lines of text to a file. Each line is a char sequence and is written to the file in sequence with each line terminated by the platform's line separator, as defined by the system property line.separator. Characters are encoded into bytes using the specified charset. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file.
      Parameters:
      fileContext - the file context with which to create the file
      path - the path to the file
      lines - a Collection to iterate over the char sequences
      cs - the charset to use for encoding
      Returns:
      the file context
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • write

      public static FileSystem write(FileSystem fs, Path path, CharSequence charseq, Charset cs) throws IOException
      Write a line of text to a file. Characters are encoded into bytes using the specified charset. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file.
      Parameters:
      fs - the file system with which to create the file
      path - the path to the file
      charseq - the char sequence to write to the file
      cs - the charset to use for encoding
      Returns:
      the file system
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • write

      public static FileContext write(FileContext fs, Path path, CharSequence charseq, Charset cs) throws IOException
      Write a line of text to a file. Characters are encoded into bytes using the specified charset. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file.
      Parameters:
      fs - the file context with which to create the file
      path - the path to the file
      charseq - the char sequence to write to the file
      cs - the charset to use for encoding
      Returns:
      the file context
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • write

      public static FileSystem write(FileSystem fs, Path path, CharSequence charseq) throws IOException
      Write a line of text to a file. Characters are encoded into bytes using UTF-8. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file.
      Parameters:
      fs - the files system with which to create the file
      path - the path to the file
      charseq - the char sequence to write to the file
      Returns:
      the file system
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • write

      public static FileContext write(FileContext fileContext, Path path, CharSequence charseq) throws IOException
      Write a line of text to a file. Characters are encoded into bytes using UTF-8. This utility method opens the file for writing, creating the file if it does not exist, or overwrites an existing file.
      Parameters:
      fileContext - the files system with which to create the file
      path - the path to the file
      charseq - the char sequence to write to the file
      Returns:
      the file context
      Throws:
      NullPointerException - if any of the arguments are null
      IOException - if an I/O error occurs creating or writing to the file

    • rename

      @LimitedPrivate("ViewDistributedFileSystem") @Unstable public static void rename(FileSystem srcFs, Path src, Path dst, Options.Rename... options) throws IOException
      Throws:
      IOException

    • maybeIgnoreMissingDirectory

      public static void maybeIgnoreMissingDirectory(FileSystem fs, Path path, FileNotFoundException e) throws FileNotFoundException
      Method to call after a FNFE has been raised on a treewalk, so as to decide whether to throw the exception (default), or, if the FS supports inconsistent directory listings, to log and ignore it. If this returns then the caller should ignore the failure and continue.
      Parameters:
      fs - filesystem
      path - path
      e - exception caught
      Throws:
      FileNotFoundException - the exception passed in, if rethrown.

    • checkFSSupportsEC

      public static boolean checkFSSupportsEC(FileSystem fs, Path path) throws IOException
      Return true if the FS implements WithErasureCoding and supports EC_POLICY option in Options.OpenFileOptions. A message is logged when the filesystem does not support Erasure coding.
      Parameters:
      fs - filesystem
      path - path
      Returns:
      true if the Filesystem supports EC
      Throws:
      IOException - if there is a failure in hasPathCapability call