org.apache.hadoop.fs
Class FileUtil

java.lang.Object
  extended by org.apache.hadoop.fs.FileUtil

@InterfaceAudience.Public
@InterfaceStability.Evolving
public class FileUtil
extends Object

A collection of file-processing util methods


Field Summary
static int SYMLINK_NO_PRIVILEGE
           
 
Constructor Summary
FileUtil()
           
 
Method Summary
static boolean canExecute(File f)
          Platform independent implementation for File.canExecute()
static boolean canRead(File f)
          Platform independent implementation for File.canRead()
static boolean canWrite(File f)
          Platform independent implementation for File.canWrite()
static int chmod(String filename, String perm)
          Change the permissions on a filename.
static int chmod(String filename, String perm, boolean recursive)
          Change the permissions on a file / directory, recursively, if needed.
static boolean copy(File src, FileSystem dstFS, Path dst, boolean deleteSource, Configuration conf)
          Copy local files to a FileSystem.
static boolean copy(FileSystem srcFS, FileStatus srcStatus, FileSystem dstFS, Path dst, boolean deleteSource, boolean overwrite, Configuration conf)
          Copy files between FileSystems.
static boolean copy(FileSystem srcFS, Path[] srcs, FileSystem dstFS, Path dst, boolean deleteSource, boolean overwrite, Configuration conf)
           
static boolean copy(FileSystem srcFS, Path src, File dst, boolean deleteSource, Configuration conf)
          Copy FileSystem files to local files.
static boolean copy(FileSystem srcFS, Path src, FileSystem dstFS, Path dst, boolean deleteSource, boolean overwrite, Configuration conf)
          Copy files between FileSystems.
static boolean copy(FileSystem srcFS, Path src, FileSystem dstFS, Path dst, boolean deleteSource, Configuration conf)
          Copy files between FileSystems.
static boolean copyMerge(FileSystem srcFS, Path srcDir, FileSystem dstFS, Path dstFile, boolean deleteSource, Configuration conf, String addString)
          Copy all files in a directory to one output file (merge).
static String[] createJarWithClassPath(String inputClassPath, Path pwd, Map<String,String> callerEnv)
           
static String[] createJarWithClassPath(String inputClassPath, Path pwd, Path targetDir, Map<String,String> callerEnv)
          Create a jar file at the given path, containing a manifest with a classpath that references all specified entries.
static File createLocalTempFile(File basefile, String prefix, boolean isDeleteOnExit)
          Create a tmp file for a base file.
static boolean fullyDelete(File dir)
          Delete a directory and all its contents.
static boolean fullyDelete(File dir, boolean tryGrantPermissions)
          Delete a directory and all its contents.
static void fullyDelete(FileSystem fs, Path dir)
          Deprecated. Use FileSystem.delete(Path, boolean)
static boolean fullyDeleteContents(File dir)
          Delete the contents of a directory, not the directory itself.
static boolean fullyDeleteContents(File dir, boolean tryGrantPermissions)
          Delete the contents of a directory, not the directory itself.
static long getDU(File dir)
          Takes an input dir and returns the du on that local directory.
static String[] list(File dir)
          A wrapper for File.list().
static File[] listFiles(File dir)
          A wrapper for File.listFiles().
static String makeShellPath(File file)
          Convert a os-native filename to a path that works for the shell.
static String makeShellPath(File file, boolean makeCanonicalPath)
          Convert a os-native filename to a path that works for the shell.
static String makeShellPath(String filename)
          Convert a os-native filename to a path that works for the shell.
static String readLink(File f)
          Returns the target of the given symlink.
static void replaceFile(File src, File target)
          Move the src file to the name specified by target.
static boolean setExecutable(File f, boolean executable)
          Platform independent implementation for File.setExecutable(boolean) File#setExecutable does not work as expected on Windows.
static void setOwner(File file, String username, String groupname)
          Set the ownership on a file / directory.
static void setPermission(File f, FsPermission permission)
          Set permissions to the required value.
static boolean setReadable(File f, boolean readable)
          Platform independent implementation for File.setReadable(boolean) File#setReadable does not work as expected on Windows.
static boolean setWritable(File f, boolean writable)
          Platform independent implementation for File.setWritable(boolean) File#setWritable does not work as expected on Windows.
static Path[] stat2Paths(FileStatus[] stats)
          convert an array of FileStatus to an array of Path
static Path[] stat2Paths(FileStatus[] stats, Path path)
          convert an array of FileStatus to an array of Path.
static int symLink(String target, String linkname)
          Create a soft link between a src and destination only on a local disk.
static void unTar(File inFile, File untarDir)
          Given a Tar File as input it will untar the file in a the untar directory passed as the second parameter This utility will untar ".tar" files and ".tar.gz","tgz" files.
static void unZip(File inFile, File unzipDir)
          Given a File input it will unzip the file in a the unzip directory passed as the second parameter
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

SYMLINK_NO_PRIVILEGE

public static final int SYMLINK_NO_PRIVILEGE
See Also:
Constant Field Values
Constructor Detail

FileUtil

public FileUtil()
Method Detail

stat2Paths

public static Path[] stat2Paths(FileStatus[] stats)
convert an array of FileStatus to an array of Path

Parameters:
stats - an array of FileStatus objects
Returns:
an array of paths corresponding to the input

stat2Paths

public static Path[] stat2Paths(FileStatus[] stats,
                                Path path)
convert an array of FileStatus to an array of Path. If stats if null, return path

Parameters:
stats - an array of FileStatus objects
path - default path to return in stats is null
Returns:
an array of paths corresponding to the input

fullyDelete

public static boolean fullyDelete(File dir)
Delete a directory and all its contents. If we return false, the directory may be partially-deleted. (1) If dir is symlink to a file, the symlink is deleted. The file pointed to by the symlink is not deleted. (2) If dir is symlink to a directory, symlink is deleted. The directory pointed to by symlink is not deleted. (3) If dir is a normal file, it is deleted. (4) If dir is a normal directory, then dir and all its contents recursively are deleted.


fullyDelete

public static boolean fullyDelete(File dir,
                                  boolean tryGrantPermissions)
Delete a directory and all its contents. If we return false, the directory may be partially-deleted. (1) If dir is symlink to a file, the symlink is deleted. The file pointed to by the symlink is not deleted. (2) If dir is symlink to a directory, symlink is deleted. The directory pointed to by symlink is not deleted. (3) If dir is a normal file, it is deleted. (4) If dir is a normal directory, then dir and all its contents recursively are deleted.

Parameters:
dir - the file or directory to be deleted
tryGrantPermissions - true if permissions should be modified to delete a file.
Returns:
true on success false on failure.

readLink

public static String readLink(File f)
Returns the target of the given symlink. Returns the empty string if the given path does not refer to a symlink or there is an error accessing the symlink.

Parameters:
f - File representing the symbolic link.
Returns:
The target of the symbolic link, empty string on error or if not a symlink.

fullyDeleteContents

public static boolean fullyDeleteContents(File dir)
Delete the contents of a directory, not the directory itself. If we return false, the directory may be partially-deleted. If dir is a symlink to a directory, all the contents of the actual directory pointed to by dir will be deleted.


fullyDeleteContents

public static boolean fullyDeleteContents(File dir,
                                          boolean tryGrantPermissions)
Delete the contents of a directory, not the directory itself. If we return false, the directory may be partially-deleted. If dir is a symlink to a directory, all the contents of the actual directory pointed to by dir will be deleted.

Parameters:
tryGrantPermissions - if 'true', try grant +rwx permissions to this and all the underlying directories before trying to delete their contents.

fullyDelete

@Deprecated
public static void fullyDelete(FileSystem fs,
                                          Path dir)
                        throws IOException
Deprecated. Use FileSystem.delete(Path, boolean)

Recursively delete a directory.

Parameters:
fs - FileSystem on which the path is present
dir - directory to recursively delete
Throws:
IOException

copy

public static boolean copy(FileSystem srcFS,
                           Path src,
                           FileSystem dstFS,
                           Path dst,
                           boolean deleteSource,
                           Configuration conf)
                    throws IOException
Copy files between FileSystems.

Throws:
IOException

copy

public static boolean copy(FileSystem srcFS,
                           Path[] srcs,
                           FileSystem dstFS,
                           Path dst,
                           boolean deleteSource,
                           boolean overwrite,
                           Configuration conf)
                    throws IOException
Throws:
IOException

copy

public static boolean copy(FileSystem srcFS,
                           Path src,
                           FileSystem dstFS,
                           Path dst,
                           boolean deleteSource,
                           boolean overwrite,
                           Configuration conf)
                    throws IOException
Copy files between FileSystems.

Throws:
IOException

copy

public static boolean copy(FileSystem srcFS,
                           FileStatus srcStatus,
                           FileSystem dstFS,
                           Path dst,
                           boolean deleteSource,
                           boolean overwrite,
                           Configuration conf)
                    throws IOException
Copy files between FileSystems.

Throws:
IOException

copyMerge

public static boolean copyMerge(FileSystem srcFS,
                                Path srcDir,
                                FileSystem dstFS,
                                Path dstFile,
                                boolean deleteSource,
                                Configuration conf,
                                String addString)
                         throws IOException
Copy all files in a directory to one output file (merge).

Throws:
IOException

copy

public static boolean copy(File src,
                           FileSystem dstFS,
                           Path dst,
                           boolean deleteSource,
                           Configuration conf)
                    throws IOException
Copy local files to a FileSystem.

Throws:
IOException

copy

public static boolean copy(FileSystem srcFS,
                           Path src,
                           File dst,
                           boolean deleteSource,
                           Configuration conf)
                    throws IOException
Copy FileSystem files to local files.

Throws:
IOException

makeShellPath

public static String makeShellPath(String filename)
                            throws IOException
Convert a os-native filename to a path that works for the shell.

Parameters:
filename - The filename to convert
Returns:
The unix pathname
Throws:
IOException - on windows, there can be problems with the subprocess

makeShellPath

public static String makeShellPath(File file)
                            throws IOException
Convert a os-native filename to a path that works for the shell.

Parameters:
file - The filename to convert
Returns:
The unix pathname
Throws:
IOException - on windows, there can be problems with the subprocess

makeShellPath

public static String makeShellPath(File file,
                                   boolean makeCanonicalPath)
                            throws IOException
Convert a os-native filename to a path that works for the shell.

Parameters:
file - The filename to convert
makeCanonicalPath - Whether to make canonical path for the file passed
Returns:
The unix pathname
Throws:
IOException - on windows, there can be problems with the subprocess

getDU

public static long getDU(File dir)
Takes an input dir and returns the du on that local directory. Very basic implementation.

Parameters:
dir - The input dir to get the disk space of this local dir
Returns:
The total disk space of the input local directory

unZip

public static void unZip(File inFile,
                         File unzipDir)
                  throws IOException
Given a File input it will unzip the file in a the unzip directory passed as the second parameter

Parameters:
inFile - The zip file as input
unzipDir - The unzip directory where to unzip the zip file.
Throws:
IOException

unTar

public static void unTar(File inFile,
                         File untarDir)
                  throws IOException
Given a Tar File as input it will untar the file in a the untar directory passed as the second parameter This utility will untar ".tar" files and ".tar.gz","tgz" files.

Parameters:
inFile - The tar file as input.
untarDir - The untar directory where to untar the tar file.
Throws:
IOException

symLink

public static int symLink(String target,
                          String linkname)
                   throws IOException
Create a soft link between a src and destination only on a local disk. HDFS does not support this. On Windows, when symlink creation fails due to security setting, we will log a warning. The return code in this case is 2.

Parameters:
target - the target for symlink
linkname - the symlink
Returns:
0 on success
Throws:
IOException

chmod

public static int chmod(String filename,
                        String perm)
                 throws IOException,
                        InterruptedException
Change the permissions on a filename.

Parameters:
filename - the name of the file to change
perm - the permission string
Returns:
the exit code from the command
Throws:
IOException
InterruptedException

chmod

public static int chmod(String filename,
                        String perm,
                        boolean recursive)
                 throws IOException
Change the permissions on a file / directory, recursively, if needed.

Parameters:
filename - name of the file whose permissions are to change
perm - permission string
recursive - true, if permissions should be changed recursively
Returns:
the exit code from the command.
Throws:
IOException

setOwner

public static void setOwner(File file,
                            String username,
                            String groupname)
                     throws IOException
Set the ownership on a file / directory. User name and group name cannot both be null.

Parameters:
file - the file to change
username - the new user owner name
groupname - the new group owner name
Throws:
IOException

setReadable

public static boolean setReadable(File f,
                                  boolean readable)
Platform independent implementation for File.setReadable(boolean) File#setReadable does not work as expected on Windows.

Parameters:
f - input file
readable -
Returns:
true on success, false otherwise

setWritable

public static boolean setWritable(File f,
                                  boolean writable)
Platform independent implementation for File.setWritable(boolean) File#setWritable does not work as expected on Windows.

Parameters:
f - input file
writable -
Returns:
true on success, false otherwise

setExecutable

public static boolean setExecutable(File f,
                                    boolean executable)
Platform independent implementation for File.setExecutable(boolean) File#setExecutable does not work as expected on Windows. Note: revoking execute permission on folders does not have the same behavior on Windows as on Unix platforms. Creating, deleting or renaming a file within that folder will still succeed on Windows.

Parameters:
f - input file
executable -
Returns:
true on success, false otherwise

canRead

public static boolean canRead(File f)
Platform independent implementation for File.canRead()

Parameters:
f - input file
Returns:
On Unix, same as File.canRead() On Windows, true if process has read access on the path

canWrite

public static boolean canWrite(File f)
Platform independent implementation for File.canWrite()

Parameters:
f - input file
Returns:
On Unix, same as File.canWrite() On Windows, true if process has write access on the path

canExecute

public static boolean canExecute(File f)
Platform independent implementation for File.canExecute()

Parameters:
f - input file
Returns:
On Unix, same as File.canExecute() On Windows, true if process has execute access on the path

setPermission

public static void setPermission(File f,
                                 FsPermission permission)
                          throws IOException
Set permissions to the required value. Uses the java primitives instead of forking if group == other.

Parameters:
f - the file to change
permission - the new permissions
Throws:
IOException

createLocalTempFile

public static final File createLocalTempFile(File basefile,
                                             String prefix,
                                             boolean isDeleteOnExit)
                                      throws IOException
Create a tmp file for a base file.

Parameters:
basefile - the base file of the tmp
prefix - file name prefix of tmp
isDeleteOnExit - if true, the tmp will be deleted when the VM exits
Returns:
a newly created tmp file
Throws:
IOException - If a tmp file cannot created
See Also:
File.createTempFile(String, String, File), File.deleteOnExit()

replaceFile

public static void replaceFile(File src,
                               File target)
                        throws IOException
Move the src file to the name specified by target.

Parameters:
src - the source file
target - the target file
Throws:
IOException - If this operation fails

listFiles

public static File[] listFiles(File dir)
                        throws IOException
A wrapper for File.listFiles(). This java.io API returns null when a dir is not a directory or for any I/O error. Instead of having null check everywhere File#listFiles() is used, we will add utility API to get around this problem. For the majority of cases where we prefer an IOException to be thrown.

Parameters:
dir - directory for which listing should be performed
Returns:
list of files or empty list
Throws:
IOException - for invalid directory or for a bad disk.

list

public static String[] list(File dir)
                     throws IOException
A wrapper for File.list(). This java.io API returns null when a dir is not a directory or for any I/O error. Instead of having null check everywhere File#list() is used, we will add utility API to get around this problem. For the majority of cases where we prefer an IOException to be thrown.

Parameters:
dir - directory for which listing should be performed
Returns:
list of file names or empty string list
Throws:
IOException - for invalid directory or for a bad disk.

createJarWithClassPath

public static String[] createJarWithClassPath(String inputClassPath,
                                              Path pwd,
                                              Map<String,String> callerEnv)
                                       throws IOException
Throws:
IOException

createJarWithClassPath

public static String[] createJarWithClassPath(String inputClassPath,
                                              Path pwd,
                                              Path targetDir,
                                              Map<String,String> callerEnv)
                                       throws IOException
Create a jar file at the given path, containing a manifest with a classpath that references all specified entries. Some platforms may have an upper limit on command line length. For example, the maximum command line length on Windows is 8191 characters, but the length of the classpath may exceed this. To work around this limitation, use this method to create a small intermediate jar with a manifest that contains the full classpath. It returns the absolute path to the new jar, which the caller may set as the classpath for a new process. Environment variable evaluation is not supported within a jar manifest, so this method expands environment variables before inserting classpath entries to the manifest. The method parses environment variables according to platform-specific syntax (%VAR% on Windows, or $VAR otherwise). On Windows, environment variables are case-insensitive. For example, %VAR% and %var% evaluate to the same value. Specifying the classpath in a jar manifest does not support wildcards, so this method expands wildcards internally. Any classpath entry that ends with * is translated to all files at that path with extension .jar or .JAR.

Parameters:
inputClassPath - String input classpath to bundle into the jar manifest
pwd - Path to working directory to save jar
targetDir - path to where the jar execution will have its working dir
callerEnv - Map caller's environment variables to use for expansion
Returns:
String[] with absolute path to new jar in position 0 and unexpanded wild card entry path in position 1
Throws:
IOException - if there is an I/O error while writing the jar file


Copyright © 2014 Apache Software Foundation. All Rights Reserved.