Package org.apache.hadoop.io

Class IOUtils

java.lang.Object
org.apache.hadoop.io.IOUtils
@Public @Evolving public class IOUtils extends Object
An utility class for I/O related functionality.

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

  • Constructor Details

    • IOUtils

      public IOUtils()

  • Method Details

    • copyBytes

      public static void copyBytes(InputStream in, OutputStream out, int buffSize, boolean close) throws IOException
      Copies from one stream to another.
      Parameters:
      in - InputStrem to read from
      out - OutputStream to write to
      buffSize - the size of the buffer
      close - whether or not close the InputStream and OutputStream at the end. The streams are closed in the finally clause.
      Throws:
      IOException - raised on errors performing I/O.

    • copyBytes

      public static void copyBytes(InputStream in, OutputStream out, int buffSize) throws IOException
      Copies from one stream to another.
      Parameters:
      in - InputStrem to read from
      out - OutputStream to write to
      buffSize - the size of the buffer.
      Throws:
      IOException - raised on errors performing I/O.

    • copyBytes

      public static void copyBytes(InputStream in, OutputStream out, Configuration conf) throws IOException
      Copies from one stream to another. closes the input and output streams at the end.
      Parameters:
      in - InputStrem to read from
      out - OutputStream to write to
      conf - the Configuration object.
      Throws:
      IOException - raised on errors performing I/O.

    • copyBytes

      public static void copyBytes(InputStream in, OutputStream out, Configuration conf, boolean close) throws IOException
      Copies from one stream to another.
      Parameters:
      in - InputStream to read from
      out - OutputStream to write to
      conf - the Configuration object
      close - whether or not close the InputStream and OutputStream at the end. The streams are closed in the finally clause.
      Throws:
      IOException - raised on errors performing I/O.

    • copyBytes

      public static void copyBytes(InputStream in, OutputStream out, long count, boolean close) throws IOException
      Copies count bytes from one stream to another.
      Parameters:
      in - InputStream to read from
      out - OutputStream to write to
      count - number of bytes to copy
      close - whether to close the streams
      Throws:
      IOException - if bytes can not be read or written

    • wrappedReadForCompressedData

      public static int wrappedReadForCompressedData(InputStream is, byte[] buf, int off, int len) throws IOException
      Utility wrapper for reading from InputStream. It catches any errors thrown by the underlying stream (either IO or decompression-related), and re-throws as an IOException.
      Parameters:
      is - - InputStream to be read from
      buf - - buffer the data is read into
      off - - offset within buf
      len - - amount of data to be read
      Returns:
      number of bytes read
      Throws:
      IOException - raised on errors performing I/O.

    • readFully

      public static void readFully(InputStream in, byte[] buf, int off, int len) throws IOException
      Reads len bytes in a loop.
      Parameters:
      in - InputStream to read from
      buf - The buffer to fill
      off - offset from the buffer
      len - the length of bytes to read
      Throws:
      IOException - if it could not read requested number of bytes for any reason (including EOF)

    • skipFully

      public static void skipFully(InputStream in, long len) throws IOException
      Similar to readFully(). Skips bytes in a loop.
      Parameters:
      in - The InputStream to skip bytes from
      len - number of bytes to skip.
      Throws:
      IOException - if it could not skip requested number of bytes for any reason (including EOF)

    • cleanupWithLogger

      public static void cleanupWithLogger(org.slf4j.Logger logger, Closeable... closeables)
      Close the Closeable objects and ignore any Throwable or null pointers. Must only be used for cleanup in exception handlers.
      Parameters:
      logger - the log to record problems to at debug level. Can be null.
      closeables - the objects to close

    • closeStream

      public static void closeStream(Closeable stream)
      Closes the stream ignoring Throwable. Must only be called in cleaning up from exception handlers.
      Parameters:
      stream - the Stream to close

    • closeStreams

      public static void closeStreams(Closeable... streams)
      Closes the streams ignoring Throwable. Must only be called in cleaning up from exception handlers.
      Parameters:
      streams - the Streams to close

    • closeSocket

      public static void closeSocket(Socket sock)
      Closes the socket ignoring IOException
      Parameters:
      sock - the Socket to close

    • writeFully

      public static void writeFully(WritableByteChannel bc, ByteBuffer buf) throws IOException
      Write a ByteBuffer to a WritableByteChannel, handling short writes.
      Parameters:
      bc - The WritableByteChannel to write to
      buf - The input buffer
      Throws:
      IOException - On I/O error

    • writeFully

      public static void writeFully(FileChannel fc, ByteBuffer buf, long offset) throws IOException
      Write a ByteBuffer to a FileChannel at a given offset, handling short writes.
      Parameters:
      fc - The FileChannel to write to
      buf - The input buffer
      offset - The offset in the file to start writing at
      Throws:
      IOException - On I/O error

    • listDirectory

      public static List<String> listDirectory(File dir, FilenameFilter filter) throws IOException
      Return the complete list of files in a directory as strings.

      This is better than File#listDir because it does not ignore IOExceptions.

      Parameters:
      dir - The directory to list.
      filter - If non-null, the filter to use when listing this directory.
      Returns:
      The list of files in the directory.
      Throws:
      IOException - On I/O error

    • fsync

      public static void fsync(File fileToSync) throws IOException
      Ensure that any writes to the given file is written to the storage device that contains it. This method opens channel on given File and closes it once the sync is done.
      Borrowed from Uwe Schindler in LUCENE-5588
      Parameters:
      fileToSync - the file to fsync
      Throws:
      IOException - raised on errors performing I/O.

    • fsync

      public static void fsync(FileChannel channel, boolean isDir) throws IOException
      Ensure that any writes to the given file is written to the storage device that contains it. This method opens channel on given File and closes it once the sync is done. Borrowed from Uwe Schindler in LUCENE-5588
      Parameters:
      channel - Channel to sync
      isDir - if true, the given file is a directory (Channel should be opened for read and ignore IOExceptions, because not all file systems and operating systems allow to fsync on a directory)
      Throws:
      IOException - raised on errors performing I/O.

    • wrapException

      public static IOException wrapException(String path, String methodName, IOException exception)
      Takes an IOException, file/directory path, and method name and returns an IOException with the input exception as the cause and also include the file,method details. The new exception provides the stack trace of the place where the exception is thrown and some extra diagnostics information. Return instance of same exception if exception class has a public string constructor; Otherwise return an PathIOException. InterruptedIOException and PathIOException are returned unwrapped.
      Parameters:
      path - file/directory path
      methodName - method name
      exception - the caught exception.
      Returns:
      an exception to throw

    • readFullyToByteArray

      public static byte[] readFullyToByteArray(DataInput in) throws IOException
      Reads a DataInput until EOF and returns a byte array. Make sure not to pass in an infinite DataInput or this will never return.
      Parameters:
      in - A DataInput
      Returns:
      a byte array containing the data from the DataInput
      Throws:
      IOException - on I/O error, other than EOF