org.apache.hadoop.fs

Interface BulkDelete

All Superinterfaces:

AutoCloseable, Closeable, org.apache.hadoop.fs.statistics.IOStatisticsSource

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public interface BulkDelete
extends org.apache.hadoop.fs.statistics.IOStatisticsSource, Closeable

API for bulk deletion of objects/files, but not directories. After use, call close() to release any resources and to guarantee store IOStatistics are updated.

Callers MUST have no expectation that parent directories will exist after the operation completes; if an object store needs to explicitly look for and create directory markers, that step will be omitted.

Be aware that on some stores (AWS S3) each object listed in a bulk delete counts against the write IOPS limit; large page sizes are counterproductive here, as are attempts at parallel submissions across multiple threads.

See Also:

HADOOP-16823. Large DeleteObject requests are their own Thundering Herd

Method Summary

Modifier and Type	Method and Description
Path	basePath() Base path of a bulk delete operation.
List<Map.Entry<Path,String>>	bulkDelete(Collection<Path> paths) Delete a list of files/objects.
int	pageSize() The maximum number of objects/files to delete in a single request.

Methods inherited from interface org.apache.hadoop.fs.statistics.IOStatisticsSource

getIOStatistics

Methods inherited from interface java.io.Closeable

close

Method Detail

pageSize

int pageSize()

The maximum number of objects/files to delete in a single request.

Returns:

a number greater than zero.

basePath

Path basePath()

Base path of a bulk delete operation. All paths submitted in bulkDelete(Collection) must be under this path.

Returns:

base path of a bulk delete operation.

bulkDelete

List<Map.Entry<Path,String>> bulkDelete(Collection<Path> paths)
                                 throws IOException,
                                        IllegalArgumentException

Delete a list of files/objects.

• Files must be under the path provided in basePath().
• The size of the list must be equal to or less than the page size declared in pageSize().
• Directories are not supported; the outcome of attempting to delete directories is undefined (ignored; undetected, listed as failures...).
• The operation is not atomic.
• The operation is treated as idempotent: network failures may trigger resubmission of the request -any new objects created under a path in the list may then be deleted.
• There is no guarantee that any parent directories exist after this call.

Parameters:

paths - list of paths which must be absolute and under the base path. provided in basePath().

Returns:

a list of paths which failed to delete, with the exception message.

Throws:

IOException - IO problems including networking, authentication and more.

IllegalArgumentException - if a path argument is invalid.