org.apache.hadoop.util.functional

Class RemoteIterators

java.lang.Object

org.apache.hadoop.util.functional.RemoteIterators

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public final class RemoteIterators
extends Object

A set of remote iterators supporting transformation and filtering, with IOStatisticsSource passthrough, and of conversions of the iterators to lists/arrays and of performing actions on the values.

This aims to make it straightforward to use lambda-expressions to transform the results of an iterator, without losing the statistics in the process, and to chain the operations together.

The closeable operation will be passed through RemoteIterators which wrap other RemoteIterators. This is to support any iterator which can be closed to release held connections, file handles etc. Unless client code is written to assume that RemoteIterator instances may be closed, this is not likely to be broadly used. It is added to make it possible to adopt this feature in a managed way.

One notable feature is that the foreach(RemoteIterator, ConsumerRaisingIOE) method will LOG at debug any IOStatistics provided by the iterator, if such statistics are provided. There's no attempt at retrieval and logging if the LOG is not set to debug, so it is a zero cost feature unless the logger org.apache.hadoop.fs.functional.RemoteIterators is at DEBUG.

Based on the S3A Listing code, and some some work on moving other code to using iterative listings so as to pick up the statistics.

Method Summary

Modifier and Type	Method and Description
static <T> void	cleanupRemoteIterator(org.apache.hadoop.fs.RemoteIterator<T> source) Clean up after an iteration.
static <S> org.apache.hadoop.fs.RemoteIterator<S>	closingRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator, Closeable toClose) This adds an extra close operation alongside the passthrough to any Closeable.close() method supported by the source iterator.
static <S> org.apache.hadoop.fs.RemoteIterator<S>	filteringRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator, org.apache.hadoop.util.functional.FunctionRaisingIOE<? super S,Boolean> filter) Create a RemoteIterator from a RemoteIterator and a filter function which returns true for every element to be passed through.
static <T> long	foreach(org.apache.hadoop.fs.RemoteIterator<T> source, org.apache.hadoop.util.functional.ConsumerRaisingIOE<? super T> consumer) Apply an operation to all values of a RemoteIterator.
static <S> org.apache.hadoop.fs.RemoteIterator<S>	haltableRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator, org.apache.hadoop.util.functional.CallableRaisingIOE<Boolean> continueWork) Wrap an iterator with one which adds a continuation probe.
static <S,T> org.apache.hadoop.fs.RemoteIterator<T>	mappingRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator, org.apache.hadoop.util.functional.FunctionRaisingIOE<? super S,T> mapper) Create an iterator from an iterator and a transformation function.
static org.apache.hadoop.fs.RemoteIterator<Long>	rangeExcludingIterator(long start, long excludedFinish) A remote iterator which simply counts up, stopping once the value is greater than the value of excludedFinish.
static <T> org.apache.hadoop.fs.RemoteIterator<T>	remoteIteratorFromArray(T[] array) Create a remote iterator from an array.
static <T> org.apache.hadoop.fs.RemoteIterator<T>	remoteIteratorFromIterable(Iterable<T> iterable) Create a remote iterator from a java.util.Iterable -e.g.
static <T> org.apache.hadoop.fs.RemoteIterator<T>	remoteIteratorFromIterator(Iterator<T> iterator) Create a remote iterator from a java.util.Iterator.
static <T> org.apache.hadoop.fs.RemoteIterator<T>	remoteIteratorFromSingleton(T singleton) Create an iterator from a singleton.
static <T> T[]	toArray(org.apache.hadoop.fs.RemoteIterator<T> source, T[] a) Build an array from a RemoteIterator.
static <T> List<T>	toList(org.apache.hadoop.fs.RemoteIterator<T> source) Build a list from a RemoteIterator.
static <S,T> org.apache.hadoop.fs.RemoteIterator<T>	typeCastingRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator) Create a RemoteIterator from a RemoteIterator, casting the type in the process.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

remoteIteratorFromSingleton

public static <T> org.apache.hadoop.fs.RemoteIterator<T> remoteIteratorFromSingleton(@Nullable
                                                                                     T singleton)

Create an iterator from a singleton.

Type Parameters:

T - type

Parameters:

singleton - instance

Returns:

a remote iterator

remoteIteratorFromIterator

public static <T> org.apache.hadoop.fs.RemoteIterator<T> remoteIteratorFromIterator(Iterator<T> iterator)

Create a remote iterator from a java.util.Iterator.

Type Parameters:

T - type

Parameters:

iterator - iterator.

Returns:

a remote iterator

remoteIteratorFromIterable

public static <T> org.apache.hadoop.fs.RemoteIterator<T> remoteIteratorFromIterable(Iterable<T> iterable)

Create a remote iterator from a java.util.Iterable -e.g. a list or other collection.

Type Parameters:

T - type

Parameters:

iterable - iterable.

Returns:

a remote iterator

remoteIteratorFromArray

public static <T> org.apache.hadoop.fs.RemoteIterator<T> remoteIteratorFromArray(T[] array)

Create a remote iterator from an array.

Type Parameters:

T - type

Parameters:

array - array.

Returns:

a remote iterator

mappingRemoteIterator

public static <S,T> org.apache.hadoop.fs.RemoteIterator<T> mappingRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator,
                                                                                 org.apache.hadoop.util.functional.FunctionRaisingIOE<? super S,T> mapper)

Create an iterator from an iterator and a transformation function.

Type Parameters:

S - source type

T - result type

Parameters:

iterator - source

mapper - transformation

Returns:

a remote iterator

typeCastingRemoteIterator

public static <S,T> org.apache.hadoop.fs.RemoteIterator<T> typeCastingRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator)

Create a RemoteIterator from a RemoteIterator, casting the type in the process. This is to help with filesystem API calls where overloading causes confusion (e.g. listStatusIterator())

Type Parameters:

S - source type

T - result type

Parameters:

iterator - source

Returns:

a remote iterator

filteringRemoteIterator

public static <S> org.apache.hadoop.fs.RemoteIterator<S> filteringRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator,
                                                                                 org.apache.hadoop.util.functional.FunctionRaisingIOE<? super S,Boolean> filter)

Create a RemoteIterator from a RemoteIterator and a filter function which returns true for every element to be passed through.

Elements are filtered in the hasNext() method; if not used the filtering will be done on demand in the next() call.

Type Parameters:

S - type

Parameters:

iterator - source

filter - filter

Returns:

a remote iterator

closingRemoteIterator

public static <S> org.apache.hadoop.fs.RemoteIterator<S> closingRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator,
                                                                               Closeable toClose)

This adds an extra close operation alongside the passthrough to any Closeable.close() method supported by the source iterator.

Type Parameters:

S - source type.

Parameters:

iterator - source

toClose - extra object to close.

Returns:

a new iterator

haltableRemoteIterator

public static <S> org.apache.hadoop.fs.RemoteIterator<S> haltableRemoteIterator(org.apache.hadoop.fs.RemoteIterator<S> iterator,
                                                                                org.apache.hadoop.util.functional.CallableRaisingIOE<Boolean> continueWork)

Wrap an iterator with one which adds a continuation probe. This allows work to exit fast without complicated breakout logic

Type Parameters:

S - source type.

Parameters:

iterator - source

continueWork - predicate which will trigger a fast halt if it returns false.

Returns:

a new iterator

rangeExcludingIterator

public static org.apache.hadoop.fs.RemoteIterator<Long> rangeExcludingIterator(long start,
                                                                               long excludedFinish)

A remote iterator which simply counts up, stopping once the value is greater than the value of excludedFinish. This is primarily for tests or when submitting work into a TaskPool. equivalent to

   for(long l = start, l < excludedFinish; l++) yield l;
 

Parameters:

start - start value

excludedFinish - excluded finish

Returns:

an iterator which returns longs from [start, finish)

toList

public static <T> List<T> toList(org.apache.hadoop.fs.RemoteIterator<T> source)
                          throws IOException

Build a list from a RemoteIterator.

Type Parameters:

T - type

Parameters:

source - source iterator

Returns:

a list of the values.

Throws:

IOException - if the source RemoteIterator raises it.

toArray

public static <T> T[] toArray(org.apache.hadoop.fs.RemoteIterator<T> source,
                              T[] a)
                       throws IOException

Build an array from a RemoteIterator.

Type Parameters:

T - type

Parameters:

source - source iterator

a - destination array; if too small a new array of the same type is created

Returns:

an array of the values.

Throws:

IOException - if the source RemoteIterator raises it.

foreach

public static <T> long foreach(org.apache.hadoop.fs.RemoteIterator<T> source,
                               org.apache.hadoop.util.functional.ConsumerRaisingIOE<? super T> consumer)
                        throws IOException

Apply an operation to all values of a RemoteIterator. If the iterator is an IOStatisticsSource returning a non-null set of statistics, and this classes log is set to DEBUG, then the statistics of the operation are evaluated and logged at debug.

The number of entries processed is returned, as it is useful to know this, especially during tests or when reporting values to users.

This does not close the iterator afterwards.

Type Parameters:

T - type of source

Parameters:

source - iterator source

consumer - consumer of the values.

Returns:

the number of elements processed

Throws:

IOException - if the source RemoteIterator or the consumer raise one.

cleanupRemoteIterator

public static <T> void cleanupRemoteIterator(org.apache.hadoop.fs.RemoteIterator<T> source)

Clean up after an iteration. If the log is at debug, calculate and log the IOStatistics. If the iterator is closeable, cast and then cleanup the iterator

Type Parameters:

T - type of source

Parameters:

source - iterator source