org.apache.hadoop.fs

Class FileSystem.Statistics

java.lang.Object

org.apache.hadoop.fs.FileSystem.Statistics

Enclosing class:

FileSystem

public static final class FileSystem.Statistics
extends Object

Tracks statistics about how many reads, writes, and so forth have been done in a FileSystem. Since there is only one of these objects per FileSystem, there will typically be many threads writing to this object. Almost every operation on an open file will involve a write to this object. In contrast, reading statistics is done infrequently by most programs, and not at all by others. Hence, this is optimized for writes. Each thread writes to its own thread-local area of memory. This removes contention and allows us to scale up to many, many threads. To read statistics, the reader thread totals up the contents of all of the thread-local data areas.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileSystem.Statistics.StatisticsData Statistics data.

Constructor Summary

Constructors

Constructor and Description
FileSystem.Statistics(FileSystem.Statistics other) Copy constructor.
FileSystem.Statistics(String scheme)

Method Summary

Methods

Modifier and Type	Method and Description
long	getBytesRead() Get the total number of bytes read
long	getBytesWritten() Get the total number of bytes written
FileSystem.Statistics.StatisticsData	getData() Get all statistics data MR or other frameworks can use the method to get all statistics at once.
int	getLargeReadOps() Get the number of large file system read operations such as list files under a large directory
int	getReadOps() Get the number of file system read operations such as list files
String	getScheme() Get the uri scheme associated with this statistics object.
FileSystem.Statistics.StatisticsData	getThreadStatistics() Get or create the thread-local data associated with the current thread.
int	getWriteOps() Get the number of file system write operations such as create, append rename etc.
void	incrementBytesRead(long newBytes) Increment the bytes read in the statistics
void	incrementBytesWritten(long newBytes) Increment the bytes written in the statistics
void	incrementLargeReadOps(int count) Increment the number of large read operations
void	incrementReadOps(int count) Increment the number of read operations
void	incrementWriteOps(int count) Increment the number of write operations
void	reset() Resets all statistics to 0.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FileSystem.Statistics

public FileSystem.Statistics(String scheme)

FileSystem.Statistics

public FileSystem.Statistics(FileSystem.Statistics other)

Copy constructor.

Parameters:

other - The input Statistics object which is cloned.

Method Detail

getThreadStatistics

public FileSystem.Statistics.StatisticsData getThreadStatistics()

Get or create the thread-local data associated with the current thread.

incrementBytesRead

public void incrementBytesRead(long newBytes)

Increment the bytes read in the statistics

Parameters:

newBytes - the additional bytes read

incrementBytesWritten

public void incrementBytesWritten(long newBytes)

Increment the bytes written in the statistics

Parameters:

newBytes - the additional bytes written

incrementReadOps

public void incrementReadOps(int count)

Increment the number of read operations

Parameters:

count - number of read operations

incrementLargeReadOps

public void incrementLargeReadOps(int count)

Increment the number of large read operations

Parameters:

count - number of large read operations

incrementWriteOps

public void incrementWriteOps(int count)

Increment the number of write operations

Parameters:

count - number of write operations

getBytesRead

public long getBytesRead()

Get the total number of bytes read

Returns:

the number of bytes

getBytesWritten

public long getBytesWritten()

Get the total number of bytes written

Returns:

the number of bytes

getReadOps

public int getReadOps()

Get the number of file system read operations such as list files

Returns:

number of read operations

getLargeReadOps

public int getLargeReadOps()

Get the number of large file system read operations such as list files under a large directory

Returns:

number of large read operations

getWriteOps

public int getWriteOps()

Get the number of file system write operations such as create, append rename etc.

Returns:

number of write operations

getData

public FileSystem.Statistics.StatisticsData getData()

Get all statistics data MR or other frameworks can use the method to get all statistics at once.

Returns:

the StatisticsData

toString

public String toString()

Overrides:

toString in class Object

reset

public void reset()

Resets all statistics to 0. In order to reset, we add up all the thread-local statistics data, and set rootData to the negative of that. This may seem like a counterintuitive way to reset the statsitics. Why can't we just zero out all the thread-local data? Well, thread-local data can only be modified by the thread that owns it. If we tried to modify the thread-local data from this thread, our modification might get interleaved with a read-modify-write operation done by the thread that owns the data. That would result in our update getting lost. The approach used here avoids this problem because it only ever reads (not writes) the thread-local data. Both reads and writes to rootData are done under the lock, so we're free to modify rootData from any thread that holds the lock.

getScheme

public String getScheme()

Get the uri scheme associated with this statistics object.

Returns:

the schema associated with this set of statistics