org.apache.hadoop.mapreduce.lib.output.committer.manifest.files

Class ManifestSuccessData

java.lang.Object

org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>

org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.ManifestSuccessData

All Implemented Interfaces:

Serializable, org.apache.hadoop.fs.statistics.IOStatisticsSource

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public class ManifestSuccessData
extends org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>

Summary data saved into a _SUCCESS marker file. This is a copy of the S3A committer success data format, with a goal of being/remaining compatible. This makes it easier for tests in downstream modules to be able to parse the success files from any of the committers. This should be considered public; it is based on the S3A format, which has proven stable over time. The JSON format SHOULD be considered public and evolving with compatibility across versions. All the Java serialization data is different and may change across versions with no stability guarantees other than "manifest summaries MAY be serialized between processes with the exact same version of this binary on their classpaths." That is sufficient for testing in Spark. To aid with Java serialization, the maps and lists are exclusively those which serialize well. IOStatisticsSnapshot has a lot of complexity in marshalling there; this class doesn't worry about concurrent access so is simpler.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Name to include in persisted data, so as to differentiate from any other manifests: "org.apache.hadoop.fs.s3a.commit.files.SuccessData/1".
static int	VERSION Supported version value: 1.

Constructor Summary

Constructors

Constructor and Description
ManifestSuccessData()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.util.JsonSerialization<ManifestSuccessData>	createSerializer() Get a (usually shared) JSON serializer.
String	dumpDiagnostics(String prefix, String middle, String suffix) Dump the diagnostics (if any) to a string.
String	dumpMetrics(String prefix, String middle, String suffix) Dump the metrics (if any) to a string.
String	getCommitter()
String	getDate()
String	getDescription()
Map<String,String>	getDiagnostics()
List<Path>	getFilenamePaths() Get the list of filenames as paths.
List<String>	getFilenames()
String	getHostname()
IOStatisticsSnapshot	getIOStatistics() Return a statistics instance.
String	getJobId()
String	getJobIdSource()
Map<String,Long>	getMetrics()
String	getName()
String	getStage()
String	getState()
boolean	getSuccess() Get the success flag.
long	getTimestamp()
protected static String	joinMap(Map<String,?> map, String prefix, String middle, String suffix) Join any map of string to value into a string, sorting the keys first.
static ManifestSuccessData	load(FileSystem fs, Path path) Load an instance from a file, then validate it.
void	putDiagnostic(String key, String value) Add a diagnostics entry.
void	recordJobFailure(Throwable thrown) Note a failure by setting success flag to false, then add the exception to the diagnostics.
void	save(FileSystem fs, Path path, boolean overwrite) Save to a hadoop filesystem.
static org.apache.hadoop.util.JsonSerialization<ManifestSuccessData>	serializer() Get a JSON serializer for this class.
void	setCommitter(String committer)
void	setDate(String date)
void	setDescription(String description)
void	setDiagnostics(TreeMap<String,String> diagnostics)
void	setFilenamePaths(List<Path> paths) Set the list of filename paths.
void	setFilenames(ArrayList<String> filenames)
void	setHostname(String hostname)
void	setIOStatistics(IOStatisticsSnapshot ioStatistics)
void	setJobId(String jobId)
void	setJobIdSource(String jobIdSource)
void	setMetrics(TreeMap<String,Long> metrics)
void	setName(String name)
void	setState(String state)
void	setSuccess(boolean success) Set the success flag.
void	setTimestamp(long timestamp)
void	snapshotIOStatistics(IOStatistics iostats) Set the IOStatistics to a snapshot of the source.
byte[]	toBytes() Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.
String	toJson() To JSON.
String	toString()
ManifestSuccessData	validate() Validate the data: those fields which must be non empty, must be set.

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData

marshallPath, unmarshallPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

VERSION

public static final int VERSION

Supported version value: 1. If this is changed the value of serialVersionUID will change, to avoid deserialization problems.

See Also:

Constant Field Values

NAME

public static final String NAME

Name to include in persisted data, so as to differentiate from any other manifests: "org.apache.hadoop.fs.s3a.commit.files.SuccessData/1".

See Also:

Constant Field Values

Constructor Detail

ManifestSuccessData

public ManifestSuccessData()

Method Detail

validate

public ManifestSuccessData validate()
                             throws IOException

Description copied from class: org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData

Validate the data: those fields which must be non empty, must be set.

Specified by:

validate in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>

Returns:

the validated instance.

Throws:

IOException - if the data is invalid

createSerializer

public org.apache.hadoop.util.JsonSerialization<ManifestSuccessData> createSerializer()

Description copied from class: org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData

Get a (usually shared) JSON serializer.

Specified by:

createSerializer in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>

Returns:

a serializer. Call

toBytes

public byte[] toBytes()
               throws IOException

Description copied from class: org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData

Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.

Specified by:

toBytes in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>

Returns:

the data in a persistable form.

Throws:

IOException - serialization problem or validation failure.

toJson

public String toJson()
              throws IOException

To JSON.

Returns:

json string value.

Throws:

IOException - failure

save

public void save(FileSystem fs,
                 Path path,
                 boolean overwrite)
          throws IOException

Description copied from class: org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData

Save to a hadoop filesystem.

Specified by:

save in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>

Parameters:

fs - filesystem

path - path

overwrite - should any existing file be overwritten

Throws:

IOException - IO exception

toString

public String toString()

Overrides:

toString in class Object

dumpMetrics

public String dumpMetrics(String prefix,
                          String middle,
                          String suffix)

Dump the metrics (if any) to a string. The metrics are sorted for ease of viewing.

Parameters:

prefix - prefix before every entry

middle - string between key and value

suffix - suffix to each entry

Returns:

the dumped string

dumpDiagnostics

public String dumpDiagnostics(String prefix,
                              String middle,
                              String suffix)

Dump the diagnostics (if any) to a string.

Parameters:

prefix - prefix before every entry

middle - string between key and value

suffix - suffix to each entry

Returns:

the dumped string

joinMap

protected static String joinMap(Map<String,?> map,
                                String prefix,
                                String middle,
                                String suffix)

Join any map of string to value into a string, sorting the keys first.

Parameters:

map - map to join

prefix - prefix before every entry

middle - string between key and value

suffix - suffix to each entry

Returns:

a string for reporting.

load

public static ManifestSuccessData load(FileSystem fs,
                                       Path path)
                                throws IOException

Load an instance from a file, then validate it.

Parameters:

fs - filesystem

path - path

Returns:

the loaded instance

Throws:

IOException - IO failure

serializer

public static org.apache.hadoop.util.JsonSerialization<ManifestSuccessData> serializer()

Get a JSON serializer for this class.

Returns:

a serializer.

getName

public String getName()

setName

public void setName(String name)

getTimestamp

public long getTimestamp()

Returns:

timestamp of creation.

setTimestamp

public void setTimestamp(long timestamp)

getDate

public String getDate()

Returns:

timestamp as date; no expectation of parseability.

setDate

public void setDate(String date)

getHostname

public String getHostname()

Returns:

host which created the file (implicitly: committed the work).

setHostname

public void setHostname(String hostname)

getCommitter

public String getCommitter()

Returns:

committer name.

setCommitter

public void setCommitter(String committer)

getDescription

public String getDescription()

Returns:

any description text.

setDescription

public void setDescription(String description)

getMetrics

public Map<String,Long> getMetrics()

Returns:

any metrics.

setMetrics

public void setMetrics(TreeMap<String,Long> metrics)

getFilenames

public List<String> getFilenames()

Returns:

a list of filenames in the commit.

getFilenamePaths

public List<Path> getFilenamePaths()

Get the list of filenames as paths.

Returns:

the paths.

setFilenamePaths

public void setFilenamePaths(List<Path> paths)

Set the list of filename paths.

setFilenames

public void setFilenames(ArrayList<String> filenames)

getDiagnostics

public Map<String,String> getDiagnostics()

setDiagnostics

public void setDiagnostics(TreeMap<String,String> diagnostics)

putDiagnostic

public void putDiagnostic(String key,
                          String value)

Add a diagnostics entry.

Parameters:

key - name

value - value

getJobId

public String getJobId()

Returns:

Job ID, if known.

setJobId

public void setJobId(String jobId)

getJobIdSource

public String getJobIdSource()

setJobIdSource

public void setJobIdSource(String jobIdSource)

getIOStatistics

public IOStatisticsSnapshot getIOStatistics()

Description copied from interface: org.apache.hadoop.fs.statistics.IOStatisticsSource

Return a statistics instance.

It is not a requirement that the same instance is returned every time. IOStatisticsSource.

If the object implementing this is Closeable, this method may return null if invoked on a closed object, even if it returns a valid instance when called earlier.

Returns:

an IOStatistics instance or null

setIOStatistics

public void setIOStatistics(IOStatisticsSnapshot ioStatistics)

snapshotIOStatistics

public void snapshotIOStatistics(IOStatistics iostats)

Set the IOStatistics to a snapshot of the source.

Parameters:

iostats. - Statistics; may be null.

setSuccess

public void setSuccess(boolean success)

Set the success flag.

Parameters:

success - did the job succeed?

getSuccess

public boolean getSuccess()

Get the success flag.

Returns:

did the job succeed?

getState

public String getState()

setState

public void setState(String state)

getStage

public String getStage()

recordJobFailure

public void recordJobFailure(Throwable thrown)

Note a failure by setting success flag to false, then add the exception to the diagnostics.

Parameters:

thrown - throwable