@InterfaceAudience.Public @InterfaceStability.Unstable public class ManifestSuccessData extends org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>
_SUCCESS
marker file.
This is a copy of the S3A committer success data format, with
a goal of being/remaining compatible.
This makes it easier for tests in downstream modules to
be able to parse the success files from any of the committers.
This should be considered public; it is based on the S3A
format, which has proven stable over time.
The JSON format SHOULD be considered public and evolving
with compatibility across versions.
All the Java serialization data is different and may change
across versions with no stability guarantees other than
"manifest summaries MAY be serialized between processes with
the exact same version of this binary on their classpaths."
That is sufficient for testing in Spark.
To aid with Java serialization, the maps and lists are
exclusively those which serialize well.
IOStatisticsSnapshot has a lot of complexity in marshalling
there; this class doesn't worry about concurrent access
so is simpler.Modifier and Type | Field and Description |
---|---|
static String |
NAME
Name to include in persisted data, so as to differentiate from
any other manifests: "org.apache.hadoop.fs.s3a.commit.files.SuccessData/1".
|
static int |
VERSION
Supported version value: 1.
|
Constructor and Description |
---|
ManifestSuccessData() |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.util.JsonSerialization<ManifestSuccessData> |
createSerializer()
Get a (usually shared) JSON serializer.
|
String |
dumpDiagnostics(String prefix,
String middle,
String suffix)
Dump the diagnostics (if any) to a string.
|
String |
dumpMetrics(String prefix,
String middle,
String suffix)
Dump the metrics (if any) to a string.
|
String |
getCommitter() |
String |
getDate() |
String |
getDescription() |
Map<String,String> |
getDiagnostics() |
List<Path> |
getFilenamePaths()
Get the list of filenames as paths.
|
List<String> |
getFilenames() |
String |
getHostname() |
IOStatisticsSnapshot |
getIOStatistics()
Return a statistics instance.
|
String |
getJobId() |
String |
getJobIdSource() |
Map<String,Long> |
getMetrics() |
String |
getName() |
String |
getStage() |
String |
getState() |
boolean |
getSuccess()
Get the success flag.
|
long |
getTimestamp() |
protected static String |
joinMap(Map<String,?> map,
String prefix,
String middle,
String suffix)
Join any map of string to value into a string, sorting the keys first.
|
static ManifestSuccessData |
load(FileSystem fs,
Path path)
Load an instance from a file, then validate it.
|
void |
putDiagnostic(String key,
String value)
Add a diagnostics entry.
|
void |
recordJobFailure(Throwable thrown)
Note a failure by setting success flag to false,
then add the exception to the diagnostics.
|
void |
save(FileSystem fs,
Path path,
boolean overwrite)
Save to a hadoop filesystem.
|
static org.apache.hadoop.util.JsonSerialization<ManifestSuccessData> |
serializer()
Get a JSON serializer for this class.
|
void |
setCommitter(String committer) |
void |
setDate(String date) |
void |
setDescription(String description) |
void |
setDiagnostics(TreeMap<String,String> diagnostics) |
void |
setFilenamePaths(List<Path> paths)
Set the list of filename paths.
|
void |
setFilenames(ArrayList<String> filenames) |
void |
setHostname(String hostname) |
void |
setIOStatistics(IOStatisticsSnapshot ioStatistics) |
void |
setJobId(String jobId) |
void |
setJobIdSource(String jobIdSource) |
void |
setMetrics(TreeMap<String,Long> metrics) |
void |
setName(String name) |
void |
setState(String state) |
void |
setSuccess(boolean success)
Set the success flag.
|
void |
setTimestamp(long timestamp) |
void |
snapshotIOStatistics(IOStatistics iostats)
Set the IOStatistics to a snapshot of the source.
|
byte[] |
toBytes()
Serialize to JSON and then to a byte array, after performing a
preflight validation of the data to be saved.
|
String |
toJson()
To JSON.
|
String |
toString() |
ManifestSuccessData |
validate()
Validate the data: those fields which must be non empty, must be set.
|
public static final int VERSION
serialVersionUID
will change,
to avoid deserialization problems.public static final String NAME
public ManifestSuccessData validate() throws IOException
org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData
validate
in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>
IOException
- if the data is invalidpublic org.apache.hadoop.util.JsonSerialization<ManifestSuccessData> createSerializer()
org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData
createSerializer
in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>
public byte[] toBytes() throws IOException
org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData
toBytes
in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>
IOException
- serialization problem or validation failure.public String toJson() throws IOException
IOException
- failurepublic void save(FileSystem fs, Path path, boolean overwrite) throws IOException
org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData
save
in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData<ManifestSuccessData>
fs
- filesystempath
- pathoverwrite
- should any existing file be overwrittenIOException
- IO exceptionpublic String dumpMetrics(String prefix, String middle, String suffix)
prefix
- prefix before every entrymiddle
- string between key and valuesuffix
- suffix to each entrypublic String dumpDiagnostics(String prefix, String middle, String suffix)
prefix
- prefix before every entrymiddle
- string between key and valuesuffix
- suffix to each entryprotected static String joinMap(Map<String,?> map, String prefix, String middle, String suffix)
map
- map to joinprefix
- prefix before every entrymiddle
- string between key and valuesuffix
- suffix to each entrypublic static ManifestSuccessData load(FileSystem fs, Path path) throws IOException
fs
- filesystempath
- pathIOException
- IO failurepublic static org.apache.hadoop.util.JsonSerialization<ManifestSuccessData> serializer()
public String getName()
public void setName(String name)
public long getTimestamp()
public void setTimestamp(long timestamp)
public String getDate()
public void setDate(String date)
public String getHostname()
public void setHostname(String hostname)
public String getCommitter()
public void setCommitter(String committer)
public String getDescription()
public void setDescription(String description)
public List<Path> getFilenamePaths()
public void putDiagnostic(String key, String value)
key
- namevalue
- valuepublic String getJobId()
public void setJobId(String jobId)
public String getJobIdSource()
public void setJobIdSource(String jobIdSource)
public IOStatisticsSnapshot getIOStatistics()
org.apache.hadoop.fs.statistics.IOStatisticsSource
It is not a requirement that the same instance is returned every time.
IOStatisticsSource
.
If the object implementing this is Closeable, this method may return null if invoked on a closed object, even if it returns a valid instance when called earlier.
public void setIOStatistics(IOStatisticsSnapshot ioStatistics)
public void snapshotIOStatistics(IOStatistics iostats)
iostats.
- Statistics; may be null.public void setSuccess(boolean success)
success
- did the job succeed?public boolean getSuccess()
public String getState()
public void setState(String state)
public String getStage()
public void recordJobFailure(Throwable thrown)
thrown
- throwableCopyright © 2024 Apache Software Foundation. All rights reserved.