org.apache.hadoop.io.file.tfile
Class TFile.Writer

java.lang.Object
  org.apache.hadoop.io.file.tfile.TFile.Writer

All Implemented Interfaces:

Closeable

Enclosing class:

TFile

public static class TFile.Writer
extends Object
implements Closeable

TFile Writer.

Constructor Summary

TFile.Writer(FSDataOutputStream fsdos, int minBlockSize, String compressName, String comparator, Configuration conf)
Constructor

Method Summary

void	append(byte[] key, byte[] value) Adding a new key-value pair to the TFile.
void	append(byte[] key, int koff, int klen, byte[] value, int voff, int vlen) Adding a new key-value pair to TFile.
void	close() Close the Writer.
DataOutputStream	prepareAppendKey(int length) Obtain an output stream for writing a key into TFile.
DataOutputStream	prepareAppendValue(int length) Obtain an output stream for writing a value into TFile.
DataOutputStream	prepareMetaBlock(String name) Obtain an output stream for creating a meta block.
DataOutputStream	prepareMetaBlock(String name, String compressName) Obtain an output stream for creating a meta block.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TFile.Writer

public TFile.Writer(FSDataOutputStream fsdos,
                    int minBlockSize,
                    String compressName,
                    String comparator,
                    Configuration conf)
             throws IOException

Constructor

Parameters:

fsdos - output stream for writing. Must be at position 0.

minBlockSize - Minimum compressed block size in bytes. A compression block will not be closed until it reaches this size except for the last block.

compressName - Name of the compression algorithm. Must be one of the strings returned by TFile.getSupportedCompressionAlgorithms().

comparator - Leave comparator as null or empty string if TFile is not sorted. Otherwise, provide the string name for the comparison algorithm for keys. Two kinds of comparators are supported.

• Algorithmic comparator: binary comparators that is language independent. Currently, only "memcmp" is supported.
• Language-specific comparator: binary comparators that can only be constructed in specific language. For Java, the syntax is "jclass:", followed by the class name of the RawComparator. Currently, we only support RawComparators that can be constructed through the default constructor (with no parameters). Parameterized RawComparators such as WritableComparator or JavaSerializationComparator may not be directly used. One should write a wrapper class that inherits from such classes and use its default constructor to perform proper initialization.

conf - The configuration object.

Throws:

IOException

Method Detail

close

public void close()
           throws IOException

Close the Writer. Resources will be released regardless of the exceptions being thrown. Future close calls will have no effect. The underlying FSDataOutputStream is not closed.

Specified by:

close in interface Closeable

Throws:

IOException

append

public void append(byte[] key,
                   byte[] value)
            throws IOException

Adding a new key-value pair to the TFile. This is synonymous to append(key, 0, key.length, value, 0, value.length)

Parameters:

key - Buffer for key.

value - Buffer for value.

Throws:

IOException

append

public void append(byte[] key,
                   int koff,
                   int klen,
                   byte[] value,
                   int voff,
                   int vlen)
            throws IOException

Adding a new key-value pair to TFile.

Parameters:

key - buffer for key.

koff - offset in key buffer.

klen - length of key.

value - buffer for value.

voff - offset in value buffer.

vlen - length of value.

Throws:

IOException - Upon IO errors.

If an exception is thrown, the TFile will be in an inconsistent state. The only legitimate call after that would be close

prepareAppendKey

public DataOutputStream prepareAppendKey(int length)
                                  throws IOException

Obtain an output stream for writing a key into TFile. This may only be called when there is no active Key appending stream or value appending stream.

Parameters:

length - The expected length of the key. If length of the key is not known, set length = -1. Otherwise, the application must write exactly as many bytes as specified here before calling close on the returned output stream.

Returns:

The key appending output stream.

Throws:

IOException

prepareAppendValue

public DataOutputStream prepareAppendValue(int length)
                                    throws IOException

Obtain an output stream for writing a value into TFile. This may only be called right after a key appending operation (the key append stream must be closed).

Parameters:

length - The expected length of the value. If length of the value is not known, set length = -1. Otherwise, the application must write exactly as many bytes as specified here before calling close on the returned output stream. Advertising the value size up-front guarantees that the value is encoded in one chunk, and avoids intermediate chunk buffering.

Throws:

IOException

prepareMetaBlock

public DataOutputStream prepareMetaBlock(String name,
                                         String compressName)
                                  throws IOException,
                                         MetaBlockAlreadyExists

Obtain an output stream for creating a meta block. This function may not be called when there is a key append stream or value append stream active. No more key-value insertion is allowed after a meta data block has been added to TFile.

Parameters:

name - Name of the meta block.

compressName - Name of the compression algorithm to be used. Must be one of the strings returned by TFile.getSupportedCompressionAlgorithms().

Returns:

A DataOutputStream that can be used to write Meta Block data. Closing the stream would signal the ending of the block.

Throws:

IOException

MetaBlockAlreadyExists - the Meta Block with the same name already exists.

prepareMetaBlock

public DataOutputStream prepareMetaBlock(String name)
                                  throws IOException,
                                         MetaBlockAlreadyExists

Obtain an output stream for creating a meta block. This function may not be called when there is a key append stream or value append stream active. No more key-value insertion is allowed after a meta data block has been added to TFile. Data will be compressed using the default compressor as defined in Writer's constructor.

Parameters:

name - Name of the meta block.

Returns:

A DataOutputStream that can be used to write Meta Block data. Closing the stream would signal the ending of the block.

Throws:

IOException

MetaBlockAlreadyExists - the Meta Block with the same name already exists.