org.apache.hadoop.hdfs

Class DFSUtilClient

java.lang.Object

org.apache.hadoop.hdfs.DFSUtilClient

public class DFSUtilClient
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static byte[]	EMPTY_BYTES

Constructor Summary

Constructors

Constructor and Description
DFSUtilClient()

Method Summary

Methods

Modifier and Type	Method and Description
static byte[]	byteArray2bytes(byte[][] pathComponents) Given a list of path components returns a byte array
static String	bytes2String(byte[] bytes) Converts a byte array to a string using UTF8 encoding.
static int	compareBytes(byte[] left, byte[] right) Compare two byte arrays by lexicographical order.
static org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol	createClientDatanodeProtocolProxy(org.apache.hadoop.hdfs.protocol.DatanodeID datanodeid, org.apache.hadoop.conf.Configuration conf, int socketTimeout, boolean connectToDnViaHostname) Create ClientDatanodeProtocol proxy using kerberos ticket
static org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol	createClientDatanodeProtocolProxy(org.apache.hadoop.hdfs.protocol.DatanodeID datanodeid, org.apache.hadoop.conf.Configuration conf, int socketTimeout, boolean connectToDnViaHostname, org.apache.hadoop.hdfs.protocol.LocatedBlock locatedBlock) Create a ClientDatanodeProtocol proxy
static org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol	createClientDatanodeProtocolProxy(InetSocketAddress addr, org.apache.hadoop.security.UserGroupInformation ticket, org.apache.hadoop.conf.Configuration conf, SocketFactory factory) Create a ClientDatanodeProtocol proxy
static org.apache.hadoop.hdfs.protocol.ReconfigurationProtocol	createReconfigurationProtocolProxy(InetSocketAddress addr, org.apache.hadoop.security.UserGroupInformation ticket, org.apache.hadoop.conf.Configuration conf, SocketFactory factory)
static URI	createUri(String scheme, InetSocketAddress address) Create a URI from the scheme and address
static String	dateToIso8601String(Date date) Converts a Date into an ISO-8601 formatted datetime string.
static String	durationToString(long durationMs) Converts a time duration in milliseconds into DDD:HH:MM:SS format.
static Map<String,Map<String,InetSocketAddress>>	getAddresses(org.apache.hadoop.conf.Configuration conf, String defaultAddress, String... keys) Returns the configured address for all NameNodes in the cluster.
static Map<String,Map<String,InetSocketAddress>>	getHaNnRpcAddresses(org.apache.hadoop.conf.Configuration conf) Returns list of InetSocketAddress corresponding to HA NN RPC addresses from the configuration.
static Map<String,Map<String,InetSocketAddress>>	getHaNnWebHdfsAddresses(org.apache.hadoop.conf.Configuration conf, String scheme) Returns list of InetSocketAddress corresponding to HA NN HTTP addresses from the configuration.
static int	getIoFileBufferSize(org.apache.hadoop.conf.Configuration conf)
static Collection<String>	getNameNodeIds(org.apache.hadoop.conf.Configuration conf, String nsId) Namenode HighAvailability related configuration.
static Collection<String>	getNameServiceIds(org.apache.hadoop.conf.Configuration conf) Returns collection of nameservice Ids from the configuration.
static InetSocketAddress	getNNAddress(org.apache.hadoop.conf.Configuration conf)
static InetSocketAddress	getNNAddress(String address)
static InetSocketAddress	getNNAddress(URI filesystemURI)
static InetSocketAddress	getNNAddressCheckLogical(org.apache.hadoop.conf.Configuration conf, URI filesystemURI) Get the NN address from the URI.
static URI	getNNUri(InetSocketAddress namenode)
static float	getPercentRemaining(long remaining, long capacity) Return remaining as percentage of capacity
static float	getPercentUsed(long used, long capacity) Return used as percentage of capacity
static int	getSmallBufferSize(org.apache.hadoop.conf.Configuration conf)
static boolean	isHDFSEncryptionEnabled(org.apache.hadoop.conf.Configuration conf) Probe for HDFS Encryption being enabled; this uses the value of the option CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH , returning true if that property contains a non-empty, non-whitespace string.
static boolean	isLocalAddress(InetSocketAddress targetAddr)
static boolean	isValidName(String src) Whether the pathname is valid.
static org.apache.hadoop.fs.BlockLocation[]	locatedBlocks2Locations(List<org.apache.hadoop.hdfs.protocol.LocatedBlock> blocks) Convert a List to BlockLocation[]
static org.apache.hadoop.fs.BlockLocation[]	locatedBlocks2Locations(org.apache.hadoop.hdfs.protocol.LocatedBlocks blocks) Convert a LocatedBlocks to BlockLocations[]
static org.apache.hadoop.hdfs.net.Peer	peerFromSocket(Socket socket)
static org.apache.hadoop.hdfs.net.Peer	peerFromSocketAndKey(org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient saslClient, Socket s, org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataEncryptionKeyFactory keyFactory, org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier> blockToken, org.apache.hadoop.hdfs.protocol.DatanodeID datanodeId, int socketTimeoutMs)
static String	percent2String(double percentage) Convert percentage to a string.
static byte[]	string2Bytes(String str) Converts a string to a byte array using UTF8 encoding.
static InterruptedIOException	toInterruptedIOException(String message, InterruptedException e)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EMPTY_BYTES

public static final byte[] EMPTY_BYTES

Constructor Detail

DFSUtilClient

public DFSUtilClient()

Method Detail

string2Bytes

public static byte[] string2Bytes(String str)

Converts a string to a byte array using UTF8 encoding.

bytes2String

public static String bytes2String(byte[] bytes)

Converts a byte array to a string using UTF8 encoding.

getPercentUsed

public static float getPercentUsed(long used,
                   long capacity)

Return used as percentage of capacity

getPercentRemaining

public static float getPercentRemaining(long remaining,
                        long capacity)

Return remaining as percentage of capacity

percent2String

public static String percent2String(double percentage)

Convert percentage to a string.

getNameServiceIds

public static Collection<String> getNameServiceIds(org.apache.hadoop.conf.Configuration conf)

Returns collection of nameservice Ids from the configuration.

Parameters:

conf - configuration

Returns:

collection of nameservice Ids, or null if not specified

getNameNodeIds

public static Collection<String> getNameNodeIds(org.apache.hadoop.conf.Configuration conf,
                                String nsId)

Namenode HighAvailability related configuration. Returns collection of namenode Ids from the configuration. One logical id for each namenode in the in the HA setup.

Parameters:

conf - configuration

nsId - the nameservice ID to look at, or null for non-federated

Returns:

collection of namenode Ids

getHaNnRpcAddresses

public static Map<String,Map<String,InetSocketAddress>> getHaNnRpcAddresses(org.apache.hadoop.conf.Configuration conf)

Returns list of InetSocketAddress corresponding to HA NN RPC addresses from the configuration.

Parameters:

conf - configuration

Returns:

list of InetSocketAddresses

getHaNnWebHdfsAddresses

public static Map<String,Map<String,InetSocketAddress>> getHaNnWebHdfsAddresses(org.apache.hadoop.conf.Configuration conf,
                                                                String scheme)

Returns list of InetSocketAddress corresponding to HA NN HTTP addresses from the configuration.

Returns:

list of InetSocketAddresses

locatedBlocks2Locations

public static org.apache.hadoop.fs.BlockLocation[] locatedBlocks2Locations(org.apache.hadoop.hdfs.protocol.LocatedBlocks blocks)

Convert a LocatedBlocks to BlockLocations[]

Parameters:

blocks - a LocatedBlocks

Returns:

an array of BlockLocations

locatedBlocks2Locations

public static org.apache.hadoop.fs.BlockLocation[] locatedBlocks2Locations(List<org.apache.hadoop.hdfs.protocol.LocatedBlock> blocks)

Convert a List to BlockLocation[]

Parameters:

blocks - A List to be converted

Returns:

converted array of BlockLocation

compareBytes

public static int compareBytes(byte[] left,
               byte[] right)

Compare two byte arrays by lexicographical order.

byteArray2bytes

public static byte[] byteArray2bytes(byte[][] pathComponents)

Given a list of path components returns a byte array

getAddresses

public static Map<String,Map<String,InetSocketAddress>> getAddresses(org.apache.hadoop.conf.Configuration conf,
                                                     String defaultAddress,
                                                     String... keys)

Returns the configured address for all NameNodes in the cluster.

Parameters:

conf - configuration

defaultAddress - default address to return in case key is not found.

keys - Set of keys to look for in the order of preference

Returns:

a map(nameserviceId to map(namenodeId to InetSocketAddress))

isValidName

public static boolean isValidName(String src)

Whether the pathname is valid. Currently prohibits relative paths, names which contain a ":" or "//", or other non-canonical paths.

durationToString

public static String durationToString(long durationMs)

Converts a time duration in milliseconds into DDD:HH:MM:SS format.

dateToIso8601String

public static String dateToIso8601String(Date date)

Converts a Date into an ISO-8601 formatted datetime string.

isLocalAddress

public static boolean isLocalAddress(InetSocketAddress targetAddr)

createClientDatanodeProtocolProxy

public static org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol createClientDatanodeProtocolProxy(org.apache.hadoop.hdfs.protocol.DatanodeID datanodeid,
                                                                                       org.apache.hadoop.conf.Configuration conf,
                                                                                       int socketTimeout,
                                                                                       boolean connectToDnViaHostname,
                                                                                       org.apache.hadoop.hdfs.protocol.LocatedBlock locatedBlock)
                                                                                                throws IOException

Create a ClientDatanodeProtocol proxy

Throws:

IOException

createClientDatanodeProtocolProxy

public static org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol createClientDatanodeProtocolProxy(org.apache.hadoop.hdfs.protocol.DatanodeID datanodeid,
                                                                                       org.apache.hadoop.conf.Configuration conf,
                                                                                       int socketTimeout,
                                                                                       boolean connectToDnViaHostname)
                                                                                                throws IOException

Create ClientDatanodeProtocol proxy using kerberos ticket

Throws:

IOException

createClientDatanodeProtocolProxy

public static org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol createClientDatanodeProtocolProxy(InetSocketAddress addr,
                                                                                       org.apache.hadoop.security.UserGroupInformation ticket,
                                                                                       org.apache.hadoop.conf.Configuration conf,
                                                                                       SocketFactory factory)
                                                                                                throws IOException

Create a ClientDatanodeProtocol proxy

Throws:

IOException

createReconfigurationProtocolProxy

public static org.apache.hadoop.hdfs.protocol.ReconfigurationProtocol createReconfigurationProtocolProxy(InetSocketAddress addr,
                                                                                         org.apache.hadoop.security.UserGroupInformation ticket,
                                                                                         org.apache.hadoop.conf.Configuration conf,
                                                                                         SocketFactory factory)
                                                                                                  throws IOException

Throws:

IOException

peerFromSocket

public static org.apache.hadoop.hdfs.net.Peer peerFromSocket(Socket socket)
                                                      throws IOException

Throws:

IOException

peerFromSocketAndKey

public static org.apache.hadoop.hdfs.net.Peer peerFromSocketAndKey(org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient saslClient,
                                                   Socket s,
                                                   org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataEncryptionKeyFactory keyFactory,
                                                   org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier> blockToken,
                                                   org.apache.hadoop.hdfs.protocol.DatanodeID datanodeId,
                                                   int socketTimeoutMs)
                                                            throws IOException

Throws:

IOException

getIoFileBufferSize

public static int getIoFileBufferSize(org.apache.hadoop.conf.Configuration conf)

getSmallBufferSize

public static int getSmallBufferSize(org.apache.hadoop.conf.Configuration conf)

isHDFSEncryptionEnabled

public static boolean isHDFSEncryptionEnabled(org.apache.hadoop.conf.Configuration conf)

Probe for HDFS Encryption being enabled; this uses the value of the option CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH , returning true if that property contains a non-empty, non-whitespace string.

Parameters:

conf - configuration to probe

Returns:

true if encryption is considered enabled.

getNNAddress

public static InetSocketAddress getNNAddress(String address)

getNNAddress

public static InetSocketAddress getNNAddress(org.apache.hadoop.conf.Configuration conf)

getNNAddress

public static InetSocketAddress getNNAddress(URI filesystemURI)

Returns:

address of file system

getNNAddressCheckLogical

public static InetSocketAddress getNNAddressCheckLogical(org.apache.hadoop.conf.Configuration conf,
                                         URI filesystemURI)

Get the NN address from the URI. If the uri is logical, default address is returned. Otherwise return the DNS-resolved address of the URI.

Parameters:

conf - configuration

filesystemURI - URI of the file system

Returns:

address of file system

getNNUri

public static URI getNNUri(InetSocketAddress namenode)

createUri

public static URI createUri(String scheme,
            InetSocketAddress address)

Create a URI from the scheme and address

toInterruptedIOException

public static InterruptedIOException toInterruptedIOException(String message,
                                              InterruptedException e)