org.apache.hadoop.hdfs.net

Class DFSNetworkTopology

java.lang.Object

org.apache.hadoop.net.NetworkTopology

org.apache.hadoop.hdfs.net.DFSNetworkTopology

public class DFSNetworkTopology
extends org.apache.hadoop.net.NetworkTopology

The HDFS specific network topology class. The main purpose of doing this subclassing is to add storage-type-aware chooseRandom method. All the remaining parts should be the same. Currently a placeholder to test storage type info.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.net.NetworkTopology

org.apache.hadoop.net.NetworkTopology.InvalidTopologyException

Field Summary

Fields inherited from class org.apache.hadoop.net.NetworkTopology

DEFAULT_RACK, LOG, netlock, numOfRacks

Constructor Summary

Constructors

Constructor and Description
DFSNetworkTopology()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.net.Node	chooseRandomWithStorageType(String scope, Collection<org.apache.hadoop.net.Node> excludedNodes, org.apache.hadoop.fs.StorageType type) Randomly choose one node from scope, with specified storage type.
org.apache.hadoop.net.Node	chooseRandomWithStorageTypeTwoTrial(String scope, Collection<org.apache.hadoop.net.Node> excludedNodes, org.apache.hadoop.fs.StorageType type) Randomly choose one node from scope with the given storage type.
static DFSNetworkTopology	getInstance(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class org.apache.hadoop.net.NetworkTopology

add, chooseRandom, chooseRandom, chooseRandom, contains, countNumOfAvailableNodes, getDatanodesInRack, getDistance, getDistanceByPath, getFirstHalf, getInstance, getLastHalf, getLeaves, getNode, getNodeForNetworkLocation, getNumOfLeaves, getNumOfRacks, getRack, getWeight, hasClusterEverBeenMultiRack, incrementRacks, init, isNodeGroupAware, isOnSameNodeGroup, isOnSameRack, isSameParents, remove, sortByDistance, sortByDistanceUsingNetworkLocation, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DFSNetworkTopology

public DFSNetworkTopology()

Method Detail

getInstance

public static DFSNetworkTopology getInstance(org.apache.hadoop.conf.Configuration conf)

chooseRandomWithStorageType

public org.apache.hadoop.net.Node chooseRandomWithStorageType(String scope,
                                                     Collection<org.apache.hadoop.net.Node> excludedNodes,
                                                     org.apache.hadoop.fs.StorageType type)

Randomly choose one node from scope, with specified storage type. If scope starts with ~, choose one from the all nodes except for the ones in scope; otherwise, choose one from scope. If excludedNodes is given, choose a node that's not in excludedNodes.

Parameters:

scope - range of nodes from which a node will be chosen

excludedNodes - nodes to be excluded from

type - the storage type we search for

Returns:

the chosen node

chooseRandomWithStorageTypeTwoTrial

public org.apache.hadoop.net.Node chooseRandomWithStorageTypeTwoTrial(String scope,
                                                             Collection<org.apache.hadoop.net.Node> excludedNodes,
                                                             org.apache.hadoop.fs.StorageType type)

Randomly choose one node from scope with the given storage type. If scope starts with ~, choose one from the all nodes except for the ones in scope; otherwise, choose one from scope. If excludedNodes is given, choose a node that's not in excludedNodes. This call would make up to two calls. It first tries to get a random node (with old method) and check if it satisfies. If yes, simply return it. Otherwise, it make a second call (with the new method) by passing in a storage type. This is for better performance reason. Put in short, the key note is that the old method is faster but may take several runs, while the new method is somewhat slower, and always succeed in one trial. See HDFS-11535 for more detail.

Parameters:

scope - range of nodes from which a node will be chosen

excludedNodes - nodes to be excluded from

type - the storage type we search for

Returns:

the chosen node