Topology mapping and distance
Topology reduces the layout of a network to one number per pair of machines. This page describes how the Master Server uses that number. The page covers how mfstopology.cfg becomes a map of racks, how the distance between two IP addresses is counted from that map, and each decision that uses the result. Locations PRO (PRO Edition only) are a separate mechanism with a separate purpose. Locations divide a cluster. Topology only measures the cluster.
A failed load keeps the old map
The Master Server turns each line of mfstopology.cfg into a range of addresses that points at a location path. Every distinct path gets a rack ID of its own. Two machines are in one rack exactly when their addresses resolve to the same path. An address that no line covers resolves to the empty path, so all unmapped machines belong to one shared rack. The reference page covers how a line is written, and which line applies when several cover one address.
The file is read when the Master Server starts and again on every configuration reload. TOPOLOGY_FILENAME in mfsmaster.cfg names the file. A reload replaces the map only when the Master Server has read a whole new map. A line that cannot be parsed is logged and skipped. The rest of the file is still loaded. A file that cannot be opened or read at all leaves the previous map in force. Deleting the file therefore does not clear the topology of a running Master Server. Only a restart without the file does.
Only the Master Server reads the file. The position of Chunkservers and clients on the map comes from the source address of their connection. Chunkservers and clients have no topology of their own.
Distance is counted in whole path components
The distance between two IP addresses depends on the paths that the map resolves them to:
- The same address: distance 0.
- Different addresses, the same path: distance 1. This includes two unmapped machines, whose paths are both empty.
- Different paths: drop the leading components that the two paths share. The distance is then one more than the number of components left on the longer side.
Components match whole or not at all. CHI and CHIP share nothing. A rack is a whole path, not its last component. Machines mapped to US alone form one rack, the same way as machines mapped to a four-component path.
| Paths | Shared start | Longest remainder | Distance |
|---|---|---|---|
US|CHI|R1 and US|CHI|R2 | US|CHI | R1 | 2 |
US|CHI|R1 and US|NYC|R7 | US | CHI|R1 | 3 |
US|CHI|R1 and US | US | CHI|R1 | 3 |
US|CHI|R1 and DE|BER|R2 | nothing | US|CHI|R1 | 4 |
US and an unmapped machine | nothing | US | 2 |
The earlier the paths diverge, the greater the distance. The largest distance that the map can give two machines is one more than the length of its longest path. The empty path has no components, so the whole remainder is the rack's own path. An unmapped machine is therefore at a distance of one plus that rack's depth: 2 from a rack mapped to US alone, 4 from US|CHI|R1. Because all unmapped machines share the empty path, two of them are at distance 1 from each other. That is as near as two machines can be without sharing an address. An instance without the file is entirely that case: one rack that holds every machine.
The Master Server reads mfstopology.cfg and sorts the copies before answering a client: own location first (PRO), then nearer before farther, at random among equals.
The Master Server never sends the distances. The client reorders only by preferred labels and its own load.
A client is handed an order, not distances
When a client asks for an existing chunk, to read or to write, the Master Server sorts the copies before answering. In an instance divided into locations PRO (PRO Edition only), the copies in the client's own location come before the rest. Each group is ordered by its Chunkservers' distance from the client. Copies at equal distances are ordered at random, again for each request, so no single copy among the equally near copies is favored. Erasure Coding parts are listed by part number instead. Distance plays no role in their order.
The distances themselves stay on the Master Server. The list includes addresses, ports, versions, and labels. The client reorders the list only by information that it has without the map. Chunkservers that match the client's preferred labels come first. For a read, among servers of equal label rank, the server with the fewest operations already running from this mount comes first. For a write, servers of equal label rank keep the Master Server's order. A read goes to the first server on the resulting list. That server is also first in the chain that a write goes down. Where address remapping gives a client a different address for a Chunkserver, distance is measured to the remapped address. The client connects to that address.
Replication measures distance only when asked to
REPLICATIONS_RESPECT_TOPOLOGY in mfsmaster.cfg decides whether a repair uses the map. The default value is 0. The page on replication priorities covers where in the repair the servers are chosen, and the random draw of the source that 0 leaves in place. At 1, the destination is chosen the same way as for any repair. The source becomes the eligible copy closest to that destination. The distance is then measured between two Chunkservers, so no client is involved. A copy on a regular disk is still preferred over a copy on a disk marked for removal, whatever their distances. Copies at the same distance are chosen at random.
At 2, the map also affects the destination. A candidate destination in the same rack as one of the copies that the repair could read from is tried before the rest. Such a destination is at distance exactly 1 from that copy. The source rule of 1 still applies, so both ends of the transfer are then in one rack. The repair therefore crosses no link between racks. Distance 0 does not count as near for that test. A Chunkserver on the same machine as a copy is deliberately sorted with the far servers. Only the same rack counts as near.
Both settings order preferences rather than restrict candidates. Both settings act on whole copies. Rebuilding an Erasure Coding part does not use the map. Creating a chunk uses the map through a setting of its own, CREATIONS_RESPECT_TOPOLOGY. The page on allocation and space selection describes that pass. Candidates closer to the writing client than the distance that you set are promoted. None are removed.
Racks separate copies by identity, not by distance
The map gives one more thing: the rack ID itself, compared for equality with no distance involved. A Storage Class whose definition distinguishes copies by rack applies the same test as the cluster-wide CHUNKS_UNIQUE_MODE set to 2. That setting is kept for compatibility and overrides the classes. Two Chunkservers count as one place when their addresses resolve to the same rack. Under that test, the servers of one rack count as a single candidate when copies are assigned. The copies of one chunk are therefore placed in different racks. The assignment itself is the matching described in label expressions and matching. The cross-server rebalance applies the same test and refuses a destination in a rack that already holds a copy of the chunk being moved. Distinguishing applies to full copies only. Erasure Coding parts are exempt here as well.