Skip to main content

Topology and rack awareness

Topology is the map that tells MooseFS how far apart two machines on your network are.

What it is​

Topology gives an IP address a location path: a sequence of names that runs from the widest part of your network down to a single rack. Chunkservers and clients are both placed on the map by their address. Machines that share a path form one rack. MooseFS derives a distance from two paths. A client and a Chunkserver on the same machine are the nearest. Two machines in the same rack come next. The earlier the two paths split, the further apart the machines are. The literature calls this rack awareness.

You write the map yourself, in mfstopology.cfg on the Master Server. The file is optional. Without the file, MooseFS has no racks defined and treats every machine as equally near to every other machine.

Topology only measures distance. Topology does not decide which Chunkservers may hold a chunk. For that, MooseFS uses labels and Storage Classes. Topology does not divide a cluster into segments either. For that, MooseFS uses locations PRO (PRO Edition only). Of the two mechanisms, topology is by far the older. MooseFS had topology long before locations were introduced with 5.x. The two mechanisms answer different questions. A cluster can use both.

Why it matters​

The copies of one chunk are not equally cheap to reach. A copy in the client's own rack is read without crossing the link between racks. Topology lets MooseFS tell such a copy apart from a distant copy. The Master Server gives a client the Chunkservers that hold a chunk in order of their distance from that client.

The same map describes failure domains. A whole rack can lose power or its switch at once. Several copies of a chunk inside one rack therefore protect less than their number suggests. Once racks are defined, a Storage Class can require the copies of a chunk to be placed in different racks.

How MooseFS implements it​

Only the Master Server reads mfstopology.cfg. The Master Server reads the file at startup and again on every configuration reload, so changing the map needs no restart. An entry covers a single address, a range of addresses, or a whole network. An address that no entry covers gets an empty path. That address shares the empty path with every other uncovered address.

Distance is not a placement rule by itself. The Master Server gives a client a list of copies. That list is always ordered by distance. When the Master Server creates and replicates chunks, it takes distance into account only where you turn that on. The Master Server keeps copies in separate racks only where a Storage Class or a cluster-wide setting requires that. Both of those act on copies: erasure-coded parts are neither ordered by distance nor separated by rack. A topology map on its own therefore moves no data.

  • Label: the other way of telling Chunkservers apart when chunks are placed.
  • Storage Class: the policy that can require copies in different racks.
  • Location: the mechanism that divides a cluster into segments.
  • Chunkserver: the machine that a location path is assigned to.