Separated backbone network

A MooseFS cluster generates traffic of its own. Chunkservers re-create the copies that a lost disk held, move chunks when the cluster rebalances, and stream data between one another whenever redundancy is repaired. That traffic either shares the client network or runs on a backbone of its own. Make this choice when you design the network, because the choice sets how many interfaces the server machines have and how they are wired. MooseFS already supports the split. A Master Server and a Chunkserver are each told which addresses they accept connections on. A Metalogger and a mount accept no connections at all. Each of them is told which local address it connects from. The Master Server can also give clients Chunkserver addresses that differ from the addresses that the servers registered and use to reach one another. The mechanism and its limits are described in the MooseFS protocol and its ports. This page covers when to build the split.

Throughput sizes the data path, latency the metadata

The network ports lists what each port carries and how much traffic to expect on it. In outline: the Chunkserver port carries the chunk data, so the network that this port runs on is sized against the whole throughput of the cluster. The Master Server's ports carry metadata and control. That traffic is small in volume, but every client operation waits on it. A backbone therefore separates two different concerns. Each side of the split is budgeted differently. The figures are in bandwidth and latency.

The cluster's own traffic peaks when the cluster is degraded

Losing a disk starts a repair that re-creates everything the disk held. Rebalancing moves chunks toward evenness whenever servers fill unevenly. Repair and rebalancing both flow between Chunkservers on the same port that serves client reads and writes. On a shared network, a repair therefore competes with the applications at the moment when the cluster has already lost a disk. A backbone removes that competition. Replication and rebalancing keep to the addresses that the Chunkservers registered. Whoever asks is given that registered address. On a backbone, the registered address is a backbone address. To give a client an address in its own network instead, configure address rewriting in mfsipmap.cfg or with MULTILAN_BITS and MULTILAN_CLASSES (which address clients dial). The traffic the cluster generates on its own therefore runs on the backbone interfaces. That page also describes one internal flow that stays on the client side of the split. The copies made by a chained write follow the addresses that the writing client was given.

An unroutable backbone also isolates the servers

A mount connects to the Master Server's client port and to the Chunkserver addresses that the mount is given. The mount never discovers a server on its own. Where address rewriting gives a mount addresses in its own network, nothing that a client does requires reaching the backbone. Keeping the backbone unroutable from the client networks then costs the clients nothing. An unroutable backbone also keeps the ports that the servers use among themselves out of the clients' reach. Without that rewriting, a mount is given backbone addresses, so the backbone has to stay routable from the client networks. For the ports that a client can reach, authentication between modules covers what a connection must present.

Separate networks are not racks and not locations

A backbone changes which wires the traffic runs on, not how the cluster places data. Telling the Master Server which machines are near one another, so that reads and repairs can prefer short paths, is topology mapping and distance. Dividing a cluster that spans sites, with redundancy maintained in each of them, is a location PRO (PRO Edition only) question: when locations help.