Bandwidth and latency budget

MooseFS sends all of its traffic over TCP/IP, so any network that carries IP can carry the cluster traffic, InfiniBand fabrics included. The faster and the more reliable that network is, the better the cluster performs. Connect the servers through the same switch when the site allows it. Keep the latency between the servers low when the site does not. Beyond that general rule, bandwidth and latency limit different workloads, so budget them separately. Budget the bandwidth from the bytes that the applications move. Budget the latency from the operations that the applications issue. One requirement is a constraint rather than a budget: the MTU. It has a page of its own.

Data needs bandwidth, metadata low latency

File data never passes through the Master Server. A client exchanges file data directly with the Chunkservers. A write continues from Chunkserver to Chunkserver. Most of the control traffic that the Master Server exchanges is sent in packets of tens of bytes. The answer that points a client at a chunk is 25 bytes plus 14 for every Chunkserver that the answer lists. Two exchanges are larger than that. The answer to a directory listing is one packet with up to 4096 entries. That packet reaches hundreds of kilobytes. A registering Chunkserver reports its chunks at 12 bytes each, in a series of packets. The largest single transfer on a Master Server port is the metadata image that a Metalogger downloads, every 24 hours by default. Network ports lists who connects to which port and what each one carries. Size the links to the Chunkservers for the whole throughput of the cluster. The links to the Master Server need low latency and reliability, not bandwidth.

The bandwidth budget is bytes streamed times copies written

A read crosses the network once. A write crosses it once per copy. The client streams the data to the first Chunkserver in the chain. Every Chunkserver forwards the stream to the next Chunkserver. The bytes per second that the applications write therefore enter the budget multiplied by the number of copies that the Storage Class requires. The write path describes the chain itself. The cluster adds traffic of its own on the same links when the cluster repairs redundancy. The cluster also adds traffic when it moves chunks between Chunkservers to even out how full the Chunkservers are. Evening out the disks inside one Chunkserver moves nothing over the network. Changing the redundancy format of a chunk is a third source. A copy is split into Erasure Coding parts. Parts are also joined back into a copy. Replication priorities and rebalancing describe when the cluster generates this traffic. When one link cannot carry the sum, bonding and LACP widens the link. A separated backbone network also moves the traffic of the cluster itself off the client-facing links. Copies kept in another location put the multiplied write on the long-distance link as well (the WAN cost of writes).

The latency budget is the metadata operation count

Every metadata operation that the caches of the client do not answer is one exchange with the Master Server. The client waits for the answer before the operation completes. Each such operation therefore costs a round trip to the Master Server, whatever the throughput of the links. A workload of many small files spends its time in exactly those operations (metadata operations and streaming land on different machines). That time comes on top of what the FUSE layer charges on the client machine. The pace of such a workload therefore depends on the round-trip time to the Master Server more than on any bandwidth figure. Streaming large files is the opposite case. It needs a few metadata operations, then bytes. That is a bandwidth cost rather than a latency cost.