Skip to main content

Bonding and LACP

Bonded links carry MooseFS traffic the same way single links do. A bond is configured in the operating system and on the switches, below TCP, where MooseFS never looks. To a MooseFS module, the bond is one interface with one address. The traffic that MooseFS puts on the wire therefore decides how useful a bond is to the cluster. This page describes where a second link justifies its cost.

MooseFS puts single TCP connections on the wire​

Each pair of communicating components is joined by ordinary TCP connections. Few of those connections matter for sizing a link. A mount holds one connection to the Master Server for all its metadata. The data of a chunk read from a copy moves over one connection between the client and one Chunkserver. A chunk read from Erasure Coding parts moves over one connection per part, with four or eight connections at once. A written block goes down a chain of Chunkservers. The block uses one connection from the client to the first server and one connection per hop after that.

An aggregate speeds many connections, never one​

LACP (Link Aggregation Control Protocol) bonds commonly balance traffic per flow. An aggregate that does so keeps each TCP connection on one member link. A single connection therefore moves at the speed of one member link, however many links the bond has. The aggregate speeds up many connections at once. For MooseFS, one sequential stream between a client and a Chunkserver therefore gains nothing from a wider bond. A machine whose traffic is many connections in parallel gains up to the full width of the bond.

The Chunkserver side of a bond fills first​

The traffic of a Chunkserver is many connections by nature. It includes every client that reads or writes on that Chunkserver, plus the chain and replication traffic of other Chunkservers. Those flows spread across a bond. A Chunkserver therefore gets more network capacity than any one of its clients can use alone. On a client, the same purchase gives less. A single mount moves few flows at a time. The throughput of a mount has a ceiling of its own on the client machine that no network raises. Reading data archived in Erasure Coding is the one case where a mount behaves otherwise. The client reads the parts of a chunk in parallel, one connection for each part, so a wider bond raises the speed of that read (the read path). Otherwise, to go past the speed of one link on a client, use parallel work across mounts before bonding.

An active-backup bond adds no throughput and is not meant to. It keeps the address reachable when a link, a port, or a switch fails. That is a reason to bond, independent of everything above.

A bond does not pick which network a connection uses. That choice is address selection. Address selection is covered in a separated backbone network. The bandwidth that the cluster needs in the first place is covered in bandwidth and latency. The packet size that the links agree on is covered in MTU.