Skip to main content

MTU

Set the same MTU (Maximum Transmission Unit) on every interface that carries MooseFS traffic. The MTU is the largest packet that the network passes without fragmentation. The requirement goes beyond the servers. The switches, routers, and VLAN configurations between the servers limit the packet size the same way the interfaces do. The one value therefore has to apply across all of these devices.

An inconsistent MTU is not visible in its symptoms​

Nodes that disagree on the MTU drop packets and delay transmissions between the Master Server, the Chunkservers, and the clients. This happens above all where jumbo frames are in use. The administrator sees timeouts, connections that fail intermittently, and performance that drops under high load and on large transfers. The data integrity issues an inconsistent MTU can cause are subtle. None of these symptoms points to the setting that causes them.

Use jumbo frames only on a path you control​

Jumbo frames (an MTU of 9000 bytes) reduce the per-packet overhead of large transfers. Use them when you control every device on the communication path and can set the same MTU on all of those devices. When any device on the path is outside your control, or you cannot confirm its MTU, use 1500 bytes instead. That value is supported everywhere and usually needs no configuration at all.

IPoIB takes its MTU from the fabric​

An InfiniBand fabric reached through IPoIB presents an ordinary IP interface, but the MTU on that interface is assigned rather than chosen. In datagram mode, the MTU of the interface is the link MTU of the fabric less the four bytes of the IPoIB header. A 2048-byte fabric therefore gives 2044 bytes. A 4096-byte fabric gives 4092. Both are below an Ethernet jumbo frame rather than above it. The subnet manager assigns the link MTU. The subnet manager runs where the fabric is managed, on a switch or as OpenSM, rather than on the server. The older connected mode reached as far as 65520 bytes by segmenting inside the adapter. Adapters that run enhanced IPoIB offer datagram mode only. Use the figure that your fabric gives you rather than a target number of your own.

The adapter and the operating system decide those figures and modes, not MooseFS. This page repeats the recommendations of the hardware and OS vendors for those figures. Check the documentation of the adapters and the distribution that you run before you plan around a value. Vendor guidance describes the common case. Your fabric determines your own case. MooseFS requires only the rule above. Whichever value the fabric uses has to apply on every interface that carries MooseFS traffic. A path that crosses from IPoIB to Ethernet is therefore limited to what the Ethernet side passes.

The interface value is not the path value​

An interface set to 4092 shows the value that the card was given, not what every switch between two nodes carries. Where the path carries less, the frames that exceed the path MTU are lost. This loss causes the symptoms described above. Those symptoms do not point to the setting that causes them. Confirm the path before the cluster depends on it. Ping between a client and the Master Server and between a client and a Chunkserver. Forbid fragmentation and use a payload of the MTU less 28 bytes:

ping -M do -s 4064 <address>

Raise the payload until the packets stop arriving. The largest size that passes the test on every leg is the value to set on every interface that carries MooseFS traffic.