Skip to main content

InfiniBand

An InfiniBand fabric can carry a MooseFS cluster. The fabric carries the cluster as an IP network. MooseFS speaks TCP/IP and nothing below it, so MooseFS reaches the fabric through IPoIB (IP over InfiniBand). IPoIB presents the fabric to the operating system as an ordinary IP interface. The page physical media and InfiniBand explains why any network that carries IP carries MooseFS.

The purchase buys IPoIB's throughput, not RDMA​

Every connection between MooseFS modules is an ordinary TCP connection. No module reaches below the IP layer. An InfiniBand fabric therefore does not give MooseFS RDMA, because the fabric's native transport runs below the IP layer. The cluster gets the bandwidth and latency that the fabric's IPoIB interface delivers to TCP. Plan with that figure rather than with the fabric's native rating. Evaluate the figure the way bandwidth and latency describes for any network.

InfiniBand competes for the data path​

Consider InfiniBand where you would consider any fast network: on the path that moves chunk data. A cluster that gives that traffic a network of its own is described in a separated backbone network. To MooseFS, an IPoIB fabric is a candidate for that network in the same way as an Ethernet network. MooseFS needs the same from both: an address to listen on and an address to connect to.