Skip to main content

Physical media and InfiniBand

MooseFS places no requirements on the network below IP. This page describes what the modules open on the network, and what follows from that for the hardware a cluster runs on.

Every connection between modules is a TCP connection​

The Master Server, Chunkservers, Metaloggers, clients, and the GUI server communicate with each other over ordinary TCP connections on IPv4. The daemons written in C open every listening socket and every outgoing connection through one shared socket layer. The Python side of the GUI and mfscli connects through the Python standard library instead, and speaks the same protocol over it. None of the modules reaches below the IP layer. Network ports lists which port carries what kind of traffic, and how much of it. The protocol the modules speak describes what is sent inside a connection.

Any network that carries IP carries MooseFS​

Because the transport is TCP/IP, MooseFS runs on whatever the operating system presents as an IP interface. Copper Ethernet, fiber Ethernet, a bonded set of links, and a VLAN all look the same to a module. Each of them is only an address to listen on and an address to connect to. An InfiniBand fabric carries MooseFS traffic the same way through IPoIB (IP over InfiniBand), which presents the fabric to the operating system as an IP interface. Everything below TCP, such as link speed, bonding modes, or the interface MTU, is configured in the operating system and is invisible to MooseFS.

A connection can be pinned to an address​

On a machine with addresses on more than one network, both ends of a connection can be pinned. Each listening port has a matching *_LISTEN_HOST option. The option limits the addresses that the daemon accepts connections on. The options are listed with their ports in network ports. One listening socket is outside that scheme: the local Master Server proxy a mount runs takes its address from the mount's own mfsproxy option. On the outgoing side, BIND_HOST in mfschunkserver.cfg and mfsmetalogger.cfg sets the local address from which the daemon connects to the Master Server. BIND_HOST PRO (PRO Edition only) also exists in mfsmaster.cfg. There it pins the connection that a Follower Master Server opens to the Leader. A mount pins its outgoing connections with mfsbind. That option covers every connection the mount opens, to a Chunkserver as well as to the Master Server. BIND_HOST and the *_LISTEN_HOST options default to *, which means any address. mfsbind has no default value, and stays unset until you give it one. In both cases the operating system chooses the address.

Pinning decides where a daemon listens and where it dials from. Pinning does not sort traffic by kind. The address that each asker is told to use sorts the traffic. A cluster whose machines are in two networks uses that address to put replication on one network and client I/O on the other network. The address a client dials describes that split.