The MooseFS protocol and its ports
The modules of a MooseFS instance communicate with one another over one binary protocol on TCP. This page describes the structure of that protocol, who opens which connection, and how the Master Server decides which address it sends. The port numbers themselves, and the options that change them, are listed in network ports.
One frame carries every message
A packet is three fields: a 32-bit command type, a 32-bit payload length, and that many bytes of payload. Every integer, in the header and in the payload alike, is sent in network byte order. There is no other framing: no magic number, no checksum, and no sequence number at this level.
The name of a command encodes its direction: CL represents a client, MA for a Master Server, CS for a Chunkserver, and AN for any module. CLTOMA_FUSE_READ_CHUNK is therefore sent from a client to the Master Server, and MATOCS_REPLICATE from the Master Server to a Chunkserver.
The frame carries no protocol version. Instead, each module declares its software version when it registers. Each side of a connection keeps the version that its peer declared. An answer is built in the layout that the declared version supports. A new field therefore extends a packet without breaking older peers, so modules of different versions can share an instance during an upgrade. The declared version also decides whether a module may register. Each feature the instance uses has a minimum version, Erasure Coding and authenticated chunk I/O among them. The Master Server refuses a module that declares a lower version.
Connections are opened toward the server side
Two modules accept protocol connections from the rest of the instance. The Master Server listens on three ports: one for clients, one for Chunkservers, and one shared by Metaloggers, supervisors, and Follower Master Servers PRO (PRO Edition only). The Chunkserver listens on one port. The GUI server listens as well, over HTTP rather than this protocol. All other modules dial in. The Master Server sends everything it has for a module, orders to a Chunkserver included, back over the connection that the module opened.
A mount listens too, but only on the machine it runs on. It starts a Master Server proxy of its own, on 127.0.0.1 by default. The mfstools programs are given that address in place of the Master Server's. A tool then works through the mount's own session, so the proxy opens no connection of its own to the Master Server. A machine with several mounts runs one proxy per mount.
| Connection | Opened | What it carries |
|---|---|---|
| Client to Master Server | at mount, kept open | Metadata operations and the session. |
| Client to Chunkserver | per read or write job | Chunk data. |
mfstools to a mount's local Master Server proxy | per tool run | The tool's metadata operation, relayed over the mount's own session. |
| Chunkserver to Master Server | at startup, kept open | Registration and reports one way, orders the other way. |
| Chunkserver to Chunkserver | per transfer | The write chain, and replication. |
| Metalogger to Master Server | at startup, kept open | The changelog stream and metadata downloads. |
| Follower Master Server PRO (PRO Edition only) to Leader Master Server | on taking the Follower role, kept open | Metadata synchronization. |
| Supervisor to Master Server | per mfssupervisor run, and per Chunkserver's scheduled pass PRO (PRO Edition only) | Orders that change the state of the Master Servers, and the state they report back. A metadata save is forced here only by a run of that tool which asks for it. |
GUI server and mfscli to Master Server | per query | The state of the instance, and orders that change it: a Chunkserver into maintenance mode and out of it, a Chunkserver removed, a client session removed. |
GUI server and mfscli to Chunkserver | per query | The charts and disk list of that Chunkserver, and the order to clear the error counter of a disk. |
The Leader Master Server assigns the schedule for those Chunkserver passes PRO (PRO Edition only). Each Chunkserver gets a slot of its own. The slots are five seconds apart. A single Chunkserver therefore reaches the Master Servers once every five seconds multiplied by the number of registered Chunkservers. Across the instance one pass runs every five seconds.
A connection that carries chunk data is not always closed when its job ends. A client, and a Chunkserver forwarding a write, keep the connection in a connection cache instead. They take the connection from that cache for the next job to the same Chunkserver.
A Chunkserver that dials another Chunkserver uses no commands of its own. It forwards a write the way a client would send one and pulls a replica with the ordinary read command. Its other requests are the queries that any module may send to a Chunkserver.
The Master Server decides which address clients dial
A Chunkserver reports the address and port of its listening socket when it registers. That socket defaults to *, which means any address. A Chunkserver left at the default has no address to report, so the Master Server uses the address that the Chunkserver connected from. The Master Server puts the address it decided on into every answer that names a Chunkserver. Those answers are the chunk locations a client reads from and writes to, the copy listing of a file, and the Chunkserver list the GUI and mfscli display. Nothing ever discovers a Chunkserver on its own. The registered address is fixed at that moment. Before the Master Server stores it, the REMAP_* options in mfsmaster.cfg replace a prefix of that address. The result is the same for every asker.
When the asking machine is in a different network than the servers, the machine may not be able to reach the address that the Master Server stores. Two mechanisms rewrite that address for each asking machine before the Master Server sends it:
mfsipmap.cfgmaps addresses explicitly. The file pairs lists of client addresses with lists of old-to-new mappings. The Master Server picks the mapping list whose client addresses cover the asking machine. If the Chunkserver's address is on that list, the Master Server sends the mapped address.MULTILAN_BITSandMULTILAN_CLASSESinmfsmaster.cfgrewrite by prefix, for networks whose machines keep the same host part in each LAN. The rewrite applies when the address of the asking machine and the address of the Chunkserver each fall into one of the listed classes. Then the firstMULTILAN_BITSbits of the Chunkserver's address are replaced with those of the asking machine's address. As a result, the client gets an address of the Chunkserver in the client's own network.
The explicit map takes precedence: the Master Server checks mfsipmap.cfg first. The prefix rewrite applies only to addresses that the map does not change. The Chunkserver list served to the GUI and mfscli contains both the registered address and the rewritten address.
The same rewriting covers the Leader Master Server's own address PRO (PRO Edition only). When a client, a Metalogger, a Chunkserver, or a supervisor is told which Master Server leads, that address is rewritten for the asker first.
Rewriting changes the address that topology distance is measured against. The distance to a Chunkserver is computed from the rewritten address, which is the address that the client dials.
Replication keeps to the registered address
An order to replicate does not go through the per-asker rewriting. The Master Server tells one Chunkserver to fetch a chunk from another Chunkserver to repair redundancy or to rebalance. That order carries the registered address of the source Chunkserver, as the Master Server stored it. That address has REMAP_* applied and nothing else. A Chunkserver connected to two networks therefore serves replication on the network it registered from. Clients in a different network get an address in their own network. The traffic that a cluster generates on its own then uses different interfaces than client reads and writes, without an option that tells the two kinds of traffic apart.
Two limits apply here. The address splits the traffic. It does not split the port or the interface that a Chunkserver dials out of. Replication arrives on the same Chunkserver port that a client uses. A CSSERV_LISTEN_HOST narrowed to one address would cut off both kinds of traffic alike. BIND_HOST pins only the connection to the Master Server. The address also splits only one kind of internal traffic. A chained write is forwarded to the addresses that the writing client was given. Copies made by a write therefore follow the client's network. Copies made by replication follow the registered network.
The protocol carries no transport encryption
Traffic between modules is unencrypted, chunk data and metadata alike. Authentication between modules covers what protects an open port against a foreign connection.