Skip to main content

The Master Server

The Master Server is the module that manages a MooseFS instance. Master Server, Leader, and Follower covers what that role is and why the cluster has it. This page describes the module's structure: the structure of the process, the connections it serves, the roles it can hold, and the files it keeps on disk.

One process keeps the whole metadata in RAM​

The program is mfsmaster. It runs as one process and holds the entire metadata of the instance in its memory: the directory tree, the file attributes, and the registry of every chunk. The Master Server answers every metadata request from that memory. What metadata holds and what it costs in RAM lists the record kinds of the metadata and their sizes.

In RAM means in RAM: the machine has to hold the whole metadata in physical memory. Swap is not a way to make up the difference. A Master Server that starts swapping answers metadata requests at the speed of a disk instead of the speed of memory. The Master Server does so on the single thread that serves every client, so one page fault delays everything queued after it. To keep that from happening, size the machine for the metadata that it will hold, not for the metadata that it holds today. Master Server memory calculates the memory required. LOCK_MEMORY in mfsmaster.cfg exists for the same reason. When the option is set to 1, the Master Server asks the operating system to lock its memory with mlockall(). The call needs an unlimited memory-lock limit. If the limit is lower, or if the call fails, the Master Server logs a warning and continues to run without the lock. Only the log, not the setting, shows whether the lock took effect.

The process does its work in a single thread: an event loop over its network connections, with periodic tasks scheduled on timers inside the same loop. Those tasks include the scans of all files and all chunks that apply the policies. The chunk loop and the file loop describes what they do. Two more processes belong to the module: a companion data writer that mfsmaster forks at startup, shown as mfsmaster (data writer) in the process list, and a short-lived child forked for each save of the metadata dump, shown as mfsmaster (metadata saver). A PRO PRO (PRO Edition only) Master Server forks two further short-lived children. One child streams the whole metadata to another Master Server that asks for it. The other child runs a supervisor pass when a Follower has no Leader. Neither child is renamed in the process list.

Each kind of peer has its own port​

The Master Server listens on three ports, one per kind of peer, each served by its own protocol server inside the module:

  • Clients connect to the client port for everything that a mount does with metadata and its session. mfscli and the GUI server read the cluster state on the same port.
  • Chunkservers connect to the Chunkserver port to register, report the chunks that they hold, and receive commands: replicate, delete, duplicate.
  • The metadata port serves every peer that asks for the metadata itself and for the state of the Master Servers: Metaloggers, Follower Master Servers PRO (PRO Edition only), and supervisors.

The port numbers, the options that change them, and who connects where are listed in network ports.

PRO roles belong to one program​

MooseFS PRO PRO (PRO Edition only) runs several Master Servers in one instance. Every one of them is the same mfsmaster. Leader, Follower, Elect, Deputy, and Usurper are roles that a Master Server holds and moves between. They are not separate programs or separate installations. Leader election and quorum covers the Leader and Follower roles and the majority rule that assigns them. Leader election describes the state machine that moves a server through all five. The Community Edition runs one Master Server per instance. That Master Server holds no role: the role state machine is not part of the CE build. The GUI and mfscli still fill the state column for that Master Server with MASTER, which is the label for a Master Server that reports no state at all.

The disk carries dumps and changelogs​

The working copy of the metadata is the copy in RAM. The Master Server writes only its recovery record and its housekeeping into its data directory, the directory that DATA_PATH in mfsmaster.cfg points at (/var/lib/mfs in the packages). The directory holds the metadata dumps with the checksums written beside them, the changelogs, the chart history, and the lock files of the process and of its background writer. Files in a data directory lists every file that can appear in that directory and how long each one stays.

Changelogs, dumps, and the recovery point covers when dumps are written, how changelogs rotate, and what a startup reads back.

File data never passes through the Master Server​

The Master Server stores no file content and moves none. A client asks the Master Server which Chunkservers hold a chunk, then exchanges the data with a Chunkserver directly. The read path describes that exchange. Replication works the same way: the Master Server orders it. The copy is sent between two Chunkservers.