System overview

A MooseFS instance presents many machines as one POSIX file system. File data is split into chunks that are spread across the Chunkservers. The file system tree is kept on one Master Server. To a program on a client machine, the mounted file system behaves like a local one. This page describes the parts and the two streams that connect them. Each part has a page of its own in this chapter.

Four modules make up an instance

A module name says which program runs on a machine. Four modules make up an instance:

• The Master Server manages the instance. It keeps all metadata in memory: the directory tree with attributes and the location of every chunk. The Master Server answers every client request about the metadata and directs the Chunkservers. One Master Server manages the instance at a time.
• Chunkservers store the file data, divided into chunks that are kept as regular files on their local disks. An instance runs any number of Chunkservers. Adding a Chunkserver adds capacity.
• The Metalogger receives a copy of the metadata from the Master Server and writes that copy to its own disk. An instance runs any number of Metaloggers. A Metalogger is not a standby Master Server and serves nothing: the Metalogger holds the metadata off the Master Server's machine.
• Clients are the machines that use the instance: through a mount point (mfsmount over FUSE, or the Windows Client PRO (PRO Edition only)), through a block device (mfsbdev), or from a program (libmfsio).

Outside the modules are the monitoring interfaces: [the GUI, mfscli, and the Prometheus exporter](./components/gui-cli-exporter.md). They read and display the state of the instance. No file operation passes through them.

The editions assemble these modules differently in one place, the Master Server. The next two sections describe each assembly in full.

A PRO instance runs several Master Servers

Fig.

A MooseFS PRO instance. One Leader Master Server with Followers, and two kinds of traffic on one network: control and metadata, and file data. Control and metadata run between the clients and the Master Servers, file data between the clients and the Chunkservers. The Metaloggers only receive, so nothing is ever served from them.

MooseFS PRO runs several Master Servers at once. Every one of them is the same module: "Leader" and "Follower" PRO (PRO Edition only) name roles that a Master Server holds, not separate programs. The Leader is the one Master Server that manages the instance. Each Follower keeps its own copy of the metadata and is ready to take over the Leader role without an operator, so the instance survives the loss of the machine that the Leader runs on. Leader election and quorum covers how the role passes from one server to another.

The other modules are the same in both editions: Chunkservers store the file data, Metaloggers write one more copy of the metadata to their own disks, and clients mount the file system.

A Community Edition instance runs one Master Server

Fig.

A Community Edition instance. The same parts and the same two kinds of traffic, with exactly one Master Server and no Follower to take its role over. Only the Master Server column differs from a PRO instance: while that one machine is down, the file system is unreachable. The Metaloggers keep a copy of the metadata off that machine, so the metadata outlives the machine.

Community Edition runs exactly one Master Server. While that Master Server is unavailable, the file system is unreachable. The metadata still survives the loss of that machine, in the copies held off it. After you lose the Master Server, you install a new Master Server and start it from the metadata that a Metalogger holds. There is no role to take over and no server to take it, so recovery is the operator's work, not the instance's.

Metadata and data travel on separate streams

Every file operation splits into two kinds of traffic. Metadata operations, such as opening, creating, or renaming a file and locating its chunks, are sent between the client and the Master Server. File data goes between the client and the Chunkservers directly: the Master Server answers with the addresses of the Chunkservers that hold or will hold the chunk. The bytes never pass through the Master Server. The read path and the write path each describe one operation across both streams. All of this traffic uses one binary protocol over TCP.

Data and metadata are redundant in different places

File data is redundant in every edition: a chunk is stored as several copies or as erasure-coded parts, each on a different Chunkserver, so no single Chunkserver is a point of failure. The two sections above describe how the editions differ in metadata. A Follower PRO (PRO Edition only) keeps the instance running. A Metalogger keeps only the metadata. No single point of failure covers which component threatens access, which threatens data, and what each protection costs.