Skip to main content

No single point of failure

MooseFS is designed to run without a single point of failure. The configuration that runs this way is MooseFS PRO with more than one Master Server.

What it is​

A single point of failure is one component whose loss stops the service. A system is free of such components when every part that it depends on runs in more than one instance, so that losing one instance leaves the rest working.

Two different things can stop. MooseFS handles each of them separately. The first is access to the data right now: whether the file system is reachable while a component is down. Continuous access to data despite component failures is High Availability (HA). The second is the data itself: whether the failure destroys something that cannot be read back afterwards. A component can be a single point of failure for the first question and not for the second.

Why it matters​

The two questions cost different things and are answered by different parts of a design. An instance can be unreachable for hours and lose nothing. It can also serve every read at full speed while the only copy of the metadata is on one machine. Knowing which of the two questions a component threatens tells you how many machines a design needs and where the copies have to be kept.

How MooseFS implements it​

File data answers both questions at once. A chunk is stored either as several copies or as erasure-coded parts, each on a Chunkserver of its own. A lost disk or a lost Chunkserver therefore costs neither access nor data. MooseFS rebuilds the missing redundancy on its own. The number of failures that a file survives at once is not a fixed property of the system. That number is the Redundancy Level that the Storage Class of the file requires. Data kept in a single copy has no redundancy at all. You get the protection that you configured. MooseFS adds no protection underneath it.

Chunkservers, Metaloggers, and clients are single points of failure for neither question. An instance runs as many Chunkservers as it needs and any number of Metaloggers. A client machine serves only itself. Its failure leaves the file system and every other client working. The risk covers only what that machine had not yet sent.

For the Master Server, the two questions have different answers. One Master Server manages the instance at a time. While no Master Server is managing the instance, the file system is unreachable, unless a Master Server has been set to keep answering reads. Community Edition runs exactly one Master Server. That single Master Server is a single point of failure for access. MooseFS PRO runs several. A Follower PRO (PRO Edition only) holds its own copy of the metadata and takes the Leader role over without an operator. The property comes from running more than one Master Server rather than from the edition. MooseFS PRO runs with a single Master Server as well. That Master Server is then a single point of failure for access, exactly as in Community Edition. MooseFS PRO reports the configuration as an error. The edition comparison states HA as the difference between the two editions.

Losing the Master Server does not have to mean losing data. That holds in Community Edition as well. The Master Server keeps metadata in its memory and writes the metadata to its local disk as a periodically refreshed image plus a changelog of every operation since. Only a copy held off that machine survives the loss. Every connected Metalogger receives the image and the changelog. In MooseFS PRO the Followers hold the metadata too. If you run neither a Metalogger nor a Follower, then the metadata exists in one place only. MooseFS reports a single Master Server with no Metalogger as an error. A Metalogger answers the second question and not the first. A Metalogger holds the metadata and does not serve the file system.