Skip to main content

MooseFS PRO only.

Leader election and quorum

Several Master Servers run in a MooseFS PRO instance. Only one of them may manage the instance at a time. Through leader election, the cluster decides which Master Server that is.

What it is​

A Master Server does not hold the Leader role because it started first, because it was installed first, or because a configuration file names that server. It holds the role because enough of the cluster's Chunkservers are connected to it. A candidate Master Server reports the state ELECT. It becomes the LEADER only once enough of the cluster is reaching it. In a cluster that was never divided, enough means more than half of the Chunkservers. In a cluster divided into locations, the locations that have Chunkservers vote instead, and a majority of them is enough. One location is always the default location. Its vote decides when the locations divide exactly in half. Either way, that majority is the quorum.

A Chunkserver votes by connecting. It keeps one connection to a Master Server. That connection counts toward the candidate's majority. The denominator of the fraction is the list of Chunkservers that the cluster has registered. The Master Server keeps this list in its metadata, so the list survives restarts. A Chunkserver that is switched off or unreachable counts against the majority rather than dropping out of the count.

Voting is not tied to holding data. A Chunkserver started with no disk space configured registers like any other Chunkserver and counts in the same fraction. It stores nothing and works as a voter only.

Why it matters​

Quorum stops a split network from producing two Leaders. When a failure cuts the cluster in two, each part sees only the Chunkservers on its own side. In a cluster that is not divided into locations, at most one part holds more than half of the Chunkservers, so at most one part can have a Leader. When the Chunkservers divide exactly in half, neither part has one. In a cluster divided into locations, each location votes for the part that holds more than half of that location's Chunkservers. At most one part can hold a majority of the locations. When the locations divide exactly in half, the vote of the default location decides. The part that holds the default location can then have a Leader. The other part cannot. In both cases, the part that is short of the majority cannot elect a Leader of its own. The glossary calls this split-brain protection. Without quorum, both parts would accept changes to the same files. Neither set of metadata could then be reconciled with the other set afterwards.

The rule applies to the Master Server that already holds the role, not only to the candidates. A Leader that loses the majority steps down and goes back to being a Follower. The rule has a cost when quorum is lost but the data itself is intact. An instance with no Leader carries out nothing that goes through a Master Server. The exception is the reads answered by a DEPUTY, a Master Server that serves reads without holding the role of Leader. A client cannot even register a session. A Master Server that is neither the Leader nor a DEPUTY opens no session. Instead, the Master Server answers with the address of the Leader that it has recorded. When there is no Leader, that is the address of the last Leader that the Master Server recorded, or no address at all. The client then waits and tries again. Until a Master Server reaches the majority again, no metadata change is accepted and no chunk is replicated or deleted.

How MooseFS implements it​

The role is a state that the Master Server reports. The Master Servers do not pick the candidate among themselves. The Chunkservers do. Each Chunkserver runs the choice in turn. The Chunkserver whose turn it is sets one Follower to ELECT when it finds neither a Leader nor a candidate. A Master Server sets the candidate itself only when no Chunkserver has done so for five seconds, which is the case in an instance with no Chunkservers registered. The supervisor forces a state, not a majority describes that choice. That candidate promotes itself to LEADER as soon as the count is in its favor. Two more states cover the cases where the count is not in its favor. An ELECT that cannot gather the majority can be allowed to move to DEPUTY. A DEPUTY answers queries but accepts no metadata change and orders no work on chunks. A DEPUTY returns to being a Follower once the majority is available again. A synchronized FOLLOWER can be set to reclaim the Leader role as soon as that Follower is able to. Such a Follower reports USURPER while it waits. A Master Server ends in DUMMY while it shuts down. The FOLLOWER and USURPER states have a suffix that says how far their metadata is from the Leader's. However, the tools print that suffix for a Follower alone.

Only a Master Server that holds metadata takes part in the election. A Master Server that has none can be a Follower and nothing else, whatever the Chunkservers are connected to.

A move out of LEADER or DEPUTY disconnects the Chunkservers, the clients, and the Metaloggers before the new state takes effect. This way, nothing keeps talking to a Master Server in a role that it no longer has. A move into either role disconnects nothing, because either role is reached only from ELECT. The promotion takes effect at once. Moves out of FOLLOWER and USURPER, whether to FOLLOWER, ELECT, or USURPER, take effect at once as well, because neither of those two roles serves anything that would have to be disconnected. An ordered move from ELECT back to FOLLOWER goes through the same disconnection as a move out of LEADER or DEPUTY.

The count itself is made per location, which votes as a unit. Leader election, quorum, and the role of DNS covers how a location's vote is worked out and what decides a count that divides exactly in half. Quorum across sites covers what the rule means in a cluster of two and of three locations.