MooseFS PRO only.
Leader election, quorum, and the role of DNS
Every role that a Master Server can hold is a state in one state machine. This page describes that state machine: what changes the state of a server, who orders the change, how long the change takes, and what the change does to the connections. Leader election and quorum defines the roles and the majority rule that those transitions depend on.
A Master Server starts as a Follower
mfsmaster enters FOLLOWER every time it starts. It enters that state desynchronized. The label on the metadata connection starts as DESYNC. Starting up does not clear the label. The Master Server writes its state: line to the log when its state changes, and when a Follower is sent to a different address for the Leader. A server that starts in FOLLOWER and reaches the Leader at once neither changes state nor is sent to a different address. No line in the log therefore names the label that the server holds. The missing line is deliberate. A Master Server that is desynchronized by its own start, rather than by the work of the instance, is not reported as desynchronized in the log. The section on the suffix of the metadata connection, further down this page, covers when that label is cleared.
LEADER and DEPUTY are the only states that no order can set. A Master Server reaches them on its own, and only from ELECT. A Master Server also moves on its own from either of those states back to FOLLOWER, and from FOLLOWER to USURPER. The sections below describe those moves. A supervisor can order ELECT, FOLLOWER, or USURPER. An order that names LEADER or DEPUTY is discarded. An order that names the state that the server already holds is also discarded. So is an order that repeats a transition that is already running. An order that would move a FOLLOWER to ELECT is discarded as well when the Follower is connected to a Leader and its metadata is not desynchronized. The process enters DUMMY when it is asked to exit. From the moment that the exit is ordered, the process accepts no order, both during the transition into DUMMY and in the state itself. The process finishes exiting only after it reaches DUMMY.
A supervisor can order only ELECT, FOLLOWER, or USURPER. An order naming LEADER or DEPUTY is discarded.
Only an ELECT reaches LEADER or DEPUTY, and only on its own.
An ELECT settles on the next main-loop pass
The Master Server checks the state machine on every pass of its main loop. A change of role therefore happens within a fraction of a second after the condition that allows it.
An ELECT that holds metadata and has the majority becomes the LEADER on that pass. An ELECT that does not have the majority keeps its state unless DEPUTY_ELECTION_TIME is above zero. In that case, the server becomes a DEPUTY after that many seconds have passed since it entered ELECT. The default 0 turns off this transition, so no ELECT ever becomes a DEPUTY. Any other value is read in seconds and kept between 30 and 86400. A smaller one is raised and a larger one is lowered. Both cases are logged.
Two transitions go in the opposite direction on the same check. A DEPUTY returns to FOLLOWER on the pass on which it regains the majority. A LEADER changes to FOLLOWER on the pass on which it loses the majority. Both label their metadata SYNCED when they leave, because each of them has been synchronized with itself the whole time.
Leaving LEADER or DEPUTY drops every connection first
A move out of LEADER or DEPUTY does not take effect at once. The server first waits for its pending work to drain. The pending work is the jobs queued for clients, for Chunkservers, for Metaloggers, for the connection to the Leader, and for the chunk loop. The server stops waiting after five seconds whether or not the queues are empty. It then disconnects the Chunkservers. On the next pass, it disconnects the clients. On the pass after that, it disconnects the Metaloggers and closes its own connection to the Leader. The new state takes effect only after this, and the age of the role starts counting again.
During this transition, the server holds neither role. It answers no to being the Leader, a Follower, an ELECT, or a DEPUTY. It is reported as a transition rather than a state. Moves out of FOLLOWER and USURPER skip all of this and take effect immediately, whether they go to FOLLOWER, ELECT, or USURPER. They skip it because neither of the two roles serves anything that would have to be disconnected. A move into LEADER or DEPUTY skips all of this too. The promotion of an ELECT takes effect on the pass that decides the promotion. No connection is closed. A Chunkserver still waiting to register then finishes registering with the new Leader. An ordered move from ELECT back to FOLLOWER runs the full transition.
A synchronized Follower can take the role back
With KEEP_LEADERSHIP on, a FOLLOWER whose metadata is labeled SYNCED moves itself to USURPER on the next pass. A supervisor then gives it the role.
Set the option on at most one Master Server of an instance. Two servers that both have the option both move themselves to USURPER as soon as they are synchronized. The behavior of the instance after that is not defined.
One flag can prevent this move. Any order that moves a server into FOLLOWER from a different role sets that flag. A deliberate handover is such an order, because it moves the current Leader into FOLLOWER. While the flag is set, the server stays a FOLLOWER, whatever the value of KEEP_LEADERSHIP. The flag is cleared while the server holds LEADER. A server that wins the role again, by handover or by election, therefore tries to take the role back after the next failure. Restarting the process clears the flag too.
The majority is counted location by location
Every location that is not switched off and has a Chunkserver registered in it casts one vote. A majority of those votes decides the election. When the votes divide exactly in half, the vote of the default location decides. Quorum across sites covers what the rule means at two and at three sites. The count itself is one loop over the locations.
For each location that is not switched off, the Master Server reads the counters that it keeps for that location. The counters give the number of Chunkservers registered there, the number of them that are disconnected, and the number that have been temporarily removed. The temporarily removed Chunkservers are subtracted from both of the first two counts. The location votes if the disconnected remainder is below half of the registered remainder, rounded up. A location with no Chunkserver left in the registered remainder casts no vote and is not counted. When no location has one, the count is in favor. The vote of the default location is kept separately as the casting vote. It is used only when the votes divide exactly in half. A default location with no Chunkserver left has no vote to cast, so an even split then goes against the candidate. Up to 5.1.0, a location with no Chunkserver left voted in favor.
A Chunkserver that is temporarily removed is left out of both figures, but only until the next Leader starts. A Master Server that enters LEADER permanently deletes every temporarily removed Chunkserver from the registered list. Maintenance mode is a different flag and changes neither figure.
DNS alone carries the list of Master Servers
Each part of the installation finds out which Master Servers exist by resolving one name into every address that the name has. For a Master Server, this name is MASTER_HOST. Elsewhere, it is the master host option of the component. The value may also name several hosts separated by commas or semicolons. In that case, the answers are merged. The merged list holds up to 256 address and port pairs. The list is sorted, and duplicates are removed with a warning in the log.
That list is the roster of the cluster. A supervisor connects to every address on that list and to no other address. A Master Server missing from the answer is therefore never asked for its state and never given a state. A Follower resolves the name again on every attempt to reach the Leader. While it has an address to try, it retries every MASTER_RECONNECTION_DELAY seconds.
An incomplete or stale answer therefore produces a cluster smaller than the real cluster. Two nodes that resolve the name differently disagree about which servers are members. The Leader checks for this disagreement. Each time the Leader runs its configuration test, it asks every Chunkserver, client, Metalogger, and Follower for the list that the node resolved. The Leader then compares each answer with its own list, entry by entry. A node whose list differs is told to resolve the name again and to answer once more. If the new answer still differs, the Leader logs the node by name. The first mismatch in a round also makes the Leader resolve the name again itself. DNS setup covers how to put the records in place.
The supervisor forces a state, not a majority
The same supervisor logic runs in the Chunkservers, in the Master Servers, and in the tool. Every Chunkserver runs this logic on a rotating schedule that the Leader assigns. The period is five seconds times the number of registered Chunkservers, with one slot per server. The slot of a Chunkserver that is not connected passes unused. Therefore, five seconds is the interval between two passes only while every registered Chunkserver is connected. When servers are missing, the gaps are multiples of that interval. A Master Server runs this logic itself only when it is a Follower, knows no Leader address, and has not been reached by any supervisor for five seconds. That copy picks a candidate only when every Master Server is a Follower. Otherwise, it only passes on the address of the current Leader. mfssupervisor runs the full algorithm when you run it by hand. mfssupervisor(8) lists its switches.
A pass reads the state, the metadata version, and the location of every Master Server on the roster. It then orders whatever those readings require. When there is more than one LEADER, or more than one DEPUTY, the pass keeps the server with the highest metadata version and moves the rest to FOLLOWER. When a pass finds a LEADER together with an ELECT or a DEPUTY, it moves that ELECT or DEPUTY to FOLLOWER. When there is more than one ELECT, the pass moves everything back to FOLLOWER. In a cluster with none of the three states, the pass sets one candidate to ELECT. The pass chooses the candidate by these criteria, in order: synchronized before not synchronized, then the highest metadata version, then a USURPER, then a server in the location of the last Leader, then the highest address. A server with no metadata is never chosen. Synchronization is also a condition, not only an order of preference. A candidate that is not synchronized is set to ELECT only when every Master Server on the roster answered and none of them is synchronized, or when the tool is forced. Otherwise, the pass chooses nobody and reports this. When the pass runs inside a Chunkserver, it also refuses a candidate whose metadata version is lower than the last version that the Leader reported to that Chunkserver. Every Follower is also told the address of the current Leader, so that it has an address to connect to.
No order supplies the majority. The supervisor sets ELECT. Only the Chunkserver count decides whether that ELECT becomes the LEADER. An ELECT that never reaches the majority stays an ELECT, or becomes a DEPUTY if DEPUTY_ELECTION_TIME allows. Giving the role to a chosen server is also not a single order. That server is set to USURPER. The current Leader steps down because a USURPER exists. The next pass then finds the USURPER and elects it. The handover request is refused outright when the cluster is not stable. This is the case when there is no single LEADER or DEPUTY, or when an ELECT or a USURPER other than the chosen server is present.
Without metadata, a Master Server stays a Follower
The rule is enforced at each point where it could be broken. An ELECT with no metadata is never promoted, whatever the Chunkservers do. When an order that names any other state arrives at a Master Server whose metadata version is zero, the order is rewritten to FOLLOWER. A supervisor also never picks such a server as a candidate.
For a rebuilt instance, this rule decides the order of work. A Master Server installed beside a running cluster starts empty, registers as a Follower, and is filled from the Leader. Until that copy is in place, the new server adds nothing to availability, because it cannot be a candidate in an election. An instance in which every Master Server is empty elects nobody, because there is no candidate to choose. The metadata must be put back on one of the Master Servers before the cluster can start.
The suffix labels the metadata connection
A Master Server reports INIT right after it connects to the Leader, before the difference between the two copies is known. DESYNC means that the server has no usable copy of the Leader's metadata. DELAYED and SYNCED mean that the server has a usable copy, and they show how current that copy is.
Two places report the label differently. The Master Server's own state report includes the label for FOLLOWER and for USURPER. mfscli and the GUI append the label to a FOLLOWER only, and only while the copy is not SYNCED. There, a synchronized Follower appears as a bare FOLLOWER, and a USURPER has no suffix at all.
A Master Server starts with DESYNC. It gets this label again whenever its copy can no longer be trusted. This happens at startup, and when the Leader answers that it no longer holds the changes that the server is missing. It also happens when the server refuses a change that it was sent, and when the Leader orders the server to resynchronize after the Leader compares checksums.
Applying more changes never clears DESYNC. A server in DESYNC receives a whole dump instead. A server that already holds metadata of its own discards that metadata first and reconnects. After the copy is in place, the connection reports INIT. The first change that arrives then sets the label to SYNCED or DELAYED, depending on how far behind the server is.
One path leaves DESYNC without any of that. A Master Server that is told to step down from LEADER, or a DEPUTY that regains the majority, is labeled SYNCED when it leaves. Both were serving their own metadata a moment earlier, so they cannot be behind any other copy. For the same reason, the first Master Server of a new instance can be elected while it is still labeled DESYNC, because it has no Leader to be synchronized with. The label describes something real only after a second Master Server exists.
The label also affects the role. A FOLLOWER that reports DESYNC does not count as having a working Leader. An order that moves it to ELECT is therefore accepted rather than refused. That Follower also loses to any synchronized Follower when a candidate is picked. For the metadata versions and delays that produce the labels, see Metadata ID, version, delay, and checksum.
The settings behind the election and the roster
| Setting PRO (PRO Edition only) | Default | What it decides here |
|---|---|---|
MASTER_HOST | mfsmaster | The name that every module resolves into the roster of Master Servers. Several names separated by commas or semicolons are merged into one list. |
DEPUTY_ELECTION_TIME | 0 | Seconds that an ELECT without the majority waits before it settles into DEPUTY. 0 means that it never does. |
KEEP_LEADERSHIP | 0 | Whether a synchronized FOLLOWER moves itself to USURPER to take the role back. 0 leaves the role where it is. |
MASTER_RECONNECTION_DELAY | 5 | Seconds between a Follower's attempts to reach the Leader. |
Every one of these settings is set on the Master Server, in mfsmaster.cfg.