Rolling upgrade

A rolling upgrade upgrades a cluster to a new version component by component, while the cluster keeps serving data. This page covers the part of a rolling upgrade that needs several Master Servers. In that part, you upgrade the machines of the group one at a time, and the Leader role moves off the last machine. The group is then never without a Leader for longer than one switch. The procedure assumes a healthy cluster and the preparation described in pre-upgrade checks. Every other step works the same way in both editions. The Community Edition cannot do the Master Server step as a rolling step, because the Community Edition runs one Master Server. Stopping that server stops the instance, and this is where the editions part.

The group is complete and in step before starting

Read the state of every Master Server from the CLI, or from the Metadata Servers (Masters) subtab of the Info tab in the GUI. mfscli prints the group as a table:

mfscli -SIM

These conditions have to be true before you stop the first machine:

• More than one Master Server answers. The Leader role needs a machine to move to.
• Exactly one of them reports LEADER. None reports ELECT or USURPER. A handover is refused while the group is in either of those states.
• Every other server reports FOLLOWER with no suffix after the state. A suffix means that the server's copy of the metadata is not synchronized with the Leader (the delay says how far behind the Leader the Follower's last applied change is). A handover to such a server is refused as well.
• All servers hold the same metadata. When mfssupervisor finds different metadata IDs in the group, it stops before it changes anything. Find the cause of that difference before an upgrade starts, not during the upgrade.
• The version that you are installing leaves no Master Server older than the Leader. Version compatibility covers which combinations work.

Stopping one of those machines does not put the current Leader at risk. The majority that keeps a Leader in the role is counted among the Chunkservers connected to the Leader, not among the Master Servers. A group with one machine missing therefore keeps its Leader and keeps serving (leader election and quorum). A group with one machine missing loses redundancy for as long as that machine is offline.

One pass, and the role moves once

Take the machines in this order:

1. Every Follower, one machine at a time. Upgrading a single Master Server machine is the ordinary step. Its commands and its check are listed in order of upgrade. The Followers are upgraded first, because a Follower may run a newer version than the Leader, but never an older one.
2. The Leader last. Stopping the Leader for its upgrade moves the role. This is the only moment of the procedure that affects the clients. The machine rejoins as a Follower.

One machine is out of service at a time. Each machine is synchronized again before the next machine stops.

The Leader moves by its own restart

Upgrade the Leader the way you upgraded the Followers: stop the service, install the new packages, and start the service again. Stopping the Leader moves the role. The group elects a new Leader from the upgraded Followers. The machine that you restarted rejoins as a Follower.

Do not move the role with mfssupervisor -l while the upgrade is running. Use that command only in a group in which every Master Server already runs the new version.

While the group has no Leader, the CLI suggests the opposite. It prints Leader master server not found! Maybe you should run 'mfssupervisor'. Once a candidate appears, the CLI prints Leader master server not found, but there is an elect, so make sure that all chunkservers are running. Both lines are expected during a switch. They are not instructions to act on.

While the election runs, the cluster has no Leader. The cluster reports this as information: "No Leader Master server in the cluster, election in process." Confirm the outcome before you continue:

mfscli -SIM

One server reports LEADER at the new version. Every other server reports FOLLOWER with no suffix. If an election has not finished within a minute, investigate it instead of repeating it.

The role stays on the server that the election chose. By default, no Master Server promotes itself. After the upgrade, the group therefore has a different Leader than before the upgrade. To move the role back to the original machine, use mfssupervisor -l. The command is allowed once every Master Server runs the new version, and it costs another switch.

One option changes this behavior. A Master Server with KEEP_LEADERSHIP set in mfsmaster.cfg takes the role back on its own once its copy is synchronized. The restart that upgrades the machine triggers this. The old Leader then takes the role again. The cluster goes through a second switch. The second switch affects the mounts in the same way as the first switch.

A mount stalls, then finds the new Leader

Upgrading a Follower does not affect the clients. A Follower opens no client session. When a mount reaches a Follower, the Follower answers with the Leader's address instead of opening a session. Therefore, no mount is connected to the machine that you are restarting.

The change of Leader affects the clients. A Master Server that leaves the role drops its Chunkservers, then its clients, then its Metaloggers before the new role takes effect. At that moment, every mount, Chunkserver, and Metalogger loses its connection. Each of them registers again, without an address given by hand. A Master Server that is not the Leader answers a registration with the Leader's address.

The next steps of the mount need no action from you or from the client machine. The mount registers again under the same session ID. The session record on the Master Server's side is kept through the gap, with its open files and its rights. The record is part of the metadata that the new Leader already holds. You do not remount anything or restart any application.

On the client machine, the log shows the switch. First, the log shows master: connection lost (header). The mount then tries the Master Servers in the group in turn and prints what each attempt returned. A machine that is down produces can't connect to master ("<address>":"9421"). A machine that answers but has no Leader to name produces mfsmaster <address> - doesn't know his leader, waiting a moment and retrying using different IP. A Master Server that the election has already picked produces the same line while it waits for the role. The log therefore does not separate those two states. mfsmaster found leader: <address> reports one answer, not the end of the search. The address in that line can belong to a machine that has already disconnected, and more attempts follow that line. A registration that arrives during the switch is refused, with master: register error (read header: EAGAIN (Resource temporarily unavailable)) or master: register status: Operation not permitted. None of these lines is a fault. The switch ends with the line registered to master using previous session.

During the gap, service pauses. Operations in progress during the gap stall rather than fail. Each operation is woken and sent again. An application receives an I/O error only if the mount's budget of attempts runs out first. By default, that budget is 30 attempts and around two minutes of waiting. The switch stays well within that budget, because the role change waits no more than five seconds for its pending work before it drops the connections. The switch therefore adds latency. Work in progress pauses for the length of the switch and continues afterwards. An application with its own timeout shorter than the switch reaches that timeout.

The group is whole before the next machine

Between machines, check the group in the same way as at the start:

mfscli -SIM

Exactly one server reports LEADER. Every other server reports FOLLOWER with no suffix and the version it now runs. A machine that is shutting down reports DUMMY and then UNREACHABLE. These states show that the stop is in progress. A server that still reports FOLLOWER (DESYNC) is not synchronized with the Leader yet. A machine that restarts loads the image from its own disk. The machine then asks the Leader for the changes that the machine missed. A whole copy of the metadata is sent only to a Master Server that holds none. Until that copy arrives, the cluster keeps reporting "The Follower's metadata is not synchronized with the Leader Master server's metadata." The message is expected right after a restart or an election. It clears within a minute or so.

Wait for the message to clear before you stop the next machine. The election chooses a Follower that is synchronized. The election chooses an unsynchronized Follower only when every Master Server in the group answers and none of them is synchronized. One machine is down during this procedure. A group whose remaining machines are all unsynchronized therefore elects no Leader and stays without a Leader. mfssupervisor names that state: None of the working followers have synchronized meta data. The group cannot leave that state on its own, because an unsynchronized Follower has no Leader to synchronize from. Starting the machine that you stopped lets the election run. Every unsynchronized machine reduces the number of machines that the election can choose from by one.

What this procedure does not cover

Chunkservers and clients are not part of this procedure. They are upgraded in the ordinary way. Order of upgrade covers their place in the sequence and the rules that decide that place.

This page also leaves out the steps before and after the procedure. The copy of the metadata to take beforehand is described in pre-upgrade checks, together with the check that all Master Servers have the same configuration. Returning to the previous version afterwards is a separate operation with its own limits, described in rollback.

The roles themselves, the majority that assigns them, and the state machine of those roles are described once, in leader election and quorum and in leader election, quorum, and the role of DNS.