Order of upgrade

This page lists the order in which the components of a running instance are upgraded to a new version. It also lists the reading that completes each step. It covers every upgrade of a cluster that is already serving data.

Work through the pre-upgrade checks before you install the first package. These steps assume a healthy chunk matrix, a metadata image written to disk, and a copy of that image that you can return to.

Compatibility runs one way, so the order follows

Version compatibility is a chain, ordered from the component that may run the newest version to the component that has to run the oldest. The order is: the GUI and the CLI, then the Metalogger, then the Follower Master Servers PRO (PRO Edition only), then the Leader Master Server, then the Chunkservers, then the clients. A component may run the same version as the component above it, or an older version. A component that runs a newer version than the component above it is not refused. It registers. Its connection then drops the first time the exchange reaches a packet that the older side does not recognize.

When you upgrade from the top of the chain downward, every state that the cluster passes through follows that rule. Every component that is not upgraded yet runs an older version than every component that is, and the chain allows that direction. Version compatibility covers what each pair does when the versions differ, and which features stay off until every component supports them.

Each step closes on a reading

Every step installs the packages of one role, which are listed in Install the server components. The step then stops and starts the service of that role, as starting and stopping the cluster describes. On Debian and Ubuntu, the package manager does more than the step requires. Installing one MooseFS package upgrades every MooseFS package on that machine. Since 5.2.0, a package leaves a running service alone: the process keeps running its old version until the step restarts the service. On a machine that runs several roles, all of those roles get the new binaries at once, whichever package you name, but each role changes version only when its service is restarted. Restart those services in the order below. Up to 5.1.0, each package also restarted the service that it replaced, so the roles of one machine changed version at once, and the order below applied only to an installation that gives each role a machine of its own. Installation leaves an edited configuration file unchanged and puts the new defaults next to it. After each package change, compare the shipped .cfg.sample with your running .cfg and merge new options by hand.

Take the reading that completes a step before you start the next step. The commands below are the CLI commands. The same numbers appear on the matching tabs of the GUI.

1. Upgrade the tools that only read the cluster: moosefs-pro-gui, moosefs-pro-cli, and moosefs-pro-netdump where it is installed. These tools decode the packets that the servers exchange. A tool older than the servers may fail to decode some of those packets. This appears as wrong values on screen, not as a refusal.

   mfscli -v

   The command prints the tool's own version. The GUI shows its own version in its footer. The version stays marked there until the servers are upgraded too.

2. Upgrade every Metalogger the instance runs. The Metalogger records the metadata stream that a Master Server sends to it. The Metalogger therefore has to support everything that the Master Servers send after they are upgraded. The protocol does not enforce this. Neither side compares versions at registration, so a Metalogger that was not upgraded refuses nothing and logs nothing. The only signal is an error-level alert in the GUI. The CLI has no alert list.

   mfscli -SML

   The Metalogger is listed again, with the new version in its version column.

3. Upgrade the Follower Master Servers, one at a time. Only a PRO instance PRO (PRO Edition only) has them. Of all the version mismatches that an upgrade can produce, only a Follower older than the Leader ends a process. A Follower that finds the Leader at a higher version logs a request to be upgraded and exits. When an older Follower asks the Leader for a full metadata image, the Leader refuses to send it. During this step, the Followers are newer than the Leader. This is the intended state.

   mfscli -SIM

   The upgraded server reports the new version and the state FOLLOWER with no suffix. FOLLOWER (DESYNC), (INIT), or (DELAYED) means that the server's copy of the metadata is not synchronized yet (what the suffix labels). Wait for the suffix to clear before you stop the next machine.

4. Upgrade the Master Server last of the metadata servers. In PRO, that Master Server is the Leader. Community Edition runs a single Master Server, which reports the state MASTER. Stop the service, upgrade moosefs-pro-master, and start the service again. A Master Server always starts as a Follower. In PRO, the machine therefore rejoins as a Follower, not in the role that it had before it stopped.

   mfscli -SIM

   One server shows LEADER at the new version, or MASTER in Community Edition. The remaining servers show FOLLOWER without a suffix.

5. Upgrade the Chunkservers, one at a time. A Chunkserver newer than the Leader is reported as an error. You cannot undo that error, because a Chunkserver is not downgraded. Stop the service, upgrade moosefs-pro-chunkserver, and start the service again. Then wait until the Chunkserver finishes registering its chunks.

   mfscli -SCS
   mfscli -SIC

   The first command lists the Chunkserver at the new version, with maintenance set back to off. The second command shows the chunk matrix. Its last line has to show zero under missing, endangered, and undergoal before you stop the next Chunkserver. Stopping a second machine while chunks still lack copies leaves those chunks with even fewer copies.

6. Upgrade the clients and remount. A mount newer than the Leader is reported as an error, so clients are upgraded last. On each machine, stop the processes that use the mount point, unmount it, upgrade moosefs-pro-client, and mount again. Clients never communicate with each other, so you can upgrade the machines in parallel.

   mfscli -SMS

   Every session reports the new version.

A correct run raises its own alerts

A run in this order produces these readings:

Reading	Severity	Expected between
The GUI or CLI is newer than the servers	Info	the tools and the last client
A Follower is newer than the Leader PRO (PRO Edition only)	Info	the Followers and the Leader
The Metalogger is newer than the Leader	Info	the Metalogger and the Leader
A Chunkserver is older than the Leader	Warning	the Leader and the last Chunkserver
A mount is older than the Leader	Info	the Leader and the last client

The same readings in the reverse direction are errors. Each reversed reading means that a component was upgraded out of order. The reversed readings are: a Follower or a Metalogger older than the Leader, a Chunkserver or a mount newer than the Leader, and the GUI or the CLI older than any server. Fix that component before you continue. The alerts reference collects the texts that the cluster shows.

Mixed versions make some changes unsafe or refused

• Do not downgrade a Chunkserver. To fix a Chunkserver that ended up ahead of the Leader, upgrade the Master Servers to that version. Do not reinstall the older Chunkserver package.
• Do not switch on a feature that needs a version floor across the whole cluster. A Storage Class that uses Erasure Coding cannot be defined as long as any connected Chunkserver or client is below the version that this format needs. The message says that one of the components is too old for the operation. Turning on USE_AUTH_FOR_IO in mfsmaster.cfg has a stronger effect. Chunkservers and clients below 5.0 are then refused registration outright. Leave both features off until the last client is upgraded.
• Do not leave a stopped Chunkserver down past its temporary maintenance window. A controlled stop suspends rebuilds only for that window. A server that stays offline longer is treated as failed. Its chunks then start being rebuilt (a controlled Chunkserver stop).
• Do not count on being able to reverse the procedure. An older Master Server refuses a metadata image that holds a section whose format is newer than the format that Master Server writes itself. When no section is newer, the image loads and the server starts. Which of the two happens is decided only at the first start of the older binary. Returning to the previous version is therefore a separate operation with its own requirements, described in rollback.

The Master Server step is where the editions part

Every step other than the Master Server step takes a single machine out of service at a time. The instance therefore keeps working during these steps in both editions. The Master Server step is the exception. In PRO PRO (PRO Edition only), the Leader role moves to a machine that already runs the new version, so the instance stays up. Rolling upgrade covers how to order that move and which machine holds the Leader role at the end. Community Edition has one Master Server, so this step pauses the whole instance. The pre-upgrade checks include the timing of this pause.