Pre-upgrade checks

Work through this list on the cluster in its current state, before the first package is replaced. Each item says what to read and what the reading means once the upgrade is under way. It also gives the value that lets you go on. The steps of the upgrade itself are listed in order of upgrade.

Redundancy is whole and every Chunkserver answers

An upgrade takes the cluster out of service one component at a time, so the upgrade uses up the redundancy that is left. A chunk that is endangered today has one copy. That copy can be stored on the first Chunkserver that you stop. Read the state of the data before you stop anything:

mfscli -SIC
mfscli -SMF
mfscli -SCS
mfscli -SHD

Go on when the summary line of the chunk matrix shows zero for missing, endangered, and undergoal, the missing files report is empty, every Chunkserver is listed as connected, and no disk has the status damaged or invalid. A count that is falling is still above zero. Let the repair finish instead of starting an upgrade while the repair runs.

Cluster health parameters covers what each figure counts, and what a value other than zero says about who resolves it. The CLI describes which section shows which reading. The options of mfscli are listed with the other command-line tools.

The metadata on disk is from this moment

Metadata is saved to disk on a schedule, hourly by default. The file on disk is therefore older than the changelog next to it. Order a new save instead of using the existing file:

mfssupervisor -s A

A selects the servers to store on: all of them, with a checksum comparison afterwards. The mode letter is available only in the PRO Edition PRO (PRO Edition only). In the Community Edition, mfssupervisor accepts only -s.

Go on when mfscli -SIM reports the save that you ordered. last meta save shows the current time, last save status shows Saved in background, and every Master Server shows the same last save version with the same last save checksum next to it. The Leader writes all followers have metadata consistent with the leader to its system log when the comparison succeeds. Metadata save states covers what the four save statuses say about the file that each save wrote.

A metadata copy sits off the Master Server

An upgrade that has to be undone is undone from metadata written by the version that you are leaving. This copy is therefore the only thing that the upgrade must not be able to reach. Make the copy after the save above, and store it on a separate machine. A dump is renamed into place only after it is finished, so a file with that name is never a partial write. mfsmetadump reads the header of the dump without a Master Server:

mfsmetadump -s HEAD /var/lib/mfs/metadata.mfs.back
scp /var/lib/mfs/metadata.mfs.back backup.example.com:/backup/mfs/metadata-before-upgrade.mfs
scp /var/lib/mfs/changelog.*.mfs backup.example.com:/backup/mfs/

Copy the changelogs together with the dump. A dump holds the state at one moment. Every operation that the cluster has carried out since that moment exists only as a changelog line. A restore loads the dump. The restore then replays those lines on top of the dump to reach the state that the cluster was in (changelogs, dumps, and the recovery point). A copy of the dump alone restores the instance to the save that you ordered. Everything the cluster did afterwards is lost.

Go on when the header names the metadata version that the save above wrote, and the dump and the changelogs are stored on the backup machine. Copying metadata off the machine covers the same task outside an upgrade.

A Metalogger keeps a second copy on a separate machine and receives the changelog stream. mfscli -SML lists the Metaloggers that are connected. A Metalogger does not replace the copy above. The dump on the Metalogger's disk is from its last scheduled download. The hours since then exist there only as changelog lines.

There is room for the stop's dump

The upgrade stops the Master Server. A Master Server writes a dump when it stops. A Master Server that has nowhere to store that dump does not finish stopping. It logs can't store metadata - try to make more space on your hdd or change privileges - retrying after 10 seconds and retries every ten seconds, without giving up. The stop command has a limit of 30 minutes instead. mfsmaster stop waits that long for the process to release its lock. The command then logs that it is giving up and leaves the process running. The shipped unit gives the stop the same 30 minutes. The cluster raises no alert about the space in that directory, so check the space on the machine itself:

df -h /var/lib/mfs

Go on when the data directory has room for another dump the size of the current metadata.mfs.back. That room is in addition to the older dumps and the changelogs already stored there. The sizing of the directory is described in the disk holds metadata images and changelogs.

On the Chunkservers, calculate the space that is left when one of them is out of service. A Chunkserver stopped in a controlled way announces a maintenance window of its own. A Chunkserver that stays offline longer than that window is treated as failed. Its chunks are then rebuilt on the remaining servers (a controlled Chunkserver stop does not set off a rebuild). Go on when no Chunkserver in mfscli -SCS is at 99 percent used or more. At that point, the Master Server stops choosing the Chunkserver as a target for replication. New chunks have a different limit. A Chunkserver receives new chunks only as long as it reports more free space than one chunk. That threshold rises by ten chunks for every chunk copy on that Chunkserver that clients hold open for writing (a candidate needs room beyond one chunk).

The license covers the version you move to

A license PRO (PRO Edition only) includes the highest MooseFS version that it permits. A Master Server with a newer version than that does not start. It prints Version in licence too old on the screen and in the system log. Check what the current license allows before you change the version:

mfscli -SLI

Go on when max version covers the version that you are moving to, and expires falls outside the window that you are working in. mfslicinfo was added in 5.1.0. Where an instance runs several Master Servers, the tool queries each of them separately and prints a single block when they all report the same license:

mfslicinfo mfsmaster

A license issued for MooseFS 4 does not start a MooseFS 5 Master Server. This also applies to a license that has no version limit. Version 5 needs a license of its own. To put a new file in place, follow install a license.

Every version is written down and checked

While the upgrade runs, the cluster runs more than one version at once. Only some mixtures are allowed. Record what runs today, component by component:

mfscli -SIM
mfscli -SCS
mfscli -SMS
mfscli -SML

Keep that output. It shows you afterwards which component was missed. Also check the output against version compatibility. That page covers whether the version that you are moving to may run next to the versions that stay.

The configuration is saved and agreed on

Installing a package does not change a configuration file that already exists. The new .cfg.sample is put next to it. Your .cfg stays unchanged. This prevents the upgrade from changing how the cluster behaves. For the same reason, an option added in the new version is absent from your file afterwards. On each machine of the cluster, keep a copy of the configuration that you run. You can then compare your file with the new .cfg.sample after the packages are upgraded:

tar czf /root/mfs-config-before-upgrade.tar.gz /etc/mfs

Go on when the archive holds every file that the components on that machine read, including mfsmaster.cfg and the files that it points to by path.

Where several Master Servers PRO (PRO Edition only) serve one instance, each Master Server reads its own copy of mfsmaster.cfg. The copies have to be the same, because the upgrade moves the Leader role from one of those machines to a different machine:

mfssupervisor -c

Go on when the Leader's system log shows no option ... has different value line after the run. The same check outside an upgrade is described in Master Server operations.

The way back is settled first

A Master Server refuses metadata with sections written by a newer version than its own. Startup options cannot override that refusal. For that reason, moving from MooseFS 4 to MooseFS 5 is a one-way move. Metadata that version 5 has written does not load under version 4. Rollback covers which components can be moved back at all, and whether the copy taken above can be used when they cannot. Read that page while the cluster still runs the version that you know.

The new version's changes are read first

A new version adds behavior that the cluster did not have. Some of that behavior needs a decision from you. An option added in the new version is absent from your configuration file after the packages are upgraded. A default that changed applies to the cluster as soon as the module starts. In release notes, read every release between the version that you run and the version that you are installing. What changed in 5.x covers what an administrator coming from version 4 has to decide before the move.

Go on when you have read the entries that name a new option or a changed default. The entries that need a value of your own also have to be written down for the step that merges the configuration files.

The upgrade has a window the mounts sit through

Every step restarts one module. Each restart pauses the mounts. Operations are not lost in the pause. A break in the Master Server connection stalls the operations caught in it rather than failing them. Each of those operations is sent again. An application receives an I/O error only if the mount's budget of attempts runs out first. The pause adds latency. The length of the pause differs between the editions. In PRO, the instance keeps serving during the whole upgrade. The change of Leader affects the clients (rolling upgrade). Community Edition serves from one Master Server. The instance is therefore unavailable for as long as that machine takes to stop, upgrade, and start again.

Go on when the upgrade has a suitable window. That window is either the quietest hours of the cluster, or a time when the client processes are stopped for the duration. An application with an I/O timeout shorter than the pause reaches that timeout. Find such applications now, not while the packages are being replaced.

When every item passes, the upgrade starts

The values to have before you start, section by section:

1. Redundancy and Chunkservers: missing, endangered, and undergoal at zero, the missing files report empty, every Chunkserver connected, no disk damaged or invalid.
2. Metadata on disk: the reported save is the save that you ordered, and every Master Server PRO (PRO Edition only) reports the same version and checksum.
3. Copy off the machine: the dump header names that same metadata version, and the dump and the changelogs are stored on a separate machine.
4. Room for the stop's dump: the data directory has room for another dump, and no Chunkserver is at 99 percent used.
5. License PRO (PRO Edition only): the license covers the version you are moving to, and its expiry falls outside your window.
6. Versions recorded: the output is kept, and the target version may run next to the versions that stay.
7. Configuration saved: the archive holds every file the machine reads, and the Master Servers report no differing option.
8. The way back: you have read and understood what can be moved back, and whether the copy can be used where a component cannot.
9. What the version changes: the entries between your version and the target are read, and the new options that need a value of your own are noted.
10. The window: the upgrade has a window, and no application on a mount holds a timeout shorter than the pause.

Continue with order of upgrade. That page describes which component is upgraded first and what the cluster is allowed to report during each step.