Rollback

An upgrade that went wrong raises two questions with different answers. The older package can always be reinstalled. The metadata itself decides whether the older Master Server reads what the newer Master Server has written. When the older Master Server cannot read it, restore the metadata instead of rolling back.

The package goes back, the metadata does not

Installing an older package changes nothing in the Master Server's data directory. The package copies metadata.mfs.empty into place only when neither metadata.mfs nor metadata.mfs.back is there. The package copies a .cfg.sample over only when the matching .cfg is missing. The outcome of a rollback is decided later, at the first start of the older binary on metadata that a newer binary wrote.

An older Master Server refuses a newer section

A metadata dump is a short header followed by sections. The sessions and Storage Classes come first, then objects, names, and locks, with the chunks last. The header names the container format, the metadata version, and the file ID. It says nothing about the sections. Each section has the number of its own format. The Master Server checks that number.

At startup, the Master Server compares every section in the file against the format that it writes itself. The Master Server reads a section in an older format than its own, and refuses a section in a newer format. The start ends there:

error reading metadata (node) - metadata in file have been stored by newer version of MFS !!!
error loading metadata.mfs - try using option '-a'
init: metadata manager failed !!!
error occurred during initialization - exiting

The first line names the section that stopped the load. The lines above it name the sections that loaded before that section. Between the first line and the second, the Master Server also prints one line for each structure that it discards from what it had read. The process exits before it writes anything. The dump, the changelogs, and the older dumps next to them therefore stay as they were. A refused start costs downtime and nothing else.

Neither mfsmaster startup option that loosens metadata handling bypasses that check.

• -i ignores metadata structure errors. This includes skipping a whole section whose name the binary does not recognize: unknown section found (leng:...,name:...) - all data from this section will be lost !!!. A known section at an unknown version is refused with -i exactly as without it.
• -a restores metadata from the changelogs. It chooses the file to start from by the highest metadata version in the data directory. Once it has a file with a valid file ID, it also scans the emergency locations. That is the same file that the check refused. A failed load shows error loading metadata file with the file's name instead of the message above.

When no section in the dump is newer than the format that the older Master Server writes, the dump loads and the server starts. The start attempt decides which of the two outcomes happens. The file has the number of every section that it holds, but the comparison uses the numbers built into the binary that reads the file.

An unrecognized changelog operation stops the replay

A changelog line is a metadata version followed by an operation and its arguments, written as text. A changelog has no header and no format number, so no part of a changelog is refused in advance. An operation that the binary does not recognize is refused when its line is read. The Master Server logs unknown entry together with the file, the metadata version, and the line. The replay stops at that point.

This can happen in two ways. A Master Server started with -a replays changelogs on top of the dump that it loaded. A Follower PRO (PRO Edition only) receives the Leader's operations as they happen. An operation that the Follower does not recognize is logged as NET:<version>: unknown entry '<operation>' and then as desync - operation (...) error: 32 (Data mismatch). The Follower then loses synchronization.

MooseFS 5 metadata does not load in MooseFS 4

The move from MooseFS 4 to MooseFS 5 is one-way. Once MooseFS 5 has written the metadata, MooseFS 4 does not read it. No startup option changes that. To go back across that boundary, restore the metadata copy taken before the upgrade.

Clients go back, Chunkservers do not, Metaloggers follow

To roll back any of these components, install the older package and restart the process. The components differ in which direction of version difference the cluster accepts. The alerts name the direction that the cluster treats as an error.

Component	Older than the Leader	Newer than the Leader
Client	Reported as information. The cluster accepts it.	Reported as an error. Downgrading the client software is one of the fixes the cluster names.
Chunkserver	Reported as a warning, and named as permissible during an upgrade.	Reported as an error, with one fix: bring the Leader and the other components forward. A Chunkserver is not downgraded, because a protocol mismatch in the write path can put incorrect data into the cluster.
Metalogger	Reported as an error.	Reported as information. The cluster accepts it.

A mount runs the version that it started with. A client package that was rolled back therefore takes effect when the file system is unmounted and mounted again. A Metalogger stores the Master Server's own dumps and changelogs in the format in which the Master Server wrote them. Changing the Metalogger package therefore does not change the format of the files that the Metalogger holds.

A rollback does not put the data on a Chunkserver's disks at risk. Chunks are ordinary files. The index that a Chunkserver keeps next to them is rebuilt by a disk scan whenever the index cannot be read. While a Chunkserver is out of the cluster, the chunk versions of the chunks stored on that Chunkserver can still change. The Chunkserver's copies of chunks that the cluster wrote to in the meantime have a version that is no longer current. The Master Server replaces them instead of counting them.

The Master Server rollback decides at first start

A group of Master Servers PRO (PRO Edition only) goes back together or not at all, because an older Master Server cannot stay in a group with a newer Master Server. The Leader logs got request for all metadata from a master FOLLOWER with version lower than mine - I'll refuse to send metadata - please upgrade this master (ip: <address>) and drops the connection. The older server cannot synchronize. It logs can't sync data likely because the master LEADER version is higher than mine - please upgrade me - exiting and ends its own process.

1. Stop the Master Server. In an installation with several Master Servers, stop all of them. A clean stop writes metadata.mfs. The order that the rest of the cluster follows is described in starting and stopping the cluster.
2. Install the older package on each machine. This step changes nothing in the data directory.
3. Start one server and read its log before you start any other server. The log shows the result. Either the metadata loads and the server reports that the metadata file has been loaded, or the start ends with the refusal of a section that the server cannot read.
4. Check the result with mfscli. mfscli -SIM reports the version and the state of every Master Server. In a group, one Master Server is the Leader. The remaining Master Servers are Followers with no synchronization suffix.

The decision closes earlier than the rollback does. From the moment the upgraded Master Server accepts its first metadata change, the copy taken before the upgrade lacks that change. The copy misses more changes for as long as the cluster keeps serving. When the metadata loads, a rollback an hour after the upgrade costs the same as a rollback a week after it. When the metadata does not load, their costs are very different.

A failed start leaves only the pre-upgrade copy

At that point the cluster has two possible states, not three: the version that it was upgraded to, with everything that the metadata holds now, or the version that it came from, with the metadata as of the copy. The copy of the metadata taken before the upgrade gives you that second state. Without that copy, there is no second state to choose. The cluster stays on the version that it was upgraded to.

A Metalogger does not replace that copy. The Metalogger stores what the Master Server sends to it. The metadata that the Metalogger holds is therefore the metadata that the newer Master Server wrote, at its newest metadata version. That is the state that the older binary refused.

On a machine that also runs a Metalogger, the Master Server and the Metalogger share a data directory. A Metalogger copy that is newer than the copy you restore stops the start: metalogger file is newer than current file - please check it manually - try using option '-a'. Move the Metalogger's files out of the directory before you start the Master Server on a restored copy.

warning

Restoring the pre-upgrade copy discards every metadata change the cluster recorded after the copy was taken. Files created, deleted, renamed, or moved to another Storage Class since that moment return to the state the copy holds. The choice is how much recent work to give up, not whether to undo the upgrade.

Diagnose a Master Server that does not start after an upgrade or after a rollback attempt as a Master Server problem, whatever the cause. Where the metadata is at stake and the message does not tell you what to do next, contact MooseFS Support before the next start attempt. A refused start leaves the files untouched. A start with -i drops what it cannot read and reports this in the log. The Master Server that starts that way stores what is left.