Version compatibility
The components of one MooseFS instance are rarely installed on the same day. Every upgrade puts a cluster into a state where the components differ on purpose. This page covers which differences a cluster is meant to run with and which ones a module refuses outright. It also covers the differences that a module accepts and that then change how the module behaves.
The Master Server sets the version for the rest of the instance. In a PRO instance the Leader Master Server PRO (PRO Edition only) sets that version. The checks inside the modules and the alerts you see compare every other component with that Master Server.
Run the newest release: a mixed cluster is temporary
Run the newest release on every component. The differences that this page allows make it possible to upgrade one component at a time, without stopping the cluster. They exist for that purpose. A cluster left on mixed versions for months is in the middle of an upgrade that never finished. Running mixed versions is not a feature of the cluster.
The distance between 3.x, 4.x, and 5.x does not measure how much the product changed. MooseFS is developed in small steps. A major number marks where a series began. It does not mark a break in what the software does. The gap between two numbers therefore shows how far apart two components were built, and little else.
Versions form a chain with the Master Server inside
Put the components in this order: first the GUI, the CLI, and other tools, then the Metalogger, then the Follower Master Servers PRO (PRO Edition only), then the Leader, then the Chunkservers, and last the mounts. A component may run a newer version than every component after it in that order. It may never run an older one. The order allows a cluster to be in any of these states:
- Every component on one version. This is the state that an upgrade returns the cluster to.
- Master Servers, Metaloggers, and Chunkservers on one version, mounts older than them.
- Master Servers and Metaloggers on one version, Chunkservers older than them, mounts older than both. This combination runs. Outside an upgrade, staying in this state has no benefit.
- A Master Server older than a Chunkserver, or a Master Server or Chunkserver older than a mount. Do not run this.
The direction of the chain follows from how the exchanges are built. A newer module supports both the older form of a message and its own form. It falls back to the older form when it talks to an older module. An older module cannot produce a form that did not exist when it was built. Therefore the newer module does the adapting. A pair of modules keeps working only as long as the newer module adapts.
The chain covers the combinations that are tested and supported. It does not mark the limit of what will start. More combinations start than the chain permits. A cluster that starts is not evidence that its combination of versions is sound. The gap in major version numbers between any two components should also stay within one. A component on 5.x therefore works with a component on 4.x, and with nothing older than that. The order of upgrade describes which component is upgraded first and what completes each step.
Which side is newer decides what happens
Registration handles a version difference in different ways. The Master Server either refuses a version difference and logs the reason, or accepts the difference. When the Master Server accepts the difference, the Master Server either trims what it sends to the form that the older side understands, or changes nothing at all. The outcome is fixed for each pair. Most pairs have no version comparison at all.
Outside those checks, every pair has the same failure mode. A module that receives a packet type that it does not recognize logs got unknown message and closes the connection. The Master Server answers an unknown request the same way, instead of returning an error to the sender. An unsupported combination therefore appears as a connection that is repeatedly dropped and re-established, not as a message that names a version. A registration that logs nothing does not mean that the versions are compatible.
Each row compares one component with the Master Server that the component talks to. That Master Server is the Leader PRO (PRO Edition only) where the instance has one.
| Component | Newer than the Master Server | Older than the Master Server |
|---|---|---|
| Chunkserver | Registration is accepted. The Master Server does not compare a Chunkserver's version against its own. Whether the pair keeps working depends on whether either module sends a packet type the other module does not know. | Registration is accepted. The Master Server drops from its side of the exchange what the Chunkserver cannot follow: a shorter acknowledgment for a Chunkserver that predates a field, no I/O authentication key below 5.0.0, and no local split of an Erasure Coding (EC) chunk below 4.14.0. Registration is refused, with the reason in the Master Server's log, when a Storage Class using EC exists and the Chunkserver is below 4.0.0, when a 4+n class exists and the Chunkserver is below 4.26.0, or when I/O authentication is on and the Chunkserver is below 5.0.0. |
| Mount | Registration is accepted. The Master Server does not compare a mount's version against its own. A request the Master Server does not know closes the connection instead of returning an error. | Registration is accepted. The answers are trimmed to what the mount understands. With a Storage Class using EC, a mount below 4.0.0, or below 4.26.0 for a 4+n class, is refused with a permission error while RESTRICT_INCOMPATIBLE_CLIENT_VERSIONS in mfsmaster.cfg holds its default of 1. When that option is set to 0, the mount registers instead. Every read of an EC chunk the mount cannot assemble returns an I/O error while the file stays as it is. With I/O authentication on, a mount below 5.0.0 is refused whatever that option says. |
| Metalogger | Accepted. Neither side compares versions with the other. Neither side logs anything about the difference. | Accepted on the same terms, and again neither side says anything. The chain still requires a Metalogger at least as new as the Master Servers. The cluster's own alerts are the only place the difference is named. |
| Follower Master Server PRO (PRO Edition only) | Allowed, and neither process reports it. | The Leader refuses to send a full metadata set to a Follower older than itself, logs that this Follower has to be upgraded, and closes the connection. That applies to a Follower which holds no metadata of its own yet, either a new Follower or a Follower started with -e. A Follower that finds the Leader newer than itself during a resynchronization logs that it has to be upgraded and stops its own process. |
| GUI, CLI, and tools | The supported direction. Individual tools carry thresholds of their own. mfspatch refuses a Master Server below 4.38.0 and says so. mfsscadmin falls back to the Storage Class protocol the Master Server understands, so a class carrying what that older protocol cannot express is refused with One of MFS instance components is too old to perform this operation. | Nothing refuses them, because they read and display rather than register a session. They may not understand every packet the servers exchange, and can present wrong values as a result. |
Trimming removes something different in each pair. The modules report none of it. A mount that predates a field receives a shorter record for every Chunkserver in an answer to a read or a write. A mount below 4.57.0 receives goal bounds instead of Storage Class groups. A supervisor below 5.0.0 receives a state answer with no location PRO (PRO Edition only) in it. The component keeps working with the smaller answer. You see that part of the answer is missing, but no message tells you so.
You can also refuse old mounts yourself, one export entry at a time. The minversion= option in mfsexports.cfg admits only clients at or above the version that it names. A client below that version is refused with a permission denied error. Give that version as X, X.Y, or X.Y.Z.
Some pairs have an absolute floor
In addition to the differences that depend on the version of the partner module, each row below is an absolute minimum. Below this minimum, the two modules do not communicate at all, whatever versions the rest of the cluster runs.
| Side that refuses | What it refuses | What it does |
|---|---|---|
| Mount | A Master Server older than the floor it accepts, which is below 3.0 | Logs incompatible mfsmaster version, drops the connection and retries. |
| Chunkserver | A Master Server older than the floor it accepts, which is below 3.0 | Logs MATOCS_MASTER_ACK - unsupported master version and stops its own process. |
| Master Server | A Chunkserver whose registration it does not recognize | Logs that the Chunkserver is too old, or that the registration is not supported, and closes the connection. |
| Master Server in the Community Edition | Any Follower Master Server | Logs ANTOMA_REGISTER - protocol not supported and closes the connection. |
A Chunkserver has one more refusal that has nothing to do with versions. The Chunkserver compares the metadata ID that the Master Server sends back with the ID in the Chunkserver's own working directory. If the two IDs differ, the Chunkserver stops its process and logs both IDs. A Chunkserver pointed at the wrong instance fails this way, and not as a version problem.
Some features wait for every component
A feature that involves several modules stays unavailable until each of those modules is new enough for it. A partly upgraded cluster does not behave like a slightly older cluster: part of what it did before is switched off. Part of what a new Master Server can do stays unavailable until the remaining components are upgraded too.
| Feature | Master Server | Chunkserver | Mount or tool |
|---|---|---|---|
| Erasure Coding 8+n | 4.0.0 | 4.0.0 | 4.0.0 |
| Erasure Coding 4+n | 4.26.0 | 4.26.0 | 4.26.0 |
| Storage class groups in export entries | 4.57.0 | not involved | 4.57.0 |
| Authentication of modules and of chunk I/O | 5.0.0 | 5.0.0 | 5.0.0 |
| Locations PRO (PRO Edition only) | 5.0.0 | not involved | 5.0.0 |
| Operation limits on a session | 5.1.0 | not involved | 5.1.0 |
The requirement also works in the other direction. The first Storage Class that raises the instance's Erasure Coding level cannot be defined while even one connected Chunkserver or one registered mount is below the version that this level needs. Once the level is raised, it stays raised, and later classes at that level are defined without another check. The Master Server answers the attempt to define that first class with One of MFS instance components is too old to perform this operation. The Master Server takes the lowest version among all connected Chunkservers and the lowest among all registered sessions. One forgotten mount therefore blocks the change for the whole instance.
Some thresholds take effect without any message. Operation limits need the most attention. A mount below 5.1.0 is sent no limit, and the mount reports none back when it reconnects. The limit configured for that session therefore does not apply. Nothing on either side says so.
The version number is the only protocol number
MooseFS has no protocol number separate from the product version. Every compatibility decision in the cluster is based on the version number that a module reports when it registers. A comparison of version numbers is therefore a comparison of protocols.
One exchange has a number of its own. Reads and writes between a mount and a Chunkserver carry a small protocol number. The mount derives that number from the product versions of the modules involved. A chain-like write runs at the version of the oldest Chunkserver in the chain, because the mount picks the form that every Chunkserver in the chain supports. For a read, the version is decided separately for each Chunkserver. Replication between Chunkservers has no version condition at all. In replication, the version of the chunk being copied has to match.
The edition is also part of the version number. The third part of the version is doubled on the wire, and its lowest bit marks PRO. The GUI and the CLI therefore print a converted number. At equal numbers, a Community Edition component ranks below a PRO component. A CE Chunkserver connected to a PRO Leader is therefore reported both as the wrong edition and as older. Two version ranges are older than this scheme and are read differently. A component that reports 4.0.0 through 4.10.x keeps the reported number, and the third part is not halved. The GUI and the CLI count that component as Community Edition, regardless of the edition it was built as. A mount or Metalogger that reports 4.23.5 through 4.47.x is counted as PRO, regardless of the edition it was built as.
Only the mount may differ in edition
Because the edition is part of the version number, a component of the wrong edition counts as a version mismatch and is treated as one. Keep every server module and every tool on one edition. The mount is the exception: no module examines its edition. A PRO mount connected to a Community Edition cluster works.
| Pair | Run it? | What happens |
|---|---|---|
| PRO Master Server and CE Chunkserver | No | Registration is refused, with without pro features - master with pro features needs chunkservers with pro features in the Master Server's log, and the connection closed. |
| CE Master Server and PRO Chunkserver | No | Nothing in the modules refuses it. Reported as an error. |
| Leader and a Follower PRO (PRO Edition only) of the other edition | No | Nothing in the modules refuses it. Reported as an error in both directions. |
| Master Server and a Metalogger of the other edition | No | Nothing in the modules refuses it. Reported as an error in both directions. |
| PRO Master Server and CE mount | With a reservation | Nothing examines a mount's edition. Reported as an error, on the ground that a CE mount does not implement part of the protocol. |
| CE Master Server and PRO mount | Yes | Nothing examines a mount's edition. Reported as a warning, for consistency alone. |
| PRO tools and a CE cluster | Yes | Reported as information. |
| CE tools and a PRO cluster | No | Reported as an error: the CE tools do not cover everything a PRO cluster does. |
The cluster reports a mismatch against the Leader
Every connected component's version is listed against the Leader's in the GUI and in the CLI. A difference raises an alert for the component that it concerns. The alert appears in the GUI, next to that component. The CLI tables have no alert list. Alerts covers what each alert says and what to do about it.
The severity follows the chain, not the size of the difference. For the tools, the Metalogger, and a Follower PRO (PRO Edition only), running older than the Leader is an error. Running newer is only noted. For a Chunkserver and a mount, the rule is reversed. Running newer than the Leader is an error. Running older is a warning for a Chunkserver and a note for a mount. A component of a different edition than the Leader is reported as the wrong edition in every case. The edition ranks below the version number. A CE component with the same number as the Leader therefore raises the older-than-the-Leader alert at the same time.
For several of these differences, these alerts are the only place where the difference is named at all. Neither a Metalogger nor the Master Server that the Metalogger logs for reports the gap between them. A Chunkserver or a mount that is too new is not refused. It registers and then drops its connection the first time the exchange reaches a packet that the partner module does not recognize.