What changed in 5.x
Compared with the 4.x line, MooseFS 5 changes things that a running cluster is built on. The changes cover what the metadata file holds, what a Master Server needs before it starts, how modules prove who they are, and how the data is divided between servers. This page lists the changes that require something from the person who runs the cluster. It also describes what each change requires.
The upgrade from 4.x to 5.x follows the same order of upgrade as an upgrade within the 4.x line. The procedure differs in one respect: it cannot be reversed, because the metadata format changes. 5.x needs its own metadata and license covers what that means before you start. Dividing the cluster into locations PRO (PRO Edition only) is new in 5.x, but it is a separate step that you take after the upgrade, not a part of it.
The release notes record what changed in each individual release. The procedures are elsewhere: Order of upgrade covers the sequence in which the modules are replaced. Pre-upgrade checks covers what to verify beforehand. Version compatibility covers which versions work together. CE and PRO covers moving between editions.
5.x needs its own metadata and license
Metadata saved by a 5.x Master Server does not load in a 4.x Master Server. The move is one way. It happens at the first Master Server that starts on 5.x. Version 5.0.0 also added a section for the location map to the file. What the metadata holds covers the rest of what that file contains.
Keep the metadata image that the 4.x cluster wrote. Keep it until the cluster has run on 5.x long enough for you to trust the cluster on that version. This image is the only way back to the older line. Writing the image out is part of pre-upgrade checks.
A PRO cluster also needs a license issued for MooseFS 5 PRO (PRO Edition only). A license granted for the 4.x line does not start a 5.x Master Server. The GUI reports the refusal as an alert that says that the license covers the previous version of the software. Ask support for the new license and install it before the version changes. Since 4.59.1, a license can be replaced while the cluster still runs 4.x. The license and the version therefore do not have to change in one maintenance window. Installing and verifying the file is described in install a license. The alert is listed with the rest in alerts.
The 5.x line also changed what happens when the license file is missing. A 4.x Master Server did not start without a license file. A 5.x Master Server has a 10 TiB Personal license inside it and runs on that license whenever it cannot open the file. When the license file is absent, or its permissions keep the Master Server from opening it, the cluster therefore starts under Personal terms rather than refusing to start. A license file that the Master Server can open but cannot load still stops the Master Server.
Options come from file, environment, or command line
Since 5.0.1 the Master Server, the Chunkserver, and the Metalogger take the value of a configuration option from the configuration file, from an environment variable named MFSCFG_<OPTION>, or from -o OPTION=value on the command line. The command line outranks the environment. The environment outranks the file.
Up to that release the file was the only source. When a daemon does not appear to honor a value, now look beyond the file that the daemon reads. Check the unit file that starts the daemon and the environment that the daemon inherits. Where a tool that exports its own variables configures a fleet, check that none of those variables collides with an option name.
Placement can answer to locations
MooseFS 5.0.0 added locations PRO (PRO Edition only). An instance can be divided into locations. Each location has its own servers. A Storage Class can define placement separately for each location. Location covers what a location is and how many an instance may have. When locations help covers what dividing a cluster gains and what it costs.
An instance upgraded from 4.x keeps running as a single location and needs no decision on the day of the upgrade. You make the decision when you create your own first location. The default location that the instance always had then becomes visible next to the new location. Every existing Storage Class definition stays attached to that default location. Plan the conversion with Location tasks. That page describes the order in which the locations are created and renamed.
The rule that keeps the cluster from deleting the last copy of a chunk came with locations. Version 5.1.0 corrected this rule for a location configuration that is not consistent. The same release corrected the location mappings that the Master Server keeps for its Chunkservers and the address remapping that goes with them. Therefore, take a cluster that is going to be divided to 5.1.0 before you create the first location, not after.
The archive flag follows the governing class
Since 5.0.0, the Master Server sets the archive flag on a chunk according to the effective Storage Class of that chunk. Where one chunk belongs to files of several classes, archiving follows the class that governs the chunk rather than each file that points at it. A snapshot creates such a case.
Check the classes on trees that share chunks with snapshots before you conclude that archiving stopped working there. Storage Class evaluation covers the whole rule, including the mode flags that decide when the flag is set and what clears it.
A slow disk is flagged, a rebalance unblocks
Version 5.1.0 adds HDD_IO_SECONDS_TO_ERROR to mfschunkserver.cfg. When this option is set, an operation on a chunk file that takes longer than the given number of seconds is recorded against the disk, even though the operation succeeded. The option is 0, which means off, until you set it. The upgrade alone therefore changes nothing here. Turn the option on where the failure that you keep seeing is a disk that hangs rather than a disk that returns errors. In 5.1.0 the record makes that disk visible without counting toward the tolerance that takes a disk out of service. Since 5.2.0 it counts, so a disk that keeps hanging is taken out of service. Disk error handling and disk removal describes this in full.
Also in 5.1.0, a slow internal rebalance no longer stops its destination disks from accepting new chunks. A disk that the rebalance is filling therefore accepts new chunks at the same time. Nothing has to be reconfigured for that. The two rebalance modes and the cost of each are described in rebalancing.
Version 5.0.1 removed the aggressive fadvise calls that a Chunkserver made around chunk files. The caching of those files is therefore left to the operating system. There is nothing to set for this change.
5.0.2 repaired chunk testing and Erasure Coding
A bug in 5.0.0 and 5.0.1 kept the background chunk test from running. This test is the pass that reads chunks from every disk at a fixed rate and compares them with their checksums. Version 5.0.2 fixed the bug and added a checksum check to the local split of a chunk. Scrubbing and CRC verification describes what the test covers, how fast it goes, and what it does with a mismatch.
The same release fixed the sorting of parts where more than one Erasure Coding set was in use PRO (PRO Edition only). Erasure Coding layout describes the layout that those parts follow. Neither fix needs configuration. A cluster left on 5.0.0 or 5.0.1 runs without the pass that finds a bad copy before a reader does. That alone is reason enough to upgrade it past those versions.
An export can cap a mount's operation rate
Version 5.1.0 added opslimit and opsperuidlimit to mfsexports.cfg. opslimit caps the I/O operations of a whole mount. opsperuidlimit caps each uid on it separately. Neither option applies unless an entry sets it, so the upgrade leaves every mount as it was.
Where a single client has been able to overload the Master Server, set these options to stop that client. The values are described in access control, together with the operations that are excluded from the count and the file on the client that shows the limit in force.
df follows the directory it runs on
Since 5.1.0, the Master Server takes the queried directory into account when it answers statfs. Under a directory that has a quota, or under a directory below it, df reports values trimmed to what that quota leaves. df -i reports values trimmed to an inode quota wherever such a quota governs the path.
Look at whatever reads free space from inside a mount before you decide that a subtree shrank overnight. From now on, a monitoring check that watches df on a path that has a quota reports the quota. Space management explains which reading answers which question.
A password on the command line outranks the configuration
In 5.1.0 the -p option of mfsmount takes precedence over mfspassword and mfsmd5pass. A mount invoked with both now authenticates with the password that -p supplies.
Look through the mount units, fstab entries, and scripts that pass a password twice. Leave only one password in place.
The client package carries fixes of its own
Version 5.1.0 brings libmfsio closer to POSIX behavior and ships Doxygen documentation with it. The same version fixes a use-after-free and the unmap protocol in mfsbdev. It also fixes a timeout bug in the write path of the client. POSIX compliance covers what MooseFS promises about POSIX behavior. Block device operations covers mapping a file as a device.
Run programs built on libmfsio against the new library before they follow the cluster. The block device fix and the write fix need nothing beyond the new package on the hosts that use them. Those hosts are the machines that map block devices and every machine that writes through a mount still on 5.0.x.
Locations and licenses got their own tools
mfslocadmin PRO (PRO Edition only) came with locations in 5.0.0. It ships under its own name and as eight links named after the operations that it performs. Location tasks covers what to do with it. Since 5.2.0, every mfslocadmin operation that changes locations needs a mount admitted with admin. Without it, the tool reports Operation not permitted (mfs admin only). Up to 5.1.0, any mount was enough. Version 5.1.0 added mfslicinfo, which reports whether every Master Server holds the same license. A license replacement raises this question on a cluster with several Master Servers. Both tools come only with PRO packages. A machine with administration scripts therefore needs the PRO client and CLI packages to run them.
The renames that break scripts belong to the 4.x line rather than to the 5.x line. In that line, mfstools was split into separate programs. The trash time tools were deprecated in favor of trash retention tools. If your scripts still call the names from before that split, commands that have been replaced lists which name replaced which.
The readings that count chunks carry the location
Since 5.0.0, the chunk loop counts per location PRO (PRO Edition only) and reports its exit reason per location. Version 5.1.0 corrected two of those reasons. On an instance divided into locations, the JSON that the CLI prints has an extra level of keys above the chunk matrix summary. Without -l, that level has one key, all_locations, for all locations together. With -l, its only key is the chosen location, as location_<id>. Since 5.2.0 these keys contain no spaces. Up to 5.1.0 they read all locations and location <id>. Anything that parses those numbers has to handle that level.
Test the monitoring that reads the cluster before you create the first location, not after. The chunk loop and the file loop covers what the loop counts. The shape of the output is described in the CLI. The same page covers the -SLO section that reports the state of each location and the -l switch that narrows a section to one location.
The cluster names what it cannot place
Version 5.1.0 makes the Master Server log the name of a module whose address maps to no location PRO (PRO Edition only). The GUI shows a warning about the same condition: the server was assigned to the default location implicitly. Assign every server to a location explicitly. As long as any server is assigned implicitly, the default location cannot be switched. The alert and its remedy are in alerts.
On an instance already divided into locations PRO (PRO Edition only), the same correction can leave Chunkservers out of the cluster on the day of the upgrade. Where the location mappings do not cover every address that a Chunkserver answers on, version 5.0.2 admitted the server anyway and decided its location as well as it could. Version 5.1.0 refuses that registration instead. Nothing in the configuration has to change for this to happen: the new Master Server reads the file that worked before the upgrade in a different way. Divide a new cluster into locations covers which addresses have to be mapped, and why.
The gap is visible before the upgrade, on the running cluster. List the addresses that belong to a location:
mfslocadmin list
Then read the address classes named in mfsmaster.cfg by REMAP_SOURCE_IP_CLASS, REMAP_DESTINATION_IP_CLASS, and MULTILAN_CLASSES. Count only the classes that the file actually sets. A class that is set there and appears in the mapping of no location is the missing mapping. mfslocadmin map adds it. Do that before the packages change, not after.
The same version makes the Metalogger check its writes and log what fails. A Metalogger that cannot write its copy of the metadata therefore reports the failure. Add the Metalogger log to what you watch, because the copy that the Metalogger keeps is current only up to the last write that succeeded. Metalogger covers what the Metalogger stores. Logs covers where its messages go.
Authentication reaches every module and every I/O operation
Version 5.0.0 extended authentication to every module that registers with the Master Server, including the Metalogger. Setting AUTH_CODE therefore now binds Metaloggers and Follower Master Servers PRO (PRO Edition only) as well as Chunkservers. Configure the modules first and the Master Server last, because a module presents the code only when it is asked. Authentication between modules covers which side proves what to whom. The steps are in access control.
The same version added USE_AUTH_FOR_IO to mfsmaster.cfg. This option makes reads and writes to Chunkservers carry a credential of their own. The option is 0, which means off, until you set it. Turning the option on shuts out everything that is not ready for it: AUTH_CODE has to be set. Every export has to have a password. Since 5.2.0, the Master Server skips every entry without one and logs a warning for it. A file left with no entry stops the Master Server at startup and at reload (export rule evaluation). No client older than 5.0 can register while the option is on. RESTRICT_INCOMPATIBLE_CLIENT_VERSIONS does not exempt those clients, so setting it to 0 does not let them back in. Upgrade every client, put passwords on the exports, and turn the option on last.
Version 5.0.1 improved the authentication between a client and a Chunkserver and fixed security issues across the modules. This is one more reason to upgrade the clients with the servers rather than long after them.