Master Server operations

This page describes the planned tasks around a running Master Server and its metadata: forcing a metadata dump, copying the dump off the machine, reading the dump offline, and applying configuration changes. An installation with several Master Servers PRO (PRO Edition only) has three more tasks: watching the states of the servers, moving leadership, and adding a server to the group. Everything on this page assumes a healthy cluster.

A metadata dump can be forced anytime

The Master Server writes a dump of its metadata on a schedule and once more at a clean stop. METADATA_SAVE_FREQ in mfsmaster.cfg sets the schedule, every hour by default. Before a planned intervention, or before you take a copy, order a dump with mfssupervisor instead of waiting for the schedule:

mfssupervisor -s A

The mode names which servers store the dump: -s L the Leader only, -s F the Followers only, -s A all servers with a CRC check, and -s C the CRC check alone, with no dump written. In PRO, the mode is required. A bare -s is refused. In the Community Edition, one Master Server does the storing. The option takes no mode there, and mfssupervisor -s is the whole command.

The tool finds the Master Servers through the host name mfsmaster. -H names a different host name. If AUTH_CODE is set in mfsmaster.cfg, the tool must present the code. -p prompts for the code. The MFSAUTHCODE environment variable supplies the code without a prompt.

To verify, print the general summary with mfscli. mfscli -SIG shows the timestamp, duration, and status of the last metadata save. The timestamp changes when the forced save finishes. With several Master Servers PRO (PRO Edition only), mfscli -SIM shows the same information for each server. Metadata save states explains what the outcomes mean.

The data directory holds the metadata on disk

The Master Server keeps its files in the data directory. DATA_PATH in mfsmaster.cfg points at this directory (/var/lib/mfs in the packages). The Master Server page lists what the directory holds and what each file is for. Changelogs, dumps, and the recovery point describes when each file is written, renamed, or dropped. The space that the directory needs is a planning decision. The formula is in the disk holds metadata images and changelogs.

mfsmetadump reads a dump without the Master Server

mfsmetadump prints what a dump holds, directly from the file:

mfsmetadump -s HEAD /var/lib/mfs/metadata.mfs.back

-s HEAD prints the file header: the version of the metadata file format, the metadata version that the file holds, and the metadata ID of the instance that the file came from. The dump does not record the MooseFS version that wrote it. Without -s, the tool prints the whole file section by section, from client sessions to chunks. mfsmetadump(8) lists the sections and their fields.

A metadata copy leaves by dump or Metalogger

You can copy a dump while the Master Server runs. A save never writes into an existing dump. The save builds the new file under a temporary name and renames only a finished file. Every file named metadata.mfs.back is therefore complete. A save that finishes during your copy renames the old dump to metadata.mfs.back.1 rather than changing it (how a dump reaches the disk). To copy a dump of this moment rather than of the last save, order a save first:

mfssupervisor -s L

The order only sends the request. The Leader writes the new dump under a temporary name in a separate process. At the end, the Leader renames the file to metadata.mfs.back. Until that save finishes, the file on disk is still the previous dump. Wait until last successful store in mfscli -SIG shows a time after the moment you gave the order, then copy:

scp /var/lib/mfs/metadata.mfs.back backup.example.com:/backup/mfs/metadata-$(date +%F).mfs

In an installation with several Master Servers PRO (PRO Edition only), the Leader does not store on the regular schedule. A Follower stores instead, so the newest dump in the Leader's own directory may be a day old. Force a save on the Leader first (mfssupervisor -s L), or copy from a Follower. To verify a copy, print its header with mfsmetadump -s HEAD. The header names the metadata version that the copy holds.

A copied dump records one moment. A Metalogger keeps a copy current instead. The Metalogger receives every changelog line as the Master Server writes the line, and downloads a new dump on a schedule of its own (the Metalogger stream). The Metalogger never takes over the Master Server's work. The Metalogger's files exist for a Master Server that is gone, so that you can start a Master Server from them. In the Community Edition, no Follower Master Server holds a second copy. There, a Metalogger is the only component that keeps the metadata current outside the Master Server's machine.

A copy is worth what a restore proves

A dump that you have never restored is an assumption. Proving the dump costs one machine and no downtime, because the proof does not need a running cluster. mfsmaster restore reads the metadata and the changelogs, writes metadata.mfs, and ends. The restore starts no listeners, elects nothing, and does not even read mfsexports.cfg, so nothing it does can reach the cluster that the copy came from.

1. Take the copy together with its changelogs. A dump records one moment. The changelogs record what happened after the dump. A restore replays only an unbroken run of the changelogs. Copy metadata.mfs.back and the changelog.*.mfs files from the Master Server's data directory, or metadata_ml.mfs.back with the changelog_ml*.mfs files from the data directory of a Metalogger.

2. Put the files in the data directory of a Master Server that is installed on a separate machine and has no cluster to join.

3. Check the file before replaying anything into it:

   mfsmetadump -s HEAD metadata.mfs.back

   The header names the file format, the metadata version, and the metadata ID. Read the exit status as well as the printed lines. The tool parses the whole file to reach the end marker of the file. A truncated or damaged copy therefore leaves the status non-zero, even though the header has already printed.

4. Replay the changelogs and write the result:

   mfsmaster restore

   The action enables the automatic restore by itself. -a adds nothing to it. Add -x to have the tool name the file it chose and every changelog it applied. A gap in the changelog numbering stops the replay, and the tool reports which entries are missing. A version that does not match stops the replay the same way.

5. Verify: while the run loads the dump, it prints the counts of inodes, directory inodes, file inodes, and chunks. The run leaves a metadata.mfs behind and consumes the copy that it read. The metadata.mfs.back that you checked in step 3 no longer exists afterwards, so a second run of this proof needs the file copied over again. Read the header of the resulting file once more with mfsmetadump -s HEAD. The metadata version that the header reports is the point that the copy reaches. Compare that version with the metadata version column of mfscli -SIM on the live cluster to see how much of the recent past the copy is missing.

This procedure does not check the copy against a checksum, and no tool can. The Master Server writes metadata.crc for comparing the save of one Master Server with the save of a second Master Server. The checksums that a running cluster reports describe the save that its Master Server performed, not the file that you are holding. You can prove that a copy has a valid format, an intact end marker, a metadata version, and a metadata ID. You cannot prove anything else about the copy.

warning

Do not test a copy by starting a Master Server on it and expecting the cluster to come up.

In MooseFS PRO the instance will not reach the Leader state. The list of Chunkservers is part of the metadata, so a restored copy comes with every Chunkserver of the original cluster already on it and every one of them disconnected. The Leader role requires more than half of the registered Chunkservers to be connected. A Chunkserver that was never in that list cannot help either: an instance still electing refuses a server it does not know. The restore action above avoids the whole question, which is why it is the step this procedure uses.

The second reason is worse and applies to both editions. The copy carries the original instance's metadata ID. Every component uses that identifier to tell one instance from another. A Chunkserver of the production cluster that reaches this Master Server finds nothing wrong and registers with it. Keep a restored copy where the production components cannot resolve or reach it.

A patch file carries changes between two snapshots

The patch tools PRO (PRO Edition only) back up a directory incrementally: mfscreatepatch, mfsapplypatch, and mfsdumppatch. mfspatch(1) describes them together. A patch is the set of operations that turns the content of one snapshot of the directory into the content of a newer snapshot. The operations create and delete objects, write data, and change attributes. You apply a patch where the older content is already stored, on this instance or on another MooseFS instance. The patch brings the backup to the newer state and contains only the difference. Without -n, the target has to be on a MooseFS mount. mfsapplypatch opens a Master Server connection on the path that it is given. The tool refuses any other path with path <path> is not on mfs, and neither -f nor -d overrides this refusal. Since 5.2.0, -n applies the patch to a directory on another filesystem. The tool then skips extended attributes, ACLs, and MooseFS-specific attributes and says so with additional attributes,xattrs and facls will not be applied. The data, ownership, mode, and times are restored, but the tree is not a faithful copy. With -n, a target on a MooseFS mount is refused with option '-n' can't be used on MooseFS. The tools use snapshot mechanics to find the smallest set of operations. When the tools run on directories that are not snapshots, the patch contains the whole content of the current directory.

mfscreatepatch -c -p /backup/patches/day.patch /mnt/mfs/snap-yesterday /mnt/mfs/snap-today
mfsapplypatch -p /backup/patches/day.patch /mnt/mfs-backup/tree

-c compares differing chunks and puts only their changed parts into the patch instead of whole chunks. -m N creates the patch with N extra threads and splits it into N+1 files. -m requires -p. mfsapplypatch recognizes the multi-file format on its own. Without -p, the patch goes to standard output, so you can also connect the two commands with a pipe. You can see the operations of a patch before you apply it:

mfsdumppatch -p /backup/patches/day.patch

lists the operations without the written data. Since 5.2.0, the listing also leaves out changes to extended attributes, ACLs, and MooseFS attributes: such a change shows only as a position: line, although mfsapplypatch applies it. To restart an interrupted application from the beginning, use -i. To resume it from a printed position, use -s. In a multi-file patch, -s positions the main file only, so an application that stopped in a side file has to be restarted with -i. mfspatch(1) describes both.

That listing is a preview of the patch file, not a check on the backup. The listing reads the patch alone and never touches the copy. To check the result, read the report that each tool prints when it finishes. When mfscreatepatch writes to a file, it prints patch file has been created successfully. With -m, it prints all patch files have been created successfully. mfsapplypatch prints patch <patch file> has been applied (<n> bytes) when it reaches the end of the patch. A multi-file patch gets one such line per file. After the last file, the tool prints all patches have been applied successfully. An application that stops early reports its position instead, in error on patch file <patch file>, at position: <n>.

Runtime options reload, startup options restart

Configuration changes reach a running Master Server through the reload action of mfsmaster:

mfsmaster reload

which sends the process the HUP signal. Sending the signal directly does the same. mfsmaster.cfg(5) divides its options into startup options and runtime options. A change to a startup option requires a process restart. A reload applies a change to a runtime option. The reference page uses the same division (startup options, runtime options). A restart is a stop and a start, and starting and stopping covers both.

To verify, read the system log of that Master Server. The reload action reports reload signal has been sent. The running process answers with reloading config files as it re-reads the files. The second line shows that the change reached the process and not only the command. If the reload cannot parse a line, it reports the line in the same place, as bad definition in config file '<path>': <line>, and skips the line. The logs page describes where these messages go and how a message is built.

The supervisor shows every Master Server's state

In an installation with several Master Servers PRO (PRO Edition only), mfssupervisor run without an action displays the current state of each server. mfscli -SIM shows the same states as a table. The page leader election, quorum, and the role of DNS describes the states and the election that assigns them.

Leadership moves by order, but through the election

To make a chosen Master Server PRO (PRO Edition only) the Leader, name it by IP address:

mfssupervisor -l 192.0.2.11

The order does not put the server into the LEADER state directly. The order sets USURPER on that server. The current Leader steps down because a USURPER exists. The next pass of the election then elects the chosen server (the supervisor forces a state, not a majority). The order is refused while the group is not stable: when there is no single Leader or Deputy, or when an ELECT or another USURPER is present. The address is checked too. The address has to name a Master Server that the group name resolves to. That server has to report FOLLOWER, be synchronized, and hold metadata of its own. An address that names anything else is ignored, not refused. The tool answers Given usurper ip (<address>) is not valid (this is not a synchronized follower), changes nothing, and still exits zero. Only the current Leader closes connections: it drops its Chunkservers, its clients, and its Metaloggers as it leaves the role. The server taking over passes through USURPER and ELECT into LEADER without a disconnection of its own.

A server with KEEP_LEADERSHIP set in mfsmaster.cfg takes the role back on its own after losing it to a malfunction or a restart. The server does not undo your decision while it keeps running. The order that moved the server into FOLLOWER sets a flag, and that flag blocks KEEP_LEADERSHIP. A restart of the process clears that flag. The next restart of the machine that you demoted, including a restart for an upgrade, therefore gives the role back to that machine.

To verify, run mfssupervisor again: the chosen server reports LEADER.

A Master Server leaves the group through DNS

No command removes a Master Server from an installation the way mfscli -CRC removes a Chunkserver, and none is needed. The installation keeps no list of Master Servers to remove a server from. Every module resolves the name that the installation is configured with, mfsmaster unless it was changed. The module then works with all the addresses that the name resolves to (DNS carries the list of Master Servers, and nothing else does). A Master Server PRO (PRO Edition only) leaves when you stop its mfsmaster process and take its address out of that name.

Leaving the address in place after the machine is removed has three costs, and none of them is data. The installation raises a permanent alert that a Follower is unreachable. The alert does not stop. The consistency check that the Leader runs against the other modules keeps reporting a server that it cannot reach. And the election loses one fallback. When no synchronized Follower is left, the election can still pick a Follower that is not synchronized. It does so only when every address that the name resolves to answers and none of them is synchronized. A machine that never answers still counts in the list that the name gives, and never among the servers that answered. The election therefore picks no candidate on its own for as long as the record is there. mfssupervisor -r -f still resolves such an instance by hand, because a forced run does not wait for every address to answer.

The departure does not change the majority that leader election requires. That majority is counted in Chunkservers, not in Master Servers. A location votes when more than half of its own Chunkservers are connected. The instance follows more than half of the voting locations. Removing a Master Server changes none of those numbers. Disks and Chunkservers describes the counts that do change.

To verify, run mfssupervisor with no action, or read mfscli -SIM. The departed server no longer appears in the listing. The remaining servers report their states, and none of them is unreachable.

Every copy of mfsmaster.cfg has to agree

Each Master Server PRO (PRO Edition only) reads its own copy of mfsmaster.cfg, and the copies have to agree. You are responsible for keeping them the same. The consequences of a difference between the copies are described on the reference page for the file. The product checks and reports, but it does not repair. The Leader tests the modules for configuration differences every CONFIGURATION_TEST_PERIOD hours, and

mfssupervisor -c

runs a consistency check on demand.

The check runs on the Leader, or on the Deputy when the group has one, and not on the machine that the order was sent from. The tool names the server that received the order: consistency check packet has been sent to LEADER: (<address>:<port>), with DEPUTY in place of LEADER for a Deputy. The findings of the check go to the system log of that server. To verify, read that log. An option that is set on one server and absent on another server appears as master <address>: option <name> is not defined. An option with different values appears as master <address>: option <name> has different value (my value: … ; remote value: …). A differing file that an option points at appears as master <address>: cfg file pointed by option <name> is different from local file. If no such line appears, the copies agree.

A Follower joins a running group empty

To add a Master Server PRO (PRO Edition only) to a group that is already serving:

1. Install the Master Server on the new machine (install server components).
2. Give it a copy of mfsmaster.cfg that agrees with the rest of the group.
3. Add the machine's address to the DNS name by which the installation resolves the group (DNS carries the list of Master Servers). The records are part of preparing the operating system.
4. Start the server without metadata:

mfsmaster -e start

A server that is started empty can only be a Follower. It registers with the Leader and receives a copy of the metadata over the Leader-Follower synchronization.

To verify, list the group with mfscli -SIM. The new server appears as a FOLLOWER. While the new server is being filled, its state has a suffix such as (INIT) or (DESYNC). The suffix disappears after the server's copy has caught up.

Replacing the machine is an arrival and a departure

Moving a Master Server PRO (PRO Edition only) to different hardware is two operations, and the sections above already describe both. You run them in an order that keeps the group whole throughout. The new machine joins as a Follower. The old machine leaves only after the new machine is synchronized. Nothing is migrated between the machines, because the new machine gets the metadata from the Leader rather than from the old machine. This procedure is for a group. An installation with a single Master Server has nothing to join and no overlap to arrange. To replace its machine, do a restore from a copy of the metadata onto the new hardware.

1. Install the Master Server on the new machine and give it a copy of mfsmaster.cfg that agrees with the rest of the group (a Follower joins a running group empty describes both steps).
2. Copy the license file to the path that LICENSE_FILENAME names on this machine. Every other Master Server runs under the same file. The file does not have to be reissued for the new hardware. A license binds to the installation, not to the machine. The new machine is joining that installation.
3. Add the new machine's address to the DNS name by which the group is resolved, and leave the old machine's address in place.
4. Start the new server empty with mfsmaster -e start. Wait until mfscli -SIM reports the server as FOLLOWER with no suffix on the state. As long as the suffix is shown, the server's copy of the metadata is still arriving, and the server cannot replace another Master Server.
5. If the machine you are replacing holds the LEADER state, move the role to another server first with mfssupervisor -l (leadership moves by order). If you replace a Follower, skip this step.
6. Stop the mfsmaster process on the old machine and remove its address from the DNS name, as in a Master Server leaves the group through DNS.

To verify, run mfssupervisor with no action. The group has the same number of servers as before the work started. The new machine is among them, and the old machine is not. One server reports LEADER. The other servers report FOLLOWER with no suffix. Confirm the license as well. When you run mfslicinfo against the new machine, it reports the same license as the other machines do. This result shows that the file arrived and was accepted.

warning

Do not copy the old machine's data directory onto the new one to save the synchronization. The empty start is safe because a Master Server with no metadata cannot stand for election until the Leader has filled it. A server that comes up on a copied directory has a metadata version above zero from its first moment, and that is the whole of the test the promotion makes: it does not ask whether the copy is current. The wait is the safeguard, and it costs nothing else.