Master Server problems

The Master Server reports each failure in a message. What you do next depends on that message. Most messages quoted on this page are lines that the Master Server writes to the system log. In a foreground start (mfsmaster -f start), the same lines also appear in the terminal. The page names the other messages where they appear. They are the lines this page calls terminal output, the text the program prints about its own options, and the answer of the retired mfsmetarestore command. None of those messages appear in the log. Read the log before you touch the data directory.

A refused start names its reason

A plain start of mfsmaster reads one file, metadata.mfs. A clean stop writes that file. The next start renames it, so after anything other than a clean stop the file is normally missing (what a startup loads).

can't find metadata.mfs - try using option '-a'
error reading metadata.mfs - try using option '-a': EACCES (Permission denied)
metadata.mfs has wrong header - try using option '-a'
metadata.mfs has wrong ending - try using option '-a'
error loading metadata.mfs - try using option '-a'

The first line appears after a killed process, a power cut, or a stop that did not finish. The second is the only one of the five that ends with the system error that stopped the read, whatever that error was. A wrong ending means that the file has no end marker, so the write that produced it was cut short. A wrong header means that the file is not a metadata dump at all. The last line reports a file that passed both of those checks and failed partway through the load. All five lead to the automatic restore.

Three further conditions protect metadata the Master Server can see but was not told to load. Each is reported twice, once for a metadata.mfs.back written by a Master Server and once for a metadata_ml.mfs.back copied in from a Metalogger:

backup file exists but current metadata file is empty - please check it manually - try using option '-a'
backup file is newer than current file - please check it manually - try using option '-a'
backup file has different file id - please check it manually - try using option '-a' and '-i'
metalogger file exists but current metadata file is empty - please check it manually - try using option '-a'
metalogger file is newer than current file - please check it manually - try using option '-a'
metalogger file has different file id - please check it manually - try using option '-a' and '-i'

A metadata.mfs.back, or a metadata_ml.mfs.back copied into the same directory, has a higher metadata version than metadata.mfs does. A start on the older file would leave the changes between the two files out of the metadata the cluster serves. The Master Server therefore stops and leaves the decision to you. mfsmetadump with -s HEAD prints the metadata version each file holds. Restore from the newer file.

found metadata file with different id number - cleanup your working directory or use '-i' flag (might be dangerous without cleaning)

The directory holds dumps from two different sets of metadata. The metadata ID tells them apart. Move the files that do not belong to this installation out of the directory. The -i flag that the message offers does not separate them: it replaces the refusal with a note in the log. The flag then lets the scan continue and take the highest metadata version it finds, whichever set that file came from.

can't find valid metadata file

No file that the automatic restore scanned passed the check, so this machine has nothing to start from. The only option left is a copy from elsewhere.

Reasons unrelated to the content of the metadata:

can't start: lockfile is already locked by another process
init: metadata manager failed !!!
error occurred during initialization - exiting
error reading metadata (node) - metadata in file have been stored by newer version of MFS !!!

A locked lock file means that a Master Server is running already. mfsmaster test prints the ID of the process that holds it. An init: line names the module that failed to start (metadata manager, exports manager, change log, and the rest). The process then exits. A dump written by a newer MooseFS version is refused section by section: run the version that wrote the file, or a newer version. In an installation with a license file PRO (PRO Edition only), a metadata ID in the license that differs from the ID in the dump stops the load with wrong meta id - can't load metadata.

The damage's depth decides the way out

Metadata that is wrong is not one condition. The location of the damage decides both what reveals it and how to recover.

• Data inside a chunk. The metadata is intact, but one copy of the data is not, so this is not a Master Server failure. Every block has a checksum, which is verified on each read and write and by the scan of stored chunks (scrubbing and CRC). The repair is done on the Chunkserver that holds the chunk (Chunkserver and disk recovery).
• An identifier inside the metadata, such as the chunk ID in a file's chunk table or the inode number that a directory entry points to. The damage stays hidden while the Master Server answers from RAM. The damage appears as soon as a second copy of the metadata is built from the changelog, because every changelog line carries the identifier the operation produced and the replay compares that identifier with its own. A restore stops on a line ending in error: 32 (Data mismatch). A Follower Master Server that refuses a change enters DESYNC PRO (PRO Edition only) and receives a whole dump instead. The line itself cannot be repaired: recovery uses the newest copy that replays without that line.
• The structure inside a dump: an entry pointing at an inode that does not exist, an entry with an empty name, an inode no directory points at. The load reports each case and names the option that would let the load continue. That option is -i. The effect is fixed per case. An empty name is replaced with a generated name. An entry pointing nowhere is dropped. An entry whose parent is missing or is not a directory is attached to the root directory. An inode with no parent at all becomes lost_node_<inode> in the root directory, so inode 5566 appears under the name lost_node_5566, with found orphaned inode: in the log. If the root directory already has that name, the load appends a number: lost_node_5566.1, lost_node_5566.2, and up. A dump with no root directory ends the load regardless of the options: root node not found !!! on the terminal, error reading metadata (no root) in the log.
• The attributes of an object: owner, group, mode, timestamps. Nothing inside a single copy contradicts them. In an installation with several Master Servers, the Leader compares checksums with the Followers section by section, on the METADATA_CRCCHECK_FREQ schedule. The Leader names the section that differs (the checksum is computed per section). A Community Edition cluster holds no second copy of the metadata to compare against, so damage at this depth is not reported there.

Recovery from a copy takes three things

A Metalogger keeps the second copy of the metadata outside the Master Server's machine: a downloaded dump plus the changelog lines as they were written (the Metalogger stream). The Metalogger never takes over the Master Server's role (it is not a server). Recovery therefore means putting a Master Server where the Metalogger's files are, or putting those files where a Master Server is.

Take these files from the Metalogger's data directory: metadata_ml.mfs.back and its older generations metadata_ml.mfs.back.1 and up, the changelogs changelog_ml.0.mfs and up, and changelog_ml_back.0.mfs with changelog_ml_back.1.mfs. A downloaded dump is renamed to metadata_ml.mfs.back only after the Metalogger has checked the dump's signature and end marker, so a file under that name is complete.

To take over the role on the Metalogger's own machine:

1. Stop the Metalogger. The process keeps its files open.
2. Install the Master Server (install server components) and put mfsmaster.cfg, mfsexports.cfg, and mfstopology.cfg in place.
3. If the Master Server's DATA_PATH is not the directory the Metalogger wrote to, copy the files into it. Leave the names as they are: the scan uses every file whose name begins with metadata. The scan also reads changelogs under the Metalogger's names, next to the Master Server's own.
4. Start with mfsmaster -a -x start and read which file it chose.
5. Point the installation at this machine. Either change the address behind the mfsmaster name in DNS, or give the machine the address the old Master Server had. With DNS, the record's TTL sets how long the old answer stays in caches.

When the Master Server's machine is alive and the metadata on it is not, the same files are copied in the opposite direction. Stop the Master Server, move the content of its data directory aside, copy in the Metalogger's newest dump and every one of its changelog files, and start with -a.

warning

A start rearranges the data directory. A successful restore deletes an empty metadata.mfs, renames a non-empty metadata.mfs to metadata.mfs.back or to a temporary name, and removes the temporary file left by an interrupted save. Copy the whole directory somewhere else before the first attempt, or the state you started from is gone.

mfsmetarestore is a message, not a tool

Older instructions refer you to mfsmetarestore. The packages install a file under that name, so the command answers instead of reporting that it is missing. The command prints only this message:

mfsmetarestore has been removed in version 1.7, use mfsmaster -a instead

The manual page mfsmetarestore(8) says the same. The command reference lists the name among the replaced commands. mfsmaster replaces mfsmetarestore, in two forms: the automatic restore option, which rebuilds the metadata and starts the server on it, and the restore action, which rebuilds the file and starts nothing.

The restore action writes metadata without starting

mfsmaster restore runs the same scan and the same merge as the automatic restore option, writes the result into the data directory as metadata.mfs, and exits. It runs without the communication modules, so it opens no sockets. Nothing can connect to it. Use it when you need the file itself: to copy to another machine, to keep as a backup, or to read with mfsmetadump. To start a server again, use the automatic restore option. It is faster because it does not wait for the file to be written first.

warning

Run restore only where no Master Server is running. Since 5.2.0, restore refuses to run next to a running process: it logs can't perform restore: master process is running and exits with status 1, and the running process keeps working. Up to 5.1.0, restore takes the lock file over: it sends that process a termination signal and waits for the process to exit, which stops the cluster.

The automatic restore option rebuilds from what remains

mfsmaster -a start replaces the single-file start with a scan of the data directory and a replay of the changelogs over what that scan chose (what a startup loads). Once the automatic restore holds a dump with a valid metadata ID, it also reads the emergency locations. The automatic restore takes a file from there when that file belongs to the same ID and has a higher version.

By default, the log does not say which file the automatic restore chose. With verbose output, the log has a line that begins with chosen most recent metadata file: and names the file, its version, and its ID:

mfsmaster -a -x start

-xx also lists every metadata file the scan found, both the valid files and the rejected files. The merge reports its own progress on standard output, which is visible in a foreground start (mfsmaster -a -x -f start).

The merge stops rather than skipping what it cannot apply. A missing range of changes ends the merge with a line beginning hole in change files. The line names the first and the last missing change. The line also names the two files on either side of the gap: the file that ends before the gap and the file that begins after it. A change that does not match the replay ends the merge with error: 32 (Data mismatch). In both cases, the start ends with this line:

error applying changelogs - fix changelogs manually or use '-i' flag

When changelogs are missing, add files from a Metalogger or from a backup instead of forcing the start. -i starts the Master Server over a gap without reporting the gap. The log does not name the changes that were skipped. Later messages do not say that the metadata is missing any changes. Read the gap from the failed start before you force the start that hides it.

The option does not do the following:

• It fetches nothing. It reads the data directory and the emergency locations, so a Metalogger's files have to be copied in first.
• It does not repair the structure inside a dump.
• It does not restore file data. Metadata does not contain the content of chunks.
• It does not read anything from other machines. It takes the newest metadata on this machine, which is not always the newest in the installation.

-i is the last resort. The program says so itself:

-i : ignore some metadata structure errors (attach orphans to root, ignore names without inode, etc.). DO NOT USE unless you are absolutely sure that there are no other options to restore your metadata.

warning

-i alters the metadata to make it loadable: entries are dropped, names are generated, and orphans are attached to the root directory. The next save writes that result over the dumps in the data directory. Copy the data directory elsewhere before starting with -i.

If you need -a at all, the last stop was not clean. That makes it a recovery tool, not a start flag. Starting and stopping covers what that means for a service unit.

Memory exhaustion ends the Master Server, no dump

The Master Server holds the whole metadata in RAM and answers every request from it (the Master Server). When memory runs short, this line appears in the log before the process ends:

fork error (store data in foreground - it will block master for a while - check /proc/sys/vm/overcommit_memory and if necessary set to 1)

A save begins by forking a second process. When the fork fails, the Master Server writes the dump itself and answers nothing until the dump is written. This is the foreground save that the cluster reports as a warning (save states, alerts). An allocation that fails outright ends the process: the log line contains out of memory: and the name of the structure that could not be allocated. Nothing is saved after that failure. The Master Server also asks the kernel not to kill the process when memory runs out, and reports the result of the request:

out of memory killer disabled
can't disable out of memory killer

A Master Server that was killed, or that ended on a failed allocation, wrote no dump. The next start therefore needs the automatic restore option, the same as after any unclean stop.

Do not start the Master Server again before the cause is removed, because the process will fail at the same point. If the cluster must be available again before you can add memory, swap lets the process start. Use swap only to recover, not to run the cluster. With swap, the single thread that serves every client answers at the speed of a disk. Connect one client, reduce the metadata, and restart. To reduce the metadata, reduce the number of records: delete what is not needed, empty the trash, remove snapshots that have served their purpose with mfsrmsnapshot, and drop extended attributes that nothing reads. The memory that the object count requires can be calculated, not guessed: Master Server RAM from the object count calculates it. Master Servers describes a machine that matches the result.

A full data directory stops the Master Server deliberately

When a save cannot be written to the data directory, the Master Server writes an emergency copy named metadata.mfs.emergency. The copy goes to the first of the locations that accepts it (a Master Server that cannot write metadata stops). A line beginning metadata file stored in emergency mode, file name: names the file the Master Server wrote. Then the Master Server starts to exit, whether that copy succeeded or not:

metadata stored in emergency mode (in non-standard location) - exiting
metadata not stored !!! (child exited) - exiting

Rescue the metadata in this order:

1. Read the log to find where the emergency copy was written.

2. Free space in the data directory while the process is still exiting. The exit is not instant: the Master Server retries the ordinary save every ten seconds and exits only after one of those retries succeeds. If you free space in that time, a retry writes a proper metadata.mfs, and the process then exits. If the directory stays full, the process stays in that loop.

   can't store metadata - try to make more space on your hdd or change privileges - retrying after 10 seconds

3. If the process was killed before a retry succeeded, copy the emergency file into the data directory. A process that exited on its own has written a proper metadata.mfs and needs no emergency copy. Its name begins with metadata, so the scan uses it as a candidate. A file left in /tmp or another of the fixed locations is read only when the data directory already holds a valid dump of the same metadata set.

4. Start with mfsmaster -a -x start, and read the line naming the file it chose.

warning

Do not make room by deleting the dumps or the changelogs in the data directory. A restore is built from them. The changelogs have to cover the whole time since the oldest dump you would fall back on (changelog retention).

The same exit follows when a changelog line cannot be appended, with changelog lost !!! and problems with data write subprocess detected - changelog line lost - force termination in the log.

The cluster raises no alert on the free space of a Master Server's disk, so the failed save is the first sign of a full disk. Plan the size of the directory in advance: the disk holds metadata images and changelogs. Around this failure the cluster reports a save made in the foreground and a save that took longer than 45 minutes (alerts).

No Leader, an unsettled election, or two Leaders

An installation with several Master Servers PRO (PRO Edition only) decides the role by election. The conditions of that election are outside the Master Servers themselves. Leader election, quorum, and the role of DNS describes the mechanism. Leader election and quorum defines the terms. The rest of this section names what to check when the outcome is wrong.

No Leader, or a candidate that stays a candidate. A candidate becomes the Leader only with the support of a majority of the Chunkservers, counted location by location. Count the connected Chunkservers and check which locations they are in before you examine the Master Servers at all: the supervisor forces a state and cannot create a majority. A Leader that loses the majority steps down on its own:

internal state change LEADER->FOLLOWER(SYNCED)

A Master Server missing from the group. Each part of the installation finds the group by resolving one name. That list comes from DNS alone. An answer that differs between machines produces a group that differs between machines. Resolve the name from every machine and compare the answers.

Clocks that disagree. The supervisor names the pair and the difference in detected time desync between leader and master and detected time desync between masters, starting at one second apart. A Follower that receives changes stamped ahead of its own clock reports got change log from the future. Synchronize the clocks in the cluster.

Two Leaders, or a Leader beside a candidate. A supervisor pass reports what it found and sets the extra servers back to Follower. The pass keeps the server with the highest metadata version:

Found more than one LEADER !!!

The condition itself is limited. A Master Server drops every Chunkserver, client, and Metalogger connection before its new role takes effect, so two servers do not answer clients as Leader at the same time. A log that contains dangerous option in use (the leader will stay as a leader even when more than half chunkservers disconnect) says that this protection was turned off on that server by hand.

Metadata from two different histories. The supervisor compares the metadata IDs of all Master Servers, reports meta id conflict found !!!, and refuses to act on such a group unless it is forced. Treat the refusal as the finding: decide which ID is correct first. A Follower with a foreign ID is refused at registration, with a line that begins with MATOAN_MASTER_ACK - wrong meta id, which names both IDs and ends with you can start master with '-e' option, to force metadata synchronization. That option starts a Master Server with no metadata of its own, so it receives the Leader's whole copy.

Nobody can be elected without going backwards. A Chunkserver stores the last metadata version that a Leader reported to it. A candidate below that version is refused with a line beginning can't choose new ELECT, best FOLLOWER version:. Use the Master Server that holds the newer metadata. Do not force an election.

A Leader with no second copy. Once an hour, the Leader reports which servers back up its metadata. The line no meta backup servers connected !!! - deploy backup master servers means that no such server is connected.

A power failure leaves the Master Server down

A power cut is not a clean stop, so metadata.mfs is missing. A plain start then refuses to run. An enabled service unit changes nothing about that: it runs the same plain start at every boot and fails the same way each time. Why the Master Server stays down on purpose covers what that gives you and why -a does not belong in the unit file.

The unit that the packages install follows this design. ExecStart runs a plain mfsmaster start. Restart=no keeps systemd from repeating the attempt. Its stop and start timeouts are 1800 seconds, the default time mfsmaster waits for the lock file to be released. The FreeBSD startup script has a safeguard of its own: it is installed with the service disabled and skipped at boot. The systemd unit has no such safeguard, so on Linux you alone decide whether to enable it. Let a person who has checked the metadata start the Master Server.