Chunkserver and disk recovery

A disk starts reporting errors, a Chunkserver does not start, or one file returns a read error but everything around it is fine. This page starts from each of those symptoms. For each one, the page leads either to a whole cluster or to a clear statement of what cannot be recovered. Planned work on hardware that still runs is described in disks and Chunkservers. Disk error handling and disk removal covers what the Chunkserver counts before it marks a disk as damaged and what the cluster does with the copies afterward.

Read the disk's status before touching hardware

The Disks tab of the GUI and mfscli with -SHD on the command line show the same two columns for every configured path. The column last error is the time of the newest failure that the disk recorded. The other column is status. The status decides which procedure applies.

status	What the Chunkserver is saying	What is left to do
ok (errors)	The disk is in service. Failures are on its record. The tolerance has not been exceeded.	Nothing is lost yet. This is a retirement rather than a recovery: mark the disk for removal and replace it as in disks and Chunkservers.
damaged	The disk is out of use. Every chunk it held has been reported to the Master Server as lost. The repairs are already queued.	Replace it.
invalid	There is a line in mfshdd.cfg with nothing behind it: the mount point is absent, the lock file cannot be written, another process holds the lock, or the disk carries the identifier of a different MooseFS instance.	Find out which of those it is before assuming the hardware failed.

Confirm the cluster no longer needs the disk

The Master Server removed the copies on a damaged disk from its record as soon as the disk was marked as damaged. The chunks that are short of copies are therefore being rebuilt from copies elsewhere. The chunk matrix table on the Info tab, or mfscli -SIC, shows when the rebuild is done: wait until endangered and undergoal are zero. Replication priorities covers the order in which that queue is processed and what limits its speed.

Chunks that stay under missing have no copy anywhere in the cluster. Waiting does not produce a copy. Two things can still help them: the disk in your hand, which holds their files untouched, and the per-file repair described under rescue a file whose chunk has no copy left. Keep the disk until the count stops changing.

Replace a disk the Chunkserver has already dropped

1. Delete the disk's line from mfshdd.cfg, or comment it out with #, and reload the Chunkserver:

   mfschunkserver reload

   The * prefix is not used here. The prefix exists to make the cluster independent of a disk that still works. The cluster stopped relying on a dropped disk when the disk was marked as damaged.

2. Verify in the system log: the reload writes hdd space manager: folder /mnt/hdd1/ will be removed. The line folder /mnt/hdd1/ successfully removed follows once the last chunk on the disk is released. The Chunkserver appends the trailing slash to every path it reads from mfshdd.cfg, so the log shows the slash whether or not the file has it. A chunk in use delays the removal. The removal attempt repeats until the chunk is released. The path also disappears from the Disks tab.

3. Replace the hardware. Format and mount the new disk, append its path to mfshdd.cfg, and reload again.

4. Verify: the new path appears on the Disks tab, scanning first and then in service. Its capacity is added to the server's totals. The disk fills without further commands. Rebalancing covers how that works and how to speed it up.

warning

Taking a line out of mfshdd.cfg is not a way to retire a disk that still works. The Chunkserver reports every chunk on a departing disk to the Master Server as lost, whether the disk was damaged or healthy, so a line removed before the disk reports removal ready costs exactly the redundancy a failure would have cost.

Check a detached disk before putting it back

A power failure or a file system repair can leave chunk files whose stored checksums no longer match the data in them, on a disk that is otherwise sound. mfschunktool reads those files directly, with no Chunkserver involved, and sorts them into the files that pass and the files that do not.

The disk must not be in service. When the tool is pointed at a storage directory that a running Chunkserver has locked, it stops with found active lock file: /mnt/hdd1/.lock - there is a working chunkserver using this disk - skip scanning. The tool does not detect that lock when it is pointed at a single chunk file or at one of the 00 to FF subdirectories, so the protection does not apply. Take the disk's line out of mfshdd.cfg and reload, or stop the process, before you start. In the commands below, replace /mnt/hdd1 with your disk's mount point and /var/tmp/mfs-damaged with an existing directory of your choice.

1. Run the file system checker for the file system in question, fsck or xfs_repair, and deal with what it reports.

2. Survey the damage without writing anything:

   mfschunktool -f -x /mnt/hdd1

   -f reads the header and the checksum of the last data block of each chunk rather than the checksums of all blocks. -x prints OK for the files that pass, so this run shows the scale of the problem before you decide what to do.

3. Move the failing chunks aside, and put the disk back with the chunks that pass:

   mfschunktool -m /var/tmp/mfs-damaged /mnt/hdd1

   The directory named after -m has to exist already: the tool does not create it and stops with /var/tmp/mfs-damaged: realpath error when the directory is missing. The tool does not write to any chunk file. The cluster rebuilds what is now absent, from copies that were never damaged. The files you set aside are still there if it turns out that no such copy exists.

4. Repair in place only when disks failed on several Chunkservers at once and every copy of some chunk may be affected:

   mfschunktool -r -m /var/tmp/mfs-damaged /mnt/hdd1

   Repair recalculates the stored checksums from the data in the file and fixes header fields from the file name. Files that it cannot fix are moved aside instead. A disk outside the cluster covers which files each of the two forms moves and why setting them aside is the safer default.

A file system repair that moves chunk files into lost+found removes their names. The tool can restore the names:

mfschunktool -n /mnt/hdd1/lost+found

The tool reads the chunk ID and version from each file's header and renames the file to the form the Chunkserver expects. A file whose header is unreadable as well is reported and left unchanged.

When the disk goes back into service, verify: its path appears on the Disks tab again, scanning first and then in service. The chunks whose files were set aside are rebuilt from copies elsewhere: watch the chunk matrix table, or mfscli -SIC, until endangered and undergoal are zero.

Chunk files you set aside can be returned to the disk later. Any of the 00 to FF subdirectories works: the Chunkserver takes the ID and the version from the file name and does not check which directory the file is in. The layout itself is described on the Chunkserver page.

warning

A chunk file returned at a version no higher than a copy the same Chunkserver already holds does not replace it. The Chunkserver queues the duplicate for deletion and keeps it for HDD_KEEP_DUPLICATES_HOURS in mfschunkserver.cfg, one week by default. Once an hour the Chunkserver logs on drive '/mnt/hdd1' 12 chunk duplicates detected - will remove them in about 160 hours - no rebalance will occur on this drive until duplicate removal. The bar is narrower than the message's wording: for the whole period the disk cannot be an internal rebalance destination, though it can still serve as a source.

Deleting .chunkdb from the disk by hand is not needed in any of these procedures. The Chunkserver discards that index on its own whenever a chunk directory is newer than the file. Offline work on the disk makes the chunk directories newer than the file. The Chunkserver deletes the file once it has read it.

Trace a failed start to its first complaint

A Chunkserver reports to the system log of its own machine and opens no log file (logs). A start that fails in one of its modules ends with error occurred during initialization - exiting. The lines above it name the module that failed. The modules start in a fixed order, so act on the first error. The first two rows of the table below are the exception. Both happen before any module starts, so the process prints that one line and exits without the closing line.

Message	What is wrong
can't start: lockfile is already locked by another process	The process is already running.
can't set working directory to /var/lib/mfs	The data directory is absent, or the account the process runs as cannot enter it.
hdd space configuration file (/etc/mfs/mfshdd.cfg) not found	The disk list is not where the process looks for it.
hdd space manager: can't create lock file '/mnt/hdd1/.lock'	A configured path is not mounted, is not writable, or is out of reach of the account. The system error at the end of the line separates the three.
hdd space manager: data folder '/mnt/hdd1' already locked (used by another process)	Another Chunkserver on the machine lists the same path.
hdd space manager: data folders '/mnt/hdd1/' and '/mnt/hdd2/' have the same lockfile !!!	Two lines reach one lock file. That entry is refused and the rest of the file is read.
main server module: can't listen on *:9422	The data port, 9422 by default, is taken.
main server module: can't resolve mfscs1:9422	CSSERV_LISTEN_HOST names something the resolver does not answer for.

The lock file message is the common case after a disk failure: the disk is missing and its line is still in mfshdd.cfg. A process that ran without trouble until the reboot will not start again until the line is removed. ALLOW_STARTING_WITH_INVALID_DISKS = 1 in mfschunkserver.cfg lets it start anyway, with that line reported as invalid. The default is 0. A reload is not affected either way, because a reload reads the file as it is.

An empty mfshdd.cfg is not one of these cases. The Chunkserver starts and records that no hdd space is defined in the file.

Trace a start that never registers

The process is running. The Chunkservers tab does not list it. Connecting to the Master Server is a separate step from starting. The log reports the connection separately. Check first that the process really is running: three of the cases below end the start instead of leaving the process running. On the Chunkserver, a healthy connection logs connecting ... and then connected to Master. The Master Server writes a chunkserver 192.0.2.21:9422 register begin line with the server's space figures, and then chunkserver 192.0.2.21:9422 register end.

Message	Where it appears	What is wrong
master connection module: can't resolve master host/port (mfsmaster:9420)	Chunkserver	MASTER_HOST names something the resolver does not answer for. At a start this ends the start: the module reports init: master connection module failed !!! and the process exits.
master connection module: localhost (127.0.0.1) can't be used for connecting with master (use ip address of network controller)	Chunkserver	MASTER_HOST resolves to a loopback address. The start ends the same way as the line above. The Master Server refuses the same case from its side, with a message of its own about localhost.
master connection module: connect failed	Chunkserver	Nothing answers on the master's port: the Master Server is down, or a firewall blocks it.
MATOCS_MASTER_ACK - master needs authorization, but password was not defined	Chunkserver	The Master Server asks for a password and AUTH_CODE is not set. A wrong password is refused on the Master Server's side with CSTOMA_REGISTER (BEGIN,192.0.2.21): access denied - check password.
MATOCS_MASTER_ACK - wrong meta data id (files .metaid:... ; received from master:...). Can't connect to master	Chunkserver	The disks belong to a different MooseFS instance. The process exits, because nothing about this can be settled while it runs.
csdb: found cs using ip:port (192.0.2.21:9422,3), but server is still connected with can't accept chunkserver 192.0.2.21:9422 (csdb)	Master Server	The Master Server still holds a live connection from that address and port. A second process on the same address, or a predecessor whose socket has not closed yet, produces this.
CSTOMA_REGISTER: chunkserver 192.0.2.21 is too old	Master Server	The Chunkserver speaks a protocol this Master Server no longer accepts. Variants of the line name the feature that sets the minimum instead, such as Erasure Coding or authentication tokens.

A single disk can be refused on the same grounds. A reload continues without that disk. A start does not, under the same ALLOW_STARTING_WITH_INVALID_DISKS rule as the lock file case above. A path whose .metaid file does not match the instance is ignored, with hdd space manager: wrong meta id (mfs instance id) in file '/mnt/hdd1/.metaid' ... (drive ignored). The file is written only after the Chunkserver has confirmed the identifier with the Master Server, so a mismatch means that the disk was in service somewhere else. Removing the file removes the check.

Decide whether a lost Chunkserver's repairs should run

A Chunkserver that loses power or loses the network sends no notice and does not enter maintenance mode. The Master Server treats every chunk that the Chunkserver held as lost and starts rebuilding. This is the right response when the machine or its disks will not return. The rebuild needs no decision from you.

The rebuild is the wrong response when the machine will return with its disks intact. A crash starts no waiting period at all, so the rebuild is under way before anyone has reached the machine. The rebuild produces copies that the returning server already holds. Switching maintenance mode on by hand for a server that is already disconnected suspends that work. The control is in the Maintenance column on the Chunkservers tab of the GUI, where disconnected servers have a separate table. You can also use the command line:

mfscli -H mfsmaster -CM1/192.0.2.21/9422

warning

The hold covers more than the server you are waiting for. As long as at least one Chunkserver is disconnected and every disconnected Chunkserver is in maintenance mode, the Master Server schedules no chunk repairs of its own, so a copy lost elsewhere in the meantime stays missing for as long as the hold lasts. Recovery a client is waiting for still runs. In a cluster divided into locations PRO (PRO Edition only), both that condition and the hold it produces are counted per location: repairs go on in the locations that hold no such Chunkserver. Switch the mode off, with -CM0 and the same address, the moment the server is back. Do not use maintenance mode on a machine that is gone for good.

Rescue a file whose chunk has no copy

The Missing Files table on the Info tab and mfscli -SMF name the affected paths, one row per missing chunk. Each row shows the file's inode, the chunk's position in the file, the chunk ID, and Type of missing chunk. The list is rebuilt by the file loop, so the list is as old as the last full pass of the loop (what to watch). Chunk versions and states describes the four types. mfsfilerepair can fix only one of the four.

Type of missing chunk	What mfsfilerepair can do
Wrong versions	Repair it. The Master Server accepts the highest version found on the Chunkservers and makes those copies current.
Invalid copies	Nothing. Copies exist and none of them can be read.
Missing EC parts	Nothing. Too few parts survive to reconstruct the chunk.
No copy	Nothing. There is nothing left to promote.

Work one file at a time, with the tools in this order.

1. mfscheckfile answers whether the file has a gap at all. The summary of a file with a gap includes a line for chunks with no redundancy. Diagnostic tools covers how to read the rest of the summary.

   /mnt/mfs/data/archive.tar:
   no copies and EC parts :          1
   2 full copies (redundancy level: 1):          2

2. mfsfileinfo names the chunk and says what is wrong with it.

   /mnt/mfs/data/archive.tar:
       chunk 0: 0000000000000244_00000001 / (id:580 ver:1) ; mtime:0 (old chunk - not set)
           missing data copies / data parts - unrecoverable

3. mfsfilerepair acts and reports what it did to the chunks of the file.

   /mnt/mfs/data/archive.tar:
    chunks not changed:          2
    chunks erased:               0
    chunks repaired:             1

chunks repaired counts two different outcomes without separating them. One outcome is a repair: the chunk's version was wrong and is now correct, so the data is available again. The other outcome is a mark: where no copy is available at all, the chunk is marked readable as zeros so that the rest of the file can be read. Writes to that chunk still fail. If you run the command a second time, it reports the same number, because the mark is set again from the start each time. Repeating the command makes no progress.

mfsfileinfo separates the two outcomes. A chunk that was repaired now lists its copies. A chunk that was only marked still reads missing data copies / data parts - unrecoverable, with one line added:

        chunk marked as readable after repair - reading this chunk will receive zeros

The mark is not final. If a Chunkserver that still holds that chunk reconnects, then reads return the real data again without any action from you. If you want to keep the file with its current content, copy the file rather than take a snapshot of it. A snapshot shares the same chunks. If a Chunkserver returns later with different data under the same version, the snapshot shows that data as well.

warning

mfsfilerepair -d is the second decision. It cannot be undone. The option removes the unrepairable chunks from the file for good. mfsfileinfo then reports chunk 0: empty. Reads over that range return zeros, with nothing left anywhere to restore them from. Use it when the data is known to be gone and a file that reads without error is worth more than a record of the gap.

mfsfilerepair writes, so it needs write permission on the file, an export that is not read-only, and a file that does not have the immutable attribute. Anything that is not a file is refused.

Both answers depend on a complete set of connected Chunkservers. The Master Server reports a chunk as lost, and serves a marked chunk as zeros, only once every registered Chunkserver is connected. While one of them is disconnected, the Master Server tells the mount to try again later. A read of the chunk therefore waits, however the chunk is marked, because the data may still return with the missing server. An application hangs on that read rather than receiving zeros. The wait ends when the disconnected server reconnects or leaves the list. With every server connected, a read of a chunk with no copy is retried before the read fails, so an application waits and then receives ENXIO. A mount with mfserroronlostchunk returns that error without the retries (mount options).

What not to do

• Do not move, rename, or delete chunk files under a path that a running Chunkserver has in its mfshdd.cfg. The lock that the tool checks on a whole storage directory does not exist when the tool is pointed at one file or one subdirectory. The Chunkserver registers what it finds only when it scans.
• Do not repair chunks in place when a good copy of them may exist on another Chunkserver. A repaired chunk is indistinguishable from a chunk that was never damaged. Afterward, the cluster cannot choose the intact copy over the repaired chunk.
• Do not run mfsfilerepair again on a file that reported chunks repaired and still reads as unrecoverable. The number will not change. Decide only whether to accept the zeros or to remove the chunks with -d.