Start from the symptom

Something is wrong. You have a symptom rather than a cause. This page lists which page covers each symptom. Find what you see, and go to the page that diagnoses and repairs it.

Take these readings before you change anything

Every repair in this chapter is based on the same readings. Taking them first keeps you from repairing a part that is working.

• The data: whether the cluster counts any chunk as missing, and how many it counts as endangered or undergoal.
• The metadata servers: whether the Master Server answers at all, and in MooseFS PRO PRO (PRO Edition only) whether one Master Server is the Leader and the rest follow it.
• Whether those counts are current: the cluster says when they are not. Do not decide anything from a reading taken at that moment.

What to watch and cluster health parameters describe where each of these is read and what a value other than zero means. Alerts lists every alert the cluster raises about itself, with its severity and the fix it suggests.

Find what you see, then go there

A symptom is often covered by more than one page. Where a row names several pages, they are listed in the order to work through them.

What you see	Where to go
Nothing under the mount point answers, on every client at once	Master Server problems when no Master Server answers or the cluster has no Leader PRO (PRO Edition only). Client and mount problems when the metadata side is sound
A mount does not come up, or one client hangs or crawls while the others keep working	Client and mount problems, and network problems when that client cannot reach every Chunkserver
A client is listed as inactive after losing its connection	Client and mount problems. Network problems when the same mount keeps reappearing there
The MooseFS drive on a Windows machine cannot be opened, or its letter is missing from Explorer PRO (PRO Edition only)	Windows Client problems
One file, or a few files, return an I/O error while the rest of the file system reads	Chunkserver and disk recovery for a single file. Cluster health problems when the cluster also counts chunks as missing
The cluster counts chunks as missing	Cluster health problems. With a Chunkserver disconnected at the same time, bring it back before treating the count as loss
An endangered or undergoal count that does not fall	Cluster health problems, once the cases that are not faults are ruled out
The cluster reports that a Storage Class cannot be satisfied	Cluster health problems
The cluster reports chunks kept in a format other than their class asks for	Cluster health problems
Reads have slowed down across the cluster	Diagnostic tools to see what a client is waiting on. A disk failing without reporting errors is Chunkserver and disk recovery. A path that drops or delays packets is network problems
Writes have slowed down or pause while reads are unaffected	Chunkserver and disk recovery first, because one failing disk affects servers other than its own (disk error handling and disk removal). Network problems when the disks are sound
Listing directories and creating files is slow while transfers are unaffected	Diagnostic tools for what a client is asking the Master Server, and limits and scale for what one Master Server can answer
A Chunkserver reports an overloaded queue	The GUI for what the state means and what it costs. Chunkserver and disk recovery when failing disks are the cause
Writes fail for lack of space while the disks still report free space	Space management for the space itself. Cluster health problems when the Storage Class cannot be satisfied
The cluster holds more space than the data you wrote	Space management, with raw space and user space for what redundancy costs
Servers or disks fill at very different rates	Rebalancing for the pace at which the cluster levels them, and space management for the figures to compare
A Chunkserver is listed as disconnected	Chunkserver and disk recovery. Network problems when it reconnects and drops again
A disk reports errors, or has been marked as damaged	Chunkserver and disk recovery
The Master Server does not start, or no Master Server is the Leader PRO (PRO Edition only)	Master Server problems
More than one Master Server claims to lead the cluster PRO (PRO Edition only)	Master Server problems
The cluster warns that it holds no second copy of its metadata	Master Server problems
A Follower is behind the Leader, or saved a different checksum PRO (PRO Edition only)	Master Server problems
Saving metadata runs long, or ends in a state you did not expect	Master Server problems
A location is offline or in I/O-only mode PRO (PRO Edition only)	Location tasks
The GUI shows no data, or stops refreshing	The GUI

Some of these symptoms are never announced. The size of the trash raises no alert, and neither does a disk that answers slowly while still answering. You find both by reading, not by being told. The trash figures are listed with the cluster totals in space management. Per-disk operation times are among the readings in what to watch.

Some alerts do not report faults that this chapter covers. The alerts about component versions report that the parts of one cluster are not on the same version. The alerts about the license report how much of a granted license is used and how long it has left. Act on both, but no procedure in this chapter repairs them. Alerts covers what each of them says. How licensing works covers the license itself.

Some faults are the cluster at work

Each case below looks like a failure. None of these cases is a failure on its own.

• An undergoal count that does not fall while files are being written. A chunk being written to cannot be replicated at the same time. A file under continuous writes therefore stays below its class for as long as the writing lasts, and catches up afterward. Self-healing describes this, together with the conditions that do hold a repair back. The symptom becomes a fault when the count does not change after the writing has stopped: cluster health problems.
• Repairs that pause for a Chunkserver that is disconnected or under maintenance. The cluster waits for a server marked as returning, and does not replicate that server's chunks elsewhere. That wait is the purpose of maintenance mode. The symptom becomes a fault when the server is connected again and out of maintenance, but the counts still do not change.
• A disk that reports that it is not ready for removal. The states that a marked disk passes through report the progress of the removal, not a fault of the disk. The procedure that produces these states is described in disks and Chunkservers. The symptom becomes a fault when a marked disk never reaches removal ready.
• A mount that stops responding for a while and then continues. A client retries during an interruption rather than returning an error. An outage between a client and the cluster therefore shows up as a pause. The retry budget that limits that pause is described in client background work. The symptom becomes a fault when the pause lasts longer than the interruption that explains it, or when no interruption explains it: client and mount problems.
• A file that could not be read for a while, and then could. When enough parts of an Erasure Coding chunk are left, the cluster rebuilds the chunk. The cluster tells the client to wait for the rebuild rather than failing the read. A disk or a server lost during a read therefore costs a delay instead of the data. The symptom becomes a fault when the read never completes, or when the chunk is still counted as missing with every Chunkserver connected.
• Deleted files that do not free their space at once. A deleted file moves to the trash and keeps its disk space until its retention ends. The cluster then frees the space through work that it paces itself. Space management covers the state of that space and how to free it sooner. The symptom becomes a fault when the trash grows beyond what the retention you set explains.

Support starts from the same readings you did

When none of these pages explains what the cluster is doing, send the readings you started from together with the request. Contacting support covers who to write to and what to send.

Logs covers which log each component writes and where that log is written.