Diagnostic tools

This page maps a diagnostic question onto the tool that answers it, and describes how to read the output. The syntax of each command and its full list of options are listed in command line tools.

What you want to know	Where the answer is
Where the chunks of one file lie, and what is wrong with them	mfsfileinfo
Whether one file is complete, in a single summary	mfscheckfile
What a subtree holds and what it occupies	mfsdirinfo
Which names lead to an inode	mfsfilepaths
What one client is doing right now, operation by operation	.oplog and .ophistory in the root of that mount
Which clients hold files open, and who holds a lock	The Mounts tab of the GUI, mfscli with -SOF and -SAL
What the cluster changed, operation by operation	changelog.0.mfs in the Master Server's data directory
What each session asked the Master Server for in the last minute	.mfsmaster_info.txt
What the metadata held at an earlier moment	mfsmetadump, mfsmetasearch, mfsmetadirinfo
What the cluster is doing as a whole	the GUI, the CLI, the system log

The file tools answer through a mount

mfsfileinfo, mfscheckfile, mfsdirinfo, and mfsfilepaths are names of one program and are described together in mfsdiagtools(1). Each of them takes a path inside a mounted MooseFS instance, walks up from that path to the root of the mount, and reads .masterinfo there to reach the Master Server. A path outside any mount ends the run with <path>: not MFS object.

mfsfileinfo and mfscheckfile take files. A directory named on the command line returns Operation not permitted. The run then continues with the next argument. To read over a directory, name its files: mfsfileinfo -w /mnt/mfs/data/* for one directory, find /mnt/mfs/data -type f -exec mfsfileinfo -w {} + for a whole tree. mfsdirinfo is the tool that takes a directory itself.

One run of a tool opens one connection and uses it for every file named on the command line. Naming many files in one call costs one connection. A shell loop that calls the tool once per file costs one connection per call.

mfsfileinfo says where every chunk is

For each chunk of a file the tool prints the chunk's identifier and version. Under that line the tool prints one line per stored copy or Erasure Coding part. Each of those lines shows the address and port of the Chunkserver that holds the copy or part, and the status that the Master Server recorded for that copy. The statuses themselves are described in chunk versions and states. WRONG VERSION says that the Chunkserver holds this chunk at a version other than the current version. Such a copy counts toward nothing, even if its data reads perfectly. A chunk with no copy and no EC part line under it has nothing stored anywhere in the cluster.

Under the copies the tool prints its assessment of them:

/mnt/mfs/data/archive.tar:
	chunk 0: 0000000000A1B2C3_00000004 / (id:10597571 ver:4) ; mtime:1787227200 (2026-08-20 14:00:00)
		copy 1: 192.0.2.31:9422 ; status:VALID
		copy 2: 192.0.2.32:9422 ; status:WRONG VERSION
	chunk 1: 0000000000A1B2C4_00000001 / (id:10597572 ver:1) ; mtime:1787227260 (2026-08-20 14:01:00)
		missing data copies / data parts - unrecoverable

Each such line describes a different situation:

Line	What it says about the chunk
missing data copies / data parts - unrecoverable	Neither a copy nor enough parts are left. The content is not in the cluster.
data parts missing - recoverable	Parts are gone and enough of them remain for the chunk to be rebuilt.
full copies have different checksums !!!	The copies carry the current version and disagree about the content, which the version alone does not reveal.
chunk marked as readable after repair - reading this chunk will receive zeros	A repair found no usable copy and marked the chunk readable, so reads of it return zeros and writes to it fail.

Chunkserver and disk recovery covers what to do about a chunk that has lost its copies.

mfsfileinfo -w prints only those chunks and prints all chunks are ok when it finds none. -q reduces the output to the number of copies and parts of each chunk. It cannot be combined with -c, -p, and -s.

What the Master Server records and what the disks hold are two different statements. -c, -p, and -s query the Chunkservers themselves. -c fetches the checksum of every copy from the Chunkserver holding it. -p fetches the path of the chunk file on that Chunkserver's own file system. -s combines the checksums into one signature per file, so two files with equal signatures hold equal chunk content. For a Chunkserver that does not answer, the output shows ; can't get checksum or ; can't get path in place of the value.

mfscheckfile is the same reading in one summary

mfscheckfile is mfsfileinfo -q under a different name. It counts the file's chunks, grouped by their redundancy, and prints one line per group, with no per-chunk detail. Checking a file shows the output for a healthy file. Look for the lines that name no redundancy at all, chunks not found: and no copies and EC parts :, because they count the chunks without which the file cannot be read. empty (zero) chunks: is not such a line. It counts the holes of a sparse file, which are stored nowhere by design.

mfsfilepaths turns an inode back into names

The operations log, the list of open files, and the Master Server's report of missing files all name inodes rather than paths. mfsfilepaths prints every path that leads to a given file. A hard-linked file has more than one. Inside a mount the tool also accepts an inode number instead of a path.

The operations log says what one client does

The root of a mount holds two names that record the file system operations that the mount performs, .oplog and .ophistory, listed with the other metafiles. They differ in when the recording starts. A read of .oplog returns the operations that happen from the moment the file is opened, so you start that read before reproducing a problem. A read of .ophistory starts with what the mount has already done and then continues in the same way. You therefore read it after the symptom has appeared. Neither read ends by itself, so you stop it yourself.

The depth of the history is measured in bytes, not in time. The mount keeps the operations in a 16 MiB ring buffer. A read of .ophistory starts up to 15 MiB back in that buffer. A line is cut off at 1000 bytes, so 15 MiB holds at least 15,000 lines, and in practice many more. The time those bytes cover depends on how busy the mount has been. For a mount that has done less than that, the history starts at the moment it was mounted.

Each mfsmount process keeps its own log, in its own memory, whether anyone reads it or not. Unmounting discards the log. Only mfsmount keeps this log: a meta mount (mfsmount -o mfsmeta) and the block device client keep none of their own. The changelog records what the cluster did for them instead.

A line carries who, what, and the outcome

Every line starts with the same header. The header holds the local date as month and day and the time to the microsecond. Then come uid, gid, and pid of the process that asked for the operation, then cmd: and the name of the operation. The header has no year. After the name come the parameters of the call in parentheses, a note in parentheses when the mount answered from its own cache, and the handle of the open file in square brackets. A colon follows, then the status and whatever the call returned.

04.22 11:30:15.822513: uid:0 gid:0 pid:1509223 cmd:open (54260061,O_RDONLY) (using cached data from lookup): OK (direct_io:0,keep_cache:0,append_mode:0) [handle:01000001]
04.22 11:30:15.823003: uid:0 gid:0 pid:1509223 cmd:lookup (54260060,archive_copy.tar): ENOENT (No such file or directory)
04.22 11:30:15.840440: uid:0 gid:0 pid:1509223 cmd:read (54260061,131072,0) [handle:01000001]: OK (131072)
#
04.22 11:30:16.001868: uid:0 gid:0 pid:0 cmd:release (54260061) [handle:01000001,uselocks:0,lock_owner:0000000000000000]: OK

These parts of a line are misread most often:

• pid:0 says that no user process caused the line. The kernel delivers some operations, release among them, after the process that caused them has already continued. The mount also writes lines of its own when it invalidates what the kernel has cached.
• A note such as (using cached data from lookup) or (using open dir cache) says that the mount answered locally and that the call never reached the Master Server. Those lines say little about how the cluster performs.
• Not every error is a fault. ENODATA on getxattr says that the attribute is not set on that object. For a program that is about to create a file, ENOENT on a lookup means that the name is free.
• A line holding a single # marks a second in which nothing was reported. It is produced for whoever is reading and is never stored, so .ophistory contains none. Two readers each get their own. The absence of # from a saved file means nothing.

Filter the log by process or by inode

The log of a busy mount is too long to read line by line. One filter is the process: start the read, reproduce the problem, then stop the read yourself, because it ends no other way. Remove the lines that the cache answered, then count what is left per process ID.

cat /mnt/mfs/.oplog > /tmp/oplog.txt
grep -v "using .*cache" /tmp/oplog.txt | cut -d " " -f 5 | grep -v "#" | sort | uniq -c | sort -n

Check the saved file before you work on it or send it anywhere. On a system whose standard tools come from an implementation other than GNU, cat writes an empty file here and reports no error (the mount's metafiles read as empty).

The other filter is the object. Every line names the inode that its operation worked on. grep on that number therefore follows one file through everything that touched it. mfsfilepaths turns the number back into a name. Locks are read this way. Look for a run of setlk lines on one inode with type:W, no type:U between them, and EAGAIN (Resource temporarily unavailable) at the end. Such a run is one process waiting for an exclusive lock that a different process holds.

The mount option -o mfsdebug adds lines to the log, including the start of each operation and the detail of lock calls, at the cost of volume.

The Windows Client PRO (PRO Edition only) writes a log of its own under the same name, in a different format, and reports to the Windows Event Log. The Windows Client covers both.

The Master Server holds open files and locks

The Master Server tracks every open file and every held lock. Both readings appear under Open files and locks on the Mounts tab of the GUI and in the CLI as mfscli -SOF and mfscli -SAL. Open files are listed for one session at a time, one row per path, so an inode reachable under several names appears several times. Locks are listed for one inode at a time.

A lock row names the session, host, and mount point holding the lock. The lock type shows the family of the lock: POSIX for a record lock on a byte range or FLOCK for a lock on a whole open file. The two families do not interact, so a row of one kind says nothing about the other kind. The read and write column says whether the lock is shared or exclusive. A whole-file lock has no byte range, so the start and end columns say nothing about it.

A session outlives the connection that made it, so a file can be held open by a client that is no longer connected.

changelog.0.mfs says what the cluster changed

The operations log answers what one client asked for. The changelog answers what the cluster changed, from every client at once. The Master Server writes each metadata change to changelog.0.mfs in its data directory as one line of plain text. grep and tail are then enough to see what happened in a given minute. Changelogs and dumps describe what a line holds and how many hours of them are kept. The changelog is not a log of messages. The modules report to the system log instead.

A run of IDLE() lines records a period without changes rather than a stall, because the Master Server writes one after a spell with nothing else to record. Names in the arguments are percent quoted, so search for a file whose name holds a comma or a parenthesis in its quoted form.

.mfsmaster_info.txt counts per session and per minute

The info action of mfsmaster and mfschunkserver writes a file in the data directory, .mfsmaster_info.txt or .mfschunkserver_info.txt. The file holds the state at the moment of the request. Contacting support describes how to produce one.

Its sections answer questions that the interfaces do not answer. [session ops] holds one line per session and operation, in the form session ip:<address> mount_point:<name> operation:<name> count:<number>, counted over the last minute. That window is shorter than the hour that the Operations table of the GUI uses. [changelog] shows how far back the changelog held in memory reaches, as a metadata version and a timestamp with its age. If that changelog is empty, the section says so.

Metadata reads without a running Master Server

A metadata dump holds what the Master Server had when the dump was written. Read it after a failure. It answers about a moment that has passed. It gives that answer whether or not a Master Server is running now.

mfsmetadump prints a dump section by section. -s limits the output to one section. Reading a dump describes how to run it. These sections answer incident questions: SESS for the client sessions, OPEN for the files that were open, FLCK and PLCK for the locks of each family, and CHNK for the chunks with their versions and timestamps. If the file can be read through at all, that is an answer as well.

mfsmetasearch answers what a failed disk leaves behind. -c takes a file that lists chunk identifiers in hexadecimal and names the files those chunks belong to. -e selects objects by an expression over their metadata, with inode, length, times, Storage Class, and the other stored attributes as symbols.

mfsmetadirinfo counts a subtree from a dump: inodes, files, directories, and chunks with their lengths and sizes. It answers what mfsdirinfo -p answers on a live cluster, without adding work for the running Master Server.

The current state is a monitoring reading

The tools on this page answer about one file, one client, or one moment that has passed. The GUI, the CLI, and the system log show what the cluster is doing as a whole. What to watch covers which of those readings deserve attention.