Client and mount problems

This page starts from what you see on the machine that has the mount. mfsmount refuses to start, the mount is there but commands on it stop returning, a message appears that no MooseFS module wrote, or a change made elsewhere shows up late. Each entry gives what the client reports, how to confirm the cause without leaving that machine, and what to change.

Those reports appear in a different place once the mount is up. mfsmount detaches into the background. A failure during startup is passed back to the process that ran the command, so the message prints on that terminal. After a successful start the same messages go to the system log. -o mfslogminlevel=LEVEL and -o mfslogelevateto=LEVEL set what the mount sends there. The two options are the mount's own names for the settings that the server modules read from their configuration files as SYSLOG_MIN_LEVEL and SYSLOG_ELEVATE_TO (system log).

The mount answers about itself as well. In its root, .params lists the options and the session in force. .stats counts what the connection to the Master Server has cost. Both files are in that root, are read by name, and are used throughout this page (special files). The Windows Client has symptoms of its own, collected on Windows Client troubleshooting.

The mount does not come up

mfsmount checks the mount point, then the options, then the Master Server, and attaches the file system to FUSE last. It reports the first thing it rejects, so the message says how far the startup got.

A mount point the client will not take:

no mount point
given mountpoint '/mnt/mfs' is not a directory
mountpoint '/mnt/mfs' is not empty

no mount point means that neither the command line nor mfsmount.cfg named a directory. A mount point that is not a directory, or is not empty, is caught by a check that runs on FUSE 3, macOS, and FreeBSD. On those systems -o nonempty turns off the emptiness check. On Linux with FUSE 2 the client mounts over a non-empty directory without comment.

A Master Server address that leads nowhere:

can't resolve master hostname and/or portname (mfsmaster:9421)
can't connect to mfsmaster ("192.0.2.10":"9421")

If the name does not resolve, the fault is in DNS or in /etc/hosts on this machine. can't connect to mfsmaster means that the name resolved and nothing answered on the port. At startup the client makes one attempt and up to ten retries, and resolves the name again before each attempt. The connection timeout is 2 seconds per attempt, so this message follows the command by up to twenty-two seconds. -o mfsdelayedinit removes name resolution and connection from the startup path: the mount comes up without contacting the Master Server. A failure to connect appears later in the system log instead.

A registration the Master Server refuses:

mfsmaster register error: Permission denied
mfsmaster register error: Password is needed
mfsmaster register error: Incorrect password
mfsmaster register error: Operation not permitted

Permission denied means that no export entry covers this combination of client address, mounted path, and client version. Password is needed and Incorrect password mean that an entry matched and has a password. Since 5.2.0, Password is needed also answers every mount that sends no password to a cluster with authentication for I/O turned on. -o mfspassword, -o mfspassfile, and -o mfsmd5pass supply that password. -p asks for it at the prompt. Operation not permitted means that the Master Server refuses the client's version. One cause is a cluster that keeps files in an erasure-coded form that this client cannot read (4.0 and newer read erasure-coded files, 4.26 and newer the form with four data parts). The other cause is a cluster with authentication for I/O turned on, which needs a client from 5.0 up and an export with a password. Entries, their passwords, and the paths they cover are the administrator's side of access control. RESTRICT_INCOMPATIBLE_CLIENT_VERSIONS on the Master Server lifts the version rule for Erasure Coding, at the cost of leaving those files unreadable to the mount that connects. The rule for authenticated I/O applies either way (mfsmaster.cfg).

A password failure is recorded on the Master Server too:

client from IP:192.0.2.30 attempted a connection with wrong password
client from IP:192.0.2.30 attempted a connection without needed password

When no export matches, the Master Server writes nothing to its log, so the client's message is the only record of the refusal. Compare the export list with the address, the path, and the version that the mount used. That comparison is then the only way to see what did not match.

A FUSE layer that will not attach the file system, once everything before it has passed:

error in fuse_session_mount

With FUSE 2 the wording is error in fuse_mount. The client runs no separate test for FUSE and has no message of its own about FUSE. This one message therefore covers a module that is not loaded, a missing /dev/fuse, and the lack of rights to that device. mfsmount -V prints the version of the client and the FUSE version it was built against.

An option the FUSE helper refuses, on the line above that message:

fusermount3: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf

mfsmount adds allow_other to every mount unless -n switches the standard options off. fusermount3 accepts that option from root alone, and from any other user only when /etc/fuse.conf contains the line user_allow_other. Add that line, or mount with -n. A mount made with -n admits only the user who made it, so processes that run under a different identity are refused inside it.

All of the messages above end the mount attempt, so the mount point stays empty. On macOS a mount that did come up returns the same words: on macOS every command on the mount returns "Operation not permitted" covers that case, which has nothing to do with the export rules.

On macOS every command on the mount returns "Operation not permitted"

The mount is up and Finder lists the files, but every command run from a terminal is refused:

$ ls /Volumes/mfs
ls: /Volumes/mfs: Operation not permitted

Neither MooseFS nor FUSE writes that message. Since macOS Catalina (10.15) the system's TCC (Transparency, Consent, and Control) mechanism decides which applications may read files on a volume such as this one. The terminal is not among those applications until you say so. Finder was allowed both before Catalina and after it, so a mount that Finder reads is not proof that anything on the MooseFS side is missing.

Running the command under sudo or as root changes nothing. TCC checks the application that requests the files. That application stays the same terminal whatever account it runs under.

Grant Full Disk Access to the terminal application you use, whether that is Terminal, iTerm2, or another terminal application. On current macOS releases the setting is at System Settings → Privacy & Security → Full Disk Access. Restart the terminal application afterwards for the change to take effect. The location of that setting has changed between macOS releases. Its name has not, so look for Full Disk Access wherever your release keeps its privacy settings.

Operations hang, then fail with I/O errors

When the Master Server stops answering, operations do not fail at once. The client repeats each one with a delay that grows by 0.3 seconds per attempt up to 10 seconds. The budget is mfsioretries attempts, 30 by default, so an application waits about two minutes before it receives an I/O error. mfstimeout caps the total time of one operation instead. Its default of 0 means no cap. Both limits are evaluated. The first to run out ends the operation. Raising mfsioretries extends the wait: once the delay reaches 10 seconds per attempt, about 8,600 attempts cover a day. The loops that produce this behavior are described in client-side background work.

Data operations stall on the Chunkserver that they are talking to. A Chunkserver that sends nothing for 5 seconds during a transfer counts as failed:

readworker: can't connect to (192.0.2.11:9422): ETIMEDOUT (Operation timed out)

A chunk with no valid copy left is retried on a budget of its own. A read tries again every 10 seconds, for about five minutes in all. A write tries again every 60 seconds and stops after five attempts, about four minutes. A cluster with no space stays on the ordinary budget. Once every known Chunkserver is connected to the Master Server, -o mfserroronlostchunk and -o mfserroronnospace return the error to the application at once instead of waiting the budget out. While a Chunkserver is missing, the operation waits its budget out.

The system log shows nothing while this happens. mfslogretry, 5 by default, is the number of failures that a repeated operation has to accumulate before the client starts logging it. The failures below that count leave no trace at all: five of them where every failure counts as one, fewer where a single failure counts for more. An empty log during a stall therefore does not prove that nothing went wrong. Lower the value to see those attempts.

.stats distinguishes a lost connection from a slow connection. master.reconnects counts reconnections. master.usec_ping is the measured round trip to the Master Server. .params names the address that the mount is working with, as working_masterip and working_masterport. The same file lists every address that the Master Server name resolved to PRO (PRO Edition only). That list is the shortest way from a suspicion about DNS to proof.

Not every stall is caused by the connection. An export can cap how many operations a mount issues per second. The limits granted to this session are listed in .params as opslimit_global and opslimit_per_uid (operation limits).

A mount started with -o mfsdelayedinit does not wait at all before its first registration. The client registers with the Master Server in the background, so mfsmount returns while the cluster is still unreachable. Until that registration goes through, every operation on the mount point returns an input/output error at once.

Every command returns "Transport endpoint is not connected"

The message is the system's, not the client's: the mount point is still attached, but the process that served it has ended. No mount option prevents it. An action on the cluster does not clear it either. Unmount the mount point and mount again.

Then find out why the process ended, in the system log of the client machine. At startup the mount asks the kernel to exclude the mount from the out-of-memory killer. The mount then reports the result:

out of memory killer disabled
can't disable out of memory killer

can't disable out of memory killer means that the request failed and that the process can be selected like any other process. -o mfsallowoomkiller turns this protection off deliberately.

The mount hangs and umount says nothing

A mount whose servers are unavailable answers nothing and refuses nothing. Every call into the mount point waits for the Master Server, on the budget in operations hang, then fail with I/O errors. A process that waits there is in uninterruptible sleep, D in ps, so a termination signal does not reach it. For the same reason a timeout around the command changes nothing. A plain umount waits with it, prints nothing, and leaves the entry in /proc/mounts. When umount refuses and says why, read umount refuses because the target is busy instead.

Two commands release the mount point, both without a message. umount -f ends the mount, and mfsmount exits at once. umount -l detaches the mount point from the directory tree and leaves mfsmount running until the last process releases the mount point. Either way the entry is removed from /proc/mounts.

umount -f has one answer that looks like a refusal:

umount: /mnt/mfs: target is busy.

It comes with exit status 32. It appears only when a plain umount is already hanging on the same mount point. The process that keeps the target busy is that waiting umount. The mount point is released all the same. When -f is issued on its own, it never answers that way.

Nothing has to be forced when the servers are available again. The client reconnects by itself. The calls that were waiting finish, and the mount serves as before. The order of stopping the cluster that avoids all of this is described in starting and stopping the cluster.

umount refuses because the target is busy

umount refuses while any process still uses the mount, with a file open on it or with a working directory inside it. MooseFS reports nothing at that point. No mount option changes it. The mount detaches once the last user releases the mount point. .oplog includes, for every operation, the pid of the process that issued it, so a process still working in the mount can be identified there. A process that only keeps a directory of the mount as its working directory issues no operations and does not appear there (client's operations log).

Device or resource busy returned to an application has a different cause. On Linux it comes from kernels older than 4.19. The client tells the kernel how long it may keep an object. On those versions the kernel keeps the object longer, so reuse of that object's inode number fails. On such a kernel the mount withdraws those directory entries itself. It holds each entry for at least 30 seconds first (correspondingly longer where mfsdirentrycacheto is set above that). When libfuse is too old to allow this, the mount says so:

your libfuse version is too old to properly fix the EBUSY bug

The Master Server enforces the same arrangement from its own side with INODE_REUSE_DELAY, one day by default. An inode number stays out of use for that long after its object is removed. A value below the longest cache timeout of any client makes the collision possible again (mfsmaster.cfg).

One mount's change shows up late on another

A file written on one mount keeps an old length or old content on a second mount for a while. This shows first in the last part of a file that is still being written.

Metadata is cached by the kernel, not by the mount. The mount tells the kernel how long an answer stays valid. For a file that the second mount holds open, the Master Server pushes the correction as soon as the writer reports the change. The delay is then in the report and not in a cache (multiple mounts). For everything else the timeout decides how long the two mounts disagree. The attribute cache is the cache that hides a length (client cache and read-ahead).

Ending that disagreement costs cache. Turn off the attribute cache for the whole mount, or for the objects that need it:

mfsmount -H mfsmaster.example.com -o mfsattrcacheto=0 /mnt/mfs
mfsseteattr -f noattrcache /mnt/mfs/projects

When the content and not only the length is stale, turn off the data cache as well:

mfsmount -H mfsmaster.example.com -o mfscachemode=DIRECT /mnt/mfs
mfsseteattr -f nodatacache /mnt/mfs/projects

The per-object form (mfsseteattr) costs less than the per-mount form, because the rest of the mount keeps caching. The opposite change, raising entry timeouts to save round trips, works against agreement between mounts. Leave mfsnegentrycacheto at its default of 0. Otherwise a mount answers from its own memory that a name is absent after a different mount created the name.

Written data is missing after a crash

A write() that returns has put the data in the mount's write cache, not on Chunkservers. fsync() returns when the Chunkservers have confirmed everything outstanding for that file. close() waits the same way, so a file closed normally is stored by the time the call returns. An error from a write that failed in the background is delivered to the next call on that file. That call may be a later write(), the fsync(), or the close() (client-side background work).

Opening the file with O_SYNC or O_DSYNC changes none of that. The client reads the flags and records them in its operations log. The data is sent on the write workers' schedule, as it is without the flags. Durability comes from the call, so an application that needs durability calls fsync() and checks what the call returns. -o mfsfsyncmintime=SEC does the opposite: a file closed sooner than SEC after it was opened skips the wait at close. The default of 0 makes every close wait.

The mount's metafiles read as empty

Some systems take cat, head, tail, and the rest of the standard tools from an implementation other than the GNU one. That implementation is the default in Ubuntu 26. On those systems, reading a metafile of the mount returns nothing instead of its content. .masterinfo is the exception and reads normally. For that reason the mfs* tools keep working there. Those systems install the GNU tools alongside, under names of their own. The GNU tools read the files:

gnucat /mnt/mfs/.params

In this case no tool reports an error, so a script that reads .params, .stats, .oplog, or .ophistory gets an empty result instead of failing. Watch for that in automation. Exclude one other cause of an empty result first: .params, .oplog, and .ophistory are readable by root alone. Any other user gets Operation not permitted whichever tool reads them (special files).

One mount is slow, the cluster is fine

When a single mount transfers less than the cluster and the network could carry, and the cluster reports nothing wrong, the limit may be the mount itself. Everything that one mount transfers passes through one kernel channel. No change on the Master Server or on the Chunkservers raises that ceiling. Mount the file system a second time on the same machine and split the work between the two mounts. Each mount is a channel of its own, so a total that rises proves that the ceiling was per mount (the per-mount throughput ceiling).