Skip to main content

Client and mount tasks

This page covers the administrator's work on the client side of a running cluster. It covers changing what a mount was started with, giving a machine a subtree instead of the whole file system, and giving the machine a password. The page also covers unmounting without losing writes and clearing the session that a mount left behind. The machine already has the client package installed and has mounted the file system once (install clients, first mount and verification). The everyday work inside a mount needs no administrator rights and is described in using the file system.

A mount holds its options until it is mounted again​

mfsmount reads its options once, at mount time. There is no way to change them afterwards. The mount has no remount and no configuration reload. A signal does not make it read mfsmount.cfg again. Editing that file changes only what the next mount uses. The same is true of what the Master Server told the mount at registration. An export entry that you edit now therefore applies to this machine only from its next registration.

While the mount runs, you can read the options that it is running with from the meta file that reports them:

cat /mnt/mfs/.params

The file is readable only by root and cannot be written. The content is a snapshot of the options in force. The mount options explain what each option means.

To change an option, unmount and mount again. The mount loses its session. A clean unmount tells the Master Server to close the session. Closing the session drops what the session was holding: the record of files kept open, the locks taken through the session, and the hold that keeps a deleted-but-open file in the sustained area. Applications on that machine have to stop using the mount point before the mount stops anyway. Plan the change as an interruption for the machine, not as a reconfiguration of it.

Verify with mfscli -SMS, described under mfscli. The machine appears among the active mounts with its own session ID. The old session ID is removed and is not listed among the inactive mounts.

Give a machine a subtree, not the whole tree​

A mount can be rooted at a directory inside the file system rather than at its top. The client asks for the directory. The Master Server decides whether the requesting machine may have that directory. The two sides therefore do different jobs. Both sides have to agree.

  1. In the export entry that admits the machine, allow the directory. An entry that names the directory admits exactly that directory. An entry that names a directory higher in the tree admits the directory only with alldirs. That option allows any subdirectory below the named directory. Reload the Master Server after the edit. For what that reload does to the mounts that are already up, read Changing an export password disconnects every mount. Every one of these mounts reconnects, whatever line you edited.

  2. Mount with the directory named:

    mkdir -p /mnt/projects
    mfsmount /mnt/projects -H mfsmaster -o mfssubfolder=/projects

    -S /projects is the same thing in short form. The third form is positional: mfsmaster:/projects as the source argument. Without any of these forms, the mount is rooted at /.

  3. Verify: ls /mnt/projects shows the content of that directory as the top of the tree. The Root dir column on the Mounts tab of the GUI, or in the output of mfscli -SMS, shows the directory that the Master Server actually gave.

The two sides fail differently. The difference tells you which side to fix. A directory that the export entry does not cover is refused at registration with Permission denied. The mount does not start. Some directories are covered by the entries but do not exist in the file system, or are not directories. These directories are refused afterwards with No such file or directory or Not a directory. When several entries match the same machine, the Master Server uses the rule described in export rule evaluation.

Four ways to pass a password outrank each other​

An export that requires a password gives the mounting machine four ways to supply it. If a machine supplies two of them, the machine does not choose which one is used. The order is fixed:

  • -p, also spelled --password, prompts for the password and takes precedence over everything else. A value given by any other route is discarded without any message.
  • -o mfspassword=PASS takes precedence over -o mfsmd5pass=MD5. The digest form is used only when no clear-text password was given.
  • -o mfspassfile=FILE reads the password from the first line of a file. It cannot be combined with either of the two options in the previous item. If you combine it with either of them, the client refuses to start instead of applying a precedence. The client reports that the option is mutually exclusive with them and stops.

Prefer the file. It keeps the password out of /etc/fstab and out of the command line, where every user of the machine can read the password. The password is never sent over the network in any of the four cases. The client sends an answer to a new challenge for each connection. Authentication between modules describes that exchange.

To verify, mount from the machine with no password at all and check that the mount is refused. Then mount with the password and check that the mount succeeds.

Unmount so that the last writes land​

A MooseFS mount holds written data in the memory of the client process until the client has sent the data to the Chunkservers. The client starts sending the data as soon as it is in the cache. A flush, an fsync(), or the release of the last open handle sends whatever is still held back and waits until the Chunkservers confirm all of it. By default, a normal close() therefore returns with the file's data stored. Data that the Chunkservers have not confirmed yet exists only in the process. Killing the process destroys that data. Writes go out on the workers' schedule describes that schedule.

Unmount the file system instead of killing the process:

umount /mnt/mfs

Use the ordinary order: stop the applications that hold files open, then unmount. A mount point that something is still using cannot be unmounted. That case is a symptom, not a task (umount refuses because the target is busy).

The write cache decides how much data can be lost: mfswritecachesize, 256 MiB by default per mount. Verify that the unmount was clean and not abandoned. On the Mounts tab, or in mfscli -SMS, the machine no longer appears among the active mounts. It has not appeared among the inactive mounts either. A mount that ended without telling the Master Server leaves an inactive session instead.

Remove the session a mount left behind​

A client that disappears without unmounting leaves its session on the Master Server. The session is not a leftover record. The session still holds the files that the mount had open and the locks that the mount had taken. It also holds the space of files that were deleted while the mount held them open. Until the session ends, any other machine that wants one of those locks waits. That space is not free.

Such a session ends on its own after SESSION_SUSTAIN_TIME in mfsmaster.cfg, which is one day by default. If waiting a day is too long, end the session by hand.

  1. Find the session ID. On the Mounts tab, the Inactive mounts table lists each session with its session ID, its count of open files, and its remaining time. mfscli -SMS prints the same list. For each row, it also prints the removal command, ready to be copied.

  2. Run the removal command with the session ID:

    mfscli -H mfsmaster -CRS/42

  3. Verify: the command answers Session 42 has been removed. The row disappears from the inactive mounts, and the locks that the session held are available. If the session was holding deleted files open, the sustained figures in the Cluster Summary table fall gradually, as the cluster processes those files.

The Master Server refuses in two cases. Read both answers instead of retrying. Session 42 can't be removed because is still active means that the mount reconnected between your reading of the list and your command. Removing the session now would take the files that the mount has open away from a machine that is using them. Session 42 hasn't been found means that the session had already expired. Remove a session only for a mount that has disconnected permanently. Removing the session of a running mount does not stop that mount, but the removal takes away everything that the session was holding for the mount.