Exporting MooseFS over NFS
A machine that has MooseFS mounted can re-export that mount with an NFS server. Hosts that cannot run a MooseFS client then reach the same files over NFS. This page describes how to set up such a gateway, and what the second protocol drops on the way. On a host that can run a MooseFS client, mount MooseFS directly instead (Install clients). Everything below follows from putting one file system protocol in front of another protocol.
One MooseFS client serves many NFS clients
The gateway is an ordinary MooseFS client host that also runs an NFS server. The NFS server exports a directory that is also a MooseFS mount point. Every call that an NFS client makes therefore reaches the mount as a local call made by the NFS server.
The Master Server has one client here: the gateway. The mount of the gateway registers a single session. The entry in mfsexports.cfg whose address range covers the gateway admits that session. That entry applies because the Master Server matches entries against the address that a connection comes from. The NFS clients behind the gateway have no entry of their own and never appear at the Master Server.
Mount MooseFS on the gateway
Install the MooseFS client on the gateway (Install clients) and mount the subtree that you intend to share. The mount command is the ordinary one:
mkdir -p /mnt/mfs
mfsmount -H mfsmaster.example.com -S /shared /mnt/mfs
-S gives the mount a subtree of the MooseFS namespace as its root. The rest of the file system is then out of reach of the export. The subtree must exist before the mount is made. A mount of a path that does not exist in the instance is refused with mfsmaster register error: No such file or directory. mfsmount adds -o allow_other to every mount unless -n switches the standard options off. The mount is therefore not restricted to the user that created it. The NFS server can then access the mount. The rest of the settings are listed under mount options and in mfsmount(8).
On the Master Server, the entry in mfsexports.cfg that covers the gateway's address decides what the gateway may do with that subtree. Access control covers editing the file and reloading it.
For a gateway that starts without manual steps, put the mount in /etc/fstab:
mfsmaster.example.com:/shared /mnt/mfs moosefs rw,_netdev,mfsdelayedinit 0 0
mfsdelayedinit connects to the Master Server in the background, so the mount can be made before the network is up. Boot then does not depend on the order of the mount and the network.
Export the mount point, then mount it
Mount MooseFS first, then let the NFS server export the mount point. One entry in /etc/exports names the network that may reach the export, and gives the export a file system ID:
/mnt/mfs 198.51.100.0/24(rw,fsid=1)
Apply the entry:
exportfs -ra
fsid= is not optional here. The Linux NFS server identifies an exported file system by its device number. FUSE file systems have no device number that the server uses. An entry without fsid= is therefore refused outright: exportfs: /mnt/mfs requires fsid= for NFS export. Any number that is unique among the exports of that server works. A UUID works as well. fsid=0 has a different meaning. It makes the export the root of the NFSv4 namespace, so clients then mount gateway:/ rather than the path.
The NFS server's own documentation covers every other option in that file. One of them, root squashing, also exists on the MooseFS side. The NFS side runs first. Under the default root_squash, uid 0 never reaches the mount as uid 0, so the Master Server has nothing left to map. An export entry that has no_root_squash passes uid 0 through. The entry in mfsexports.cfg then decides. A MooseFS entry that sets neither maproot= nor mapall= maps uid 0 to uid 999 and gid 999.
On an NFS client, mount the share like any other share:
mount -t nfs 192.0.2.10:/mnt/mfs /mnt/shared
To verify, write a file through the share and look for it in /mnt/mfs on the gateway. The file is an ordinary MooseFS file, recorded with the user and group ID that arrived with the call.
A user's supplementary groups stop at the gateway
FUSE gives the client the user ID, the primary group ID, and the process ID of the calling process. FUSE gives nothing else about the identity of that process. The client fills in the rest itself. The client asks the operating system for the group list of the process that made the call. The client then sends the whole list to the Master Server with the request. On a gateway, the process that makes the call belongs to the NFS server. The group list that reaches the Master Server is therefore the list that the gateway's operating system holds for that process. The list that the user has on the NFS client does not reach the Master Server.
The Master Server grants group permissions when the object's group is in the list that the Master Server received. A user whose access to a file depends on a supplementary group is therefore refused through the share. On a MooseFS mount, the same user is admitted.
Nothing on the kernel NFS server passes the missing groups through. rpc.mountd --manage-gids replaces the group list that arrived from the NFS client with the list that the gateway holds for that user, but neither list reaches the mount.
A host whose users need group-based access can run a MooseFS client and mount the file system directly. Where the share must stay, a user-space NFS server passes the groups through. A third way remains, and it is deprecated. The gateway's export entry can have ignoregid. Since 5.2.0, the Master Server logs export option 'ignoregid' is deprecated and will be removed in a future release for each entry that sets it, at startup and at every reload. This option makes the Master Server skip the group test. For a caller who is not the owner of the object, the Master Server then uses the group and other permission bits together. An object that has a POSIX ACL is evaluated against that ACL instead, so ignoregid has no effect on that object. The option was added when FUSE could not pass auxiliary groups at all. FUSE passes them now, so the option serves this case as a side effect, not by design.
ignoregid applies to every caller admitted through that export, not only to those arriving over NFS. A file whose group bits are wider than its other bits becomes readable, and where the bits say so writable, by every user the export admits.
The export options apply to the gateway
The entry in mfsexports.cfg decides once, for the gateway's session. Every NFS client inherits the result: the read-only or read-write mode, the operations withheld by disable=, the identities that maproot= and mapall= substitute, and the rate that opslimit= allows. All NFS clients share the single opslimit= budget. opsperuidlimit= instead gives each user ID that arrives through the share a budget of its own.
To tell NFS clients apart, use the NFS server's export list or a second gateway. A gateway at a different address is a different session. That gateway can be admitted by an entry with different options and a different subtree.
Everything the share carries crosses one mount
All traffic between the gateway and the cluster passes through the gateway's single mount. That mount has a throughput ceiling set by the client machine rather than by the cluster. All the NFS clients' reads and writes share that one ceiling. Each read and write also crosses the hop between the NFS client and the gateway.
A MooseFS client mounted on the host that needs the data reads and writes with the Chunkservers directly, with no gateway in the path. Where the ceiling of the share is the constraint, split the NFS clients over several gateways. Each group then gets a mount, and a ceiling, of its own.
What a lock over the share reaches depends on the NFS version
The kernel NFS server passes the locks of its clients to the file system under the export. The MooseFS mount either takes these locks or refuses them. Which outcome happens depends on the version that the NFS client mounts with.
Over NFSv4, the lock reaches MooseFS. The mount passes the lock to the Master Server. The Master Server holds the lock for the whole instance, so the lock binds every other client as well. A MooseFS client on another host that asks for a conflicting exclusive lock is refused. The Master Server records the lock against the gateway's session, the same way as any other lock that this mount takes. A flock from an NFS client arrives as a POSIX lock over the whole file, because the NFS client sends the lock in that form.
Over NFSv3, locking fails instead. The call returns ENOLCK, No locks available, because the kernel lock manager does not lock files on a FUSE file system. A MooseFS mount is a FUSE file system. The same lock succeeds when that gateway exports a local directory.
An NFS client that locks a file over NFSv3 is protected from nothing: the call fails, and a program that writes without checking writes anyway. Where two writers share a file, keep the share on NFSv4, serve it with NFS-Ganesha, or give every host a MooseFS client of its own.
File locking covers the lock families and what a mount can keep to itself.
A user-space NFS server passes what the kernel server drops
The gateway can serve the share with NFS-Ganesha instead of the kernel NFS server. Ganesha is an ordinary process on the gateway. Ganesha's VFS back end reaches the mount through the system calls that any other program uses. The identity that arrived with the NFS request is passed with these calls.
The limit above then no longer applies: a user is admitted through a supplementary group. Locks behave the way they do with the kernel server over NFSv4, in both directions. A lock taken by an NFS client becomes a MooseFS lock that binds every client of the instance. A lock held on a MooseFS mount blocks the NFS client the same way.
An export block for NFS-Ganesha 6.5 over the mount:
EXPORT {
Export_Id = 77;
Path = /mnt/mfs;
Pseudo = /mfs;
Access_Type = RW;
FSAL { Name = VFS; }
}
NFS clients mount the pseudo path rather than the gateway's own path:
mount -t nfs 192.0.2.10:/mfs /mnt/shared
NFS-Ganesha's own documentation covers every other setting in that file. The entry in mfsexports.cfg that covers the gateway's address still decides what the gateway may do with the subtree. All the traffic of the share still crosses one mount.
MooseFS-specific operations run where the mount is
The mfs* tools manage snapshots, Storage Classes, quotas, trash retention, and the extra attributes. A tool needs a MooseFS mount to work in. The tool reads the .masterinfo metafile from the mount that contains the path that the tool was given. The address stored there is that mount's local proxy, 127.0.0.1 unless the mount was started with -L. Run the tools on the gateway. The changes apply to the same file system that the NFS clients read.