What the FUSE layer costs

mfsmount presents a MooseFS mount through FUSE (Filesystem in Userspace). The file system logic runs in an ordinary process. The kernel forwards file system calls to that process. That arrangement makes the mount look like a local file system to every application, but the arrangement adds a cost to every operation. This page describes how one operation passes through the layer, what the cost is, and what the client turns on and off to keep the cost down. An application that does not need the mount avoids that cost completely. The last section covers that case.

Every operation crosses the kernel boundary twice

An application call on the mount enters the kernel as an ordinary system call. The kernel does not answer the call. Instead, the kernel wraps the call into a FUSE request and queues the request on the FUSE device, /dev/fuse. In userspace, the mfsmount process reads the request from that device, handles the request, and writes the reply back. The kernel then completes the application's call.

Each such round costs context switches between the application, the kernel, and mfsmount. Each such round also costs copies. The payload of a read or a write is copied between the kernel and the mfsmount process on the way through. FUSE offers splice capabilities that would move data between the kernel and the process without copying. mfsmount turns these capabilities off, so every data byte that crosses the mount is copied at the boundary.

All requests of one mount go through one FUSE connection. That connection is opened when the mount is created. The requests are not handled one at a time. mfsmount runs a pool of worker threads that read from the connection, unless -s forces single-threaded operation. Per-mount throughput covers the ceiling that the single connection puts on the throughput of one mount. Multiple mounts covers how to spread work over several connections.

io_uring carries the same requests without the device round trip

In the classic FUSE transport, the process reads a request from /dev/fuse and writes the reply back to the device. A MooseFS mount uses this transport by default. Linux offers a second transport for the same traffic, in which the kernel and the process pass requests through shared ring buffers instead. The classic path serializes the requests of one connection. That serialization bounds how many small operations one mount completes per second, whatever the application does. The second transport removes that serialization. Changes above the transport do not remove the serialization. More application threads, deeper queues, and asynchronous I/O engines were all measured against that bound. The bound stayed the same. The application does not need io_uring. The mount does.

MooseFS does not name the transport anywhere. mfsmount passes every -o option that is not one of its own to libfuse unchanged. -o io_uring therefore goes through the client to the library. The library and the kernel decide the rest. Whether the transport is available does not depend on the MooseFS version. Three things on the client machine decide availability. All three have to hold at once:

• A Linux kernel 6.14 or newer, built with the options that the transport needs. A distribution kernel does not always have these options turned on.
• That kernel's fuse module, with its enable_uring parameter set to 1.
• libfuse 3.18.3 or newer, with -o io_uring given at mount time. Releases before that version either do not include the transport or include it with faults that stop it from working. If the linked library offers the transport, the client lists -o io_uring in its own mount help. Only that mount help shows whether the linked library has the transport.

warning

Check the libfuse version on the machine before you rely on the transport. The version that the operating system installs is the version that decides. That version lags the library's own releases. A package older than 3.18.3 does not refuse -o io_uring. With such a package, the option is either unavailable or broken. Where the package is too old, build the library from source and link mfsmount against that build.

The gain was measured on one test cluster. A single mount on this transport reached the same random 4 KiB read rate as four classic mounts, at a fifth of the time per operation. That cluster kept its Chunkservers on ramdisks, connected by a 40 Gb/s fabric. This setup removes the cost of the storage from the measurement. Only the client path and the network remain. The ratio describes those two layers. It is not a rate to expect from an installation whose Chunkservers hold data on disks.

The client asks for few, large requests

mfsmount builds against libfuse 3 (3.2.1 or newer) where it is available and falls back to libfuse 2 (2.6 or newer). The libfuse 2 build is a compatibility fallback, not a supported configuration. See clients. The two builds differ in the request sizes that the client asks for. With libfuse 3, the client leaves the kernel's write and read-ahead request sizes at their defaults. Those defaults are the maximums. The client also declares no limit of its own on the read request size. With libfuse 2, the client sets the write and read-ahead request sizes to 128 KiB and requests the big_writes capability to let write requests reach that size. A large sequential read or write therefore crosses the boundary in as few requests as the kernel allows. Each crossing carries as much data as it can.

Directory listing uses readdirplus. The reply includes each entry together with its attributes, so listing a directory does not cost an extra attribute request for each entry. A MooseFS Meta mount is the exception and is listed without readdirplus.

A repeated operation never reaches the process

The client keeps the cost down in a second way, by skipping the trip instead of making the trip cheaper. Replies to lookups and attribute reads include validity timeouts. While a timeout runs, the kernel answers a repeated operation itself. The operation ends where it began, with no crossing into userspace. File data read once stays in the kernel page cache. Depending on the cache mode, that data is kept between opens of the same file. Client cache and read-ahead covers which caches exist, what their timeouts are, and how read-ahead fills them.

Writing is the only case in which the client does not use a kernel cache. mfsmount turns the kernel's writeback cache off, because that cache introduces POSIX incompatibilities. Instead, the client keeps a write cache of its own inside its process. A write therefore always crosses the boundary, and completes against the client's cache rather than against the cluster.

The layer ends where the network begins

The trip through FUSE is local to the client machine and is the same for every operation. The operations differ in what mfsmount must do next. A metadata operation costs a network round trip to the Master Server unless one of the client's own caches answers the operation. Those caches and the background work that maintains them are covered by client cache and read-ahead and client background work. File data does not make that round trip at all. File data goes directly between the client and the Chunkservers, along the read path and the write path. The FUSE layer adds its crossing cost to every operation equally. The layer does not change which servers the operation communicates with.

libmfsio leaves the layer out

All the costs above come from presenting the cluster as a mounted file system. An application that does not need a mount can link libmfsio instead. The library contains the same client logic that mfsmount runs. The library's calls go from the application to the Master Server and the Chunkservers without entering the kernel on the way. Requests never cross /dev/fuse. Data is never copied at that boundary. The ceiling that one FUSE connection puts on a mount, described in per-mount throughput, does not apply. FUSE is also no longer a dependency: mfsmount is linked against libfuse, the library against the threading library alone. mfsbdev is built on the library, so a block device client does not need FUSE either. Client and mount covers what the library is, together with every other form a client takes.

The mount is given up. Nothing appears in the machine's namespace. The data is therefore reachable only from the application that links the library, and only through the calls that the library offers. That means that the application has to be written or rewritten for those calls. Every process that links the library registers a session of its own and keeps its own caches, so what one process cached is of no use to any other process. Where an application cannot be changed, or where other tools on the machine have to see the same files, use the mount to reach the data.