Skip to main content

The per-mount throughput ceiling

A single mount has an upper bound on how much data it can move per second. The bound is a property of the client machine, not of the cluster. A change on the Master Server, on the Chunkservers, or in the network between them does not raise the bound. The bound caps the sum of everything that the mount does, reads and writes together.

All of one mount crosses one kernel channel​

mfsmount attaches the file system through FUSE (Filesystem in Userspace). The kernel passes every operation of the mount to the mfsmount process through a single pipe. The kernel also receives every reply through that pipe. The pipe belongs to the FUSE connection of that mount. Every byte that the mount reads or writes crosses that one pipe, so the pipe's speed is the mount's ceiling.

That speed depends on the kernel version and on the hardware that the machine runs on, so there is no fixed number to plan around. On reasonably modern hardware, that speed is of the order of 4 to 5 GiB/s.

Adding processing threads does not raise the ceiling. mfsmount already handles requests on several threads unless the -s option forces single-threaded operation. The bound is in the channel that the requests go through, not in how fast the requests are handled after they arrive. The channel itself can be exchanged: the cost of the FUSE layer covers a second transport between the kernel and the process. That page also covers the limit on small operations per second that the second transport removes, and what a machine needs before that transport can be used.

The cost of the FUSE layer covers the cost of a single operation on its way through the FUSE layer. Client cache and read-ahead covers how the client's cache and read-ahead shape the traffic before the traffic reaches this channel.

The ceiling is per mount, not per machine​

Every mount is a separate FUSE connection with a pipe of its own. A machine with four mounts has four ceilings, not one shared ceiling. Mounts on different machines have nothing of this in common at all. The page on multiple mounts covers how to split work across several mounts on one machine, and what that costs.

The bound does not limit the cluster. File data goes directly between clients and Chunkservers. Every additional mount adds a new channel. A single machine can therefore saturate a network faster than one pipe, by mounting several times or by leaving the kernel out entirely with libmfsio.

libmfsio leaves the kernel out​

A program that links libmfsio does not mount anything. The program itself becomes the client. The program talks to the Master Server and the Chunkservers directly, with POSIX-like mfs_* calls instead of system calls. The library runs the same chunk I/O code as mfsmount. The FUSE connection no longer exists, so the pipe and its ceiling do not exist either.

MooseFS Block Device (mfsbdev) does not pass through FUSE either. MooseFS Block Device is built on libmfsio and gives the kernel a block device through the NBD interface instead.