Client cache and read-ahead
There are several caches between an application and the cluster, in more than one place. The kernel page cache and buffers inside mfsmount hold file data. Metadata is cached partly in the kernel and partly in mfsmount. This page covers the client-side caching policy: what each cache holds, for how long, and what removes stale entries. The read path and the write path cover how a read or a write proceeds when the caches do not answer. The client page covers what a client is and the forms it takes.
The cache mode decides what the kernel caches and keeps
Data read through the mount is stored in the kernel page cache, like data of a local file system. The mfscachemode mount option decides two things: whether the kernel may cache the file's data at all, and whether pages cached earlier are kept when the file is opened again.
| Mode | While the file is open | When the file is opened again |
|---|---|---|
AUTO (default) | Pages are cached. | Pages are kept when the Master Server allows it (below). |
YES (or ALWAYS) | Pages are cached. | Pages are always kept. |
NO (or NONE, NEVER) | Pages are cached. | Pages are always dropped. |
DIRECT | Direct I/O: every read and write goes through to mfsmount. | Nothing is kept. |
On FreeBSD the default is FBSDAUTO, a fifth mode that resolves once, at mount time. With the FUSE protocol of FreeBSD 12.1 or newer, the mode becomes AUTO. On older kernels the mode becomes DIRECT.
Under AUTO the Master Server decides, because only the Master Server has information about every mount. For each file the Master Server records which session last read or wrote that file. A write erases every other session's record. At each open the Master Server answers either keep or drop. The pages are kept unless another client has written to the file since this mount last read or wrote it. A mount that already corrected its cache for that write keeps its pages too. The last section of this page covers the pushed invalidation that corrects the cache.
Per-file switches override the policy. A file that has the nodatacache extra attribute is served with direct I/O, regardless of the mode. The noattrcache attribute on a file and noentrycache on a directory set the matching kernel metadata timeouts below to zero.
Read-ahead fills mount buffers that expire
Reads are served from a pool of buffers that mfsmount keeps in its own memory. Read-ahead is the policy that fills those buffers early. Every file starts at the first of four read-ahead levels when it is read from the beginning. A file that is first read at another position reaches the first level after mfsreadaheadtrigger bytes read sequentially, ten times mfsreadaheadleng by default. Each further amount of sequential reading equal to the trigger raises the level by one. Each level quadruples the window fetched past the current read: from mfsreadaheadleng, 1 MiB by default, at the first level to 64 times that at the fourth. A jump of more than half of mfsreadaheadleng away from the sequential position lowers the level, so scattered reads stop fetching data that no application requests.
mfsreadaheadsize caps the buffers of all files together: 256 MiB per mount by default, with accepted values between 16 and 2048 MiB. When the buffers fill to between five eighths and three quarters of that cap, the mount steps higher read-ahead levels back down, so one file does not use up the whole pool. The exact fraction depends on the level. A buffer is also not kept indefinitely. At the next read of the file, buffers filled more than 60 seconds earlier are discarded instead of served, so buffered data is never more than a minute old. Read-ahead data that no read uses is dropped. Client-side background work covers which threads fetch the data.
Writes cache in the mount, never the kernel
mfsmount keeps a write cache of its own, mfswritecachesize, 256 MiB per mount by default and accepted in the same 16 to 2048 MiB range. The kernel's writeback cache stays off, for the reason given in what the FUSE layer costs. When and how that cache is emptied is described in client-side background work. The write cache and the read-ahead buffers together limit the data memory of a mount to the sum of the two caps, half a GiB by default.
Metadata caches split between kernel and mount
Metadata answers are cached in two places. The kernel serves what it caches without calling mfsmount at all. What mfsmount caches still crosses the FUSE boundary but saves the round trip to the Master Server. Each cache keeps an entry for a fixed time, set per mount:
| What is cached | Where | Option | Default |
|---|---|---|---|
| File attributes | kernel | mfsattrcacheto | 1.0 s |
| Directory entries (name of a directory) | kernel | mfsdirentrycacheto | 1.0 s |
| File entries (name of a file) | kernel | mfsentrycacheto | 0.0 s |
| Negative lookups (name known to be absent) | mfsmount | mfsnegentrycacheto | 0.0 s |
| Symbolic link targets | mfsmount | mfssymlinkcacheto | 300.0 s |
| Extended attributes | mfsmount | mfsxattrcacheto | 30.0 s |
The two zero defaults mean that file names are looked up again every time, but a name resolved to a directory is cached for a second. Setting the options trades staleness for round trips. Beyond the timed caches, mfsmount keeps caches that depend on the traffic rather than on a clock. These caches hold directory listings with their attributes, chunk locations, and the record that a lookup returns. That record lets the following open skip the request to the Master Server. Directory listings arrive with attributes included (readdirplus). A MooseFS Meta mount is listed without them, as what the FUSE layer costs describes. The full list of mount options, these and all others, is listed in command line tools.
Open files are updated by push, everything else by timeout
For files that a mount holds open, the timeouts do not decide staleness: when another session changes a file, the Master Server pushes a notification. The mount then drops the read buffers that cover the changed range, withdraws the kernel's pages for it, and corrects the cached length, as multiple mounts describes. On Linux kernels older than 4.19, a thread of the mount also withdraws expired directory entries from the kernel itself. That thread is covered in client-side background work.
For everything else, the timeouts above set the consistency window: a change made elsewhere can stay invisible here until the longest cache entry that covers it expires. A writer closes the window for open files with fsync() or close(), whose reports trigger the push. That reasoning is explained in multiple mounts.