Skip to main content

Client-side background work

Between the calls that an application makes, mfsmount is not idle. Threads of its own keep the Master Server session alive and push buffered writes out. They also fetch data ahead of reads, apply invalidations that the Master Server sends, and give back what the mount no longer uses. This page describes those loops, because they decide what you observe: when written data really leaves the machine, what a short network break costs, and how another mount's changes reach yours. Client and mount covers what a client is and the forms it comes in.

Two threads service the Master Server connection​

The mount keeps one connection to the Master Server and sends every metadata request through that connection. One thread receives every answer and passes it to the operation that waits for it. A second thread wakes every second and sends the data that belongs to no operation:

  • A keepalive, when nothing else has gone out for two seconds.
  • The mount's traffic counters: bytes and operations read and written since the previous report. The Master Server adds them to the per-mount statistics that it serves.
  • The access and modification times that this mount's reads and writes produced, batched into one packet. The mount stamps them with the Master Server's clock, which the mount requests once a minute. What these timestamps mean is described in timestamps.
  • Once a minute, the list of inodes that this mount holds open.

The open list makes a session hold files. Because of this, a file that is deleted but still open becomes a sustained file. The list includes more than the open descriptors. Every 30 seconds, a scan of the processes on the host adds every directory that a local process uses as its working directory. The most recent 5000 closed inodes stay on the list as well.

The list also removes entries with a delay. A closed inode stays on the list for ten more of these reports before it is dropped. The Master Server therefore continues to count the file as open for about eleven minutes after the application closed the file. The delay is deliberate and costs little. While the inode stays reserved, stray I/O operations that still arrive for the file after it was closed are easier to handle. Nothing that an application does shortens that delay. After close() returns and the descriptor no longer exists, the cluster still answers that a session holds the file. Unmounting does shorten the delay, because unmounting ends the session and the whole list at the same time. Every decision that the Master Server makes about whether a file is open therefore lags behind the application by that much. A reader sees that lag in the two decisions that the Master Server makes about the trash.

The mfs* tools use the same connection. The mount listens on a local proxy address. The address is published to the tools through the .masterinfo metafile. The mount forwards the commands of the tools over its own session.

A short break stalls operations, not fails them​

When the connection drops, the receive thread handles the reconnection. Every two seconds, the receive thread tries to register again under the same session ID. The session record stays on the Master Server, with its open files and its rights. Session and export covers how long the record is kept.

Operations that are in progress when the connection drops are woken with an internal error. They are sent again once the connection is available again. The delay between attempts grows by 0.3 seconds each time, up to 10 seconds. The budget is mfsioretries attempts, 30 by default. With this budget, an operation keeps retrying through an outage of about two minutes before the application receives an I/O error. mfstimeout instead limits the total time of one operation, in seconds. By default mfstimeout is 0, so there is no limit. When the Master Server no longer recognizes the session, waiting operations fail at once. The mount then registers as a new session.

On every reconnection the mount discards what it cached about chunk locations, because the mount cannot determine what changed during the outage. The client's caches as a whole are described in client cache and read-ahead.

Writes go out on the workers' schedule​

write() copies data into the mount's write cache, 256 MiB by default (mfswritecachesize), and returns. A pool of write worker threads empties the cache in the background. Sending starts as soon as data is in the cache. A block that is not yet full is held back for 0.05 seconds in case the next write completes it. How the data then goes down the Chunkserver chain is described in the write path.

The size of the pool follows the load. One worker exists at startup. Whenever a job comes in and no worker is free, another worker is started, up to 250. Idle workers beyond the fiftieth exit instead of waiting. One file is written through at most 16 chunks at a time. The FUSE pipeline sets a ceiling of its own on how much one mount can send. This ceiling is covered in per-mount throughput.

fsync() does not start any transfer that was not already in progress. It stops holding back blocks, sends every block that is still waiting, and returns when the Chunkservers have confirmed all of them. Closing a file waits the same way, so by default close() returns with the file's data stored. A write error that is still pending is returned by close(). When mfsfsyncmintime is set, the wait is skipped for a file that is closed sooner than that time after it was opened. The blocks of that file are still sent after the close.

A write that fails is retried with the same increasing delays and the same mfsioretries budget as a metadata operation. A chunk with no copy that can be used for writing is retried once a minute. Because the sending is asynchronous, the error that remains after the budget is used up is not returned by the call that buffered the data. The next operation on that file returns the error: a later write(), fsync(), or close(). A Chunkserver that sends nothing for 5 seconds during a transfer counts as failed. The attempt is then repeated.

A long write inside one chunk does not hold its transaction open to the end. The worker closes the transaction after about a minute, or after 5 seconds when the pool has more than 150 workers. The worker then reports the length reached to the Master Server and queues the rest as a new job. With that report, other mounts that hold the file open receive the information that the file has grown.

Read-ahead has a worker pool of its own​

Read requests are queued for a second pool of workers. These requests include the read-ahead that the mount decides to attempt. This pool has the same sizes: one worker at startup, up to 250 under load, and fifty kept. When the read-ahead fetched the right data, the data is already in memory before the application requests it. When the read-ahead fetched data that is not needed, the fetched buffers are dropped unused. What is read ahead, when, and how much memory it may hold are described in client cache and read-ahead. Errors and retries work as they do for writes, except that the error is returned by the read() that needs the data.

A worker chooses the copy to read by two sort keys: preferred labels first, then the number of unfinished operations that this mount has on each Chunkserver. That number comes from a private count that the workers update when operations start and finish. The count includes only the traffic of this mount. Its entries are kept as long as the mount exists.

Invalidations arrive on a thread of their own​

For every file that this mount holds open, the Master Server reports what other sessions do to that file: a chunk rewritten, a length changed. A dedicated thread applies each report. The thread drops the read buffers that cover the changed range, invalidates the kernel's cached pages for that range, and updates the cached length. Because the Master Server sends these reports, two mounts that share an open file see each other's writes without waiting for any cache to expire.

No loop goes through the timed caches. Attributes, directory entries, negative entries, symlink targets, and extended attributes are evaluated at the moment they are used, against the timeouts listed with the mount options. Two small cleaners are the exception. One cleaner discards expired entries of the supplementary groups cache. On Linux kernels older than 4.19, a second cleaner actively removes directory entries from the kernel's cache once their timeout passes, 30 seconds at the earliest.

Idle Chunkserver connections stay open and get keepalives​

A finished transfer does not close its connection. The socket is kept open instead, in a pool of up to 200 sockets. Once a second, a sweep sends a keepalive through every socket in the pool and drops the sockets that the other end has closed. A later read or write to the same Chunkserver uses a socket from the pool and skips the connection setup. When the pool is full, the socket that has been unused for the longest time is closed to make room.