Multiple mounts and parallel access

The same MooseFS file system is mounted as many times as needed. There can be several mounts on one machine and mounts on many machines, all attached to the same namespace at once. This page describes what each mount has of its own, how writes from one mount reach every other mount, and how writes from two mounts to the same file are kept apart.

Each mount has a session of its own

Every mount registers with the Master Server separately and receives its own session, even when another mount is already on the same machine. Two mounts share nothing. Each mount holds its own connection to the Master Server, runs worker threads of its own, fills caches of its own, and has a throughput ceiling of its own. Client and mount covers what a client is and the forms it takes.

The session is the whole record of the mount on the Master Server. The record includes the rights that the mount was admitted with, the files that the mount holds open, and its operation counters. A mount that disappears without unmounting leaves that record behind, open files included, for the time described in session and export. Holding a file open also affects deletion. A file that is deleted while any session holds the file open remains as a sustained file until every such session closes the file.

Mounts multiply streams and share one Master Server

File data goes directly between a mount and the Chunkservers. Every added mount therefore adds streams of its own, on the same machine or on any other machine. Many mounts share the cluster itself: the Chunkservers and disks that hold their data, and the single Master Server that answers the metadata operations of every mount. Adding mounts multiplies the data paths. There is still only one metadata path, so added mounts add load to that path without adding capacity.

A mount sees another mount's write once reported

A write goes to the Chunkservers as a transaction. The writing mount ends each transaction with a report to the Master Server. The Master Server then notifies every other session that holds the file open. The notice names the chunk that changed and the changed range, or gives the new length when only the length grew. The notified mount drops its read buffers over the changed range and withdraws the kernel's cached pages for that range. The mount also corrects the length that it holds, so its next read fetches current data. No polling and no cache timeout is involved. The push arrives as soon as the report does.

Data is in the writer's cache first. Even a transaction that has already started is reported only when it ends. A transaction ends on flush, on fsync(), on close(), and, inside a long write, on the schedule that client-side background work describes, about once a minute. Only fsync() or close() on the writing mount guarantees that other mounts read the complete and correct data. Either call pushes out everything still waiting and ends the transactions. The end of those transactions is the report. Between write() and that call, what another mount reads is undefined. The reading mount can get the old length or the new one, and with the new length either the correct data or garbage. A longer wait between the write and the read makes correct data more likely, not certain.

The push covers open files only. A mount that does not hold the file open gets the changes in a slower way, after its timed caches for attributes and directory entries expire. Those caches and their timeouts are covered by client cache and read-ahead.

Writes to one chunk take turns

The Master Server grants a chunk to one write transaction at a time. The grant locks the chunk. The lock is released with the end-of-transaction report, or after 120 seconds when no report comes. A mount that dies mid-write therefore blocks a chunk for at most that long. A second mount that asks to write the locked chunk is told that the chunk is locked. That mount then asks again, first after a millisecond, then after a tenth of a second, then every half second. Only the first few of these retries count against the budget that turns repeated errors into an I/O error. After those retries, the counter stops, so a waiting write waits until any lock is released instead of failing. A read that has to ask the Master Server about a locked chunk waits the same way. A mount that already has the location of the chunk keeps reading from the Chunkservers directly.

The lock is per chunk, so two mounts writing different chunks of one file do not block each other at all. A mount that flushes a long write also does not lose the chunk between its own transactions. The mount marks the next request as a continuation of the previous request. The Master Server then admits that request past the lock. Reading takes no lock of its own. Any number of mounts can read one file at once. A reader holds nothing that any other reader, or the writer, waits for.

The Master Server places appends, so they stack

For a write to a file opened with O_APPEND, the offset does not come from the length that the mount holds. The mount asks the Master Server to reserve the write's size past the end of the file. The Master Server answers with the offset at which the write starts. The reservations advance a running end of file that the Master Server holds. Each reservation starts where the previous reservation ended, so appends from any number of mounts stack one after another instead of overwriting each other. Truncating the file drops its outstanding reservations.

The chunk lock and the append reservation serialize MooseFS's own steps, nothing above them. They do not order the applications' writes or separate the data of different mounts. Two programs that share a file coordinate with each other through the file locks that the Master Server holds across all mounts.