The write path
A write becomes durable in three steps. The client asks the Master Server for a grant on one chunk. The data passes through a chain of Chunkservers. Finally the client reports back to the Master Server so that the change becomes visible. The Master Server arranges the first and the last step. The bytes themselves never pass through the Master Server. MooseFS calls the middle step a chain-like write: every copy is written in the same pass, while the Chunkservers pass the data down the chain.
- The grant creates, duplicates, or joins the chunk from EC parts and raises its version. A WRITE entry is written to the changelog.
- LENGTH + UNLOCK: the chunk lock is released.
The Master Server never transfers file data. The client sends each block once: the chain fans each block out, so the uplink carries one copy's worth regardless of the redundancy.
The Master Server grants the chunk first
The client gathers written bytes into 64 KiB blocks of its write cache and groups them by chunk. The mount sends the blocks on its own schedule. A write call returns as soon as the bytes are cached. For each chunk with pending blocks, the client asks the Master Server for a write grant. The grant does whatever the chunk needs before it may be written:
- A chunk that does not exist yet is created. The Master Server picks the Chunkservers for its copies, as described in allocation and space selection, and orders those Chunkservers to create the chunk.
- A chunk that another file also holds, after a snapshot, is duplicated first, as described in snapshot copy-on-write.
- A chunk stored as Erasure Coding (EC) parts is joined back into whole copies, as described in EC part layout. The grant waits for the join.
- A copy may have missed the last change: the chunk is newly created, a server holding a copy disconnected, an earlier operation was interrupted, or leftover EC parts remain. In each of these cases the Master Server raises the chunk version and tells every reachable copy the new number.
The Master Server refuses a write that would exceed a quota at this point, before any chunk work starts. Otherwise the Master Server records the grant in the metadata changelog as a WRITE entry and waits for the Chunkservers to confirm the work above. The Master Server then replies with the chunk ID, the chunk version, and the list of Chunkservers that hold the copies.
From the grant on, the chunk is locked for this writer. A second mount that asks to write the same chunk waits its turn, as described in multiple mounts. The lock is a deadline rather than a flag: if a dead client leaves the lock unclosed, the lock expires on its own after 120 seconds.
The chain carries data down and status back
The client first orders the list so that servers matching the mount's preferred labels come first. The client then connects only to the first Chunkserver on the list. The opening command includes the rest of the list. Each Chunkserver takes the next server off the list, connects to that server, and passes the remainder on. This is how the chain is built. The command also includes an authentication code issued by the Master Server. Each server in the chain checks that code, as described in authentication between modules.
The client sends each block with a checksum that it computed. Every Chunkserver in the chain verifies the block against that checksum before writing it and refuses the block on a mismatch. The Chunkserver stores the checksum next to the block in the chunk file, as described in scrubbing and CRC verification. A server likewise refuses a block that names a chunk version other than the version it holds.
A status goes back along the chain, one per block, over the same hops that the block came down. A server acknowledges a block to its predecessor only when two things have happened: its own disk accepted the block, and its successor acknowledged the same block. The status that the client receives for a block therefore comes from the first server of the chain and covers every copy at once. There is no primary copy that fills in the remaining copies later. The copies exist as soon as the write does.
The client does not wait for that status before sending the next block. Each block has a write ID of its own. The status names the ID of the block that it answers. The client keeps up to 64 blocks outstanding on one chain. Blocks therefore go down the chain while the statuses of earlier blocks are still on their way back.
The client closes the transaction
When every block is acknowledged, the client tells the chain that the write is finished. The client then reports to the Master Server that the chunk is done, together with the file length that the write produced. If the file grew, the Master Server extends the file and writes a LENGTH entry to the metadata changelog. In both cases the Master Server records an UNLOCK entry for the chunk. The Master Server then releases the chunk lock and notifies other sessions that hold the file open. The write therefore becomes visible according to the rules in multiple mounts.
A transaction also closes after a time limit: one chain accepts new blocks for about 65 seconds at most. After that, the client closes the transaction and continues under a new grant. That pacing, and what fsync and close force, are described in client-side background work.
A failing Chunkserver ends the transaction, not the write
The client does not repair a chain. An error status that comes back, a reset connection, and five seconds without anything received from the first server all end the same way. The client closes the transaction, reports to the Master Server, and asks for a new grant after a pause. The new grant builds the chain from the copies that are still reachable and raises the chunk version again. The copy on the failed server missed that raise. When the server reconnects, the copy is therefore rejected as wrong-version. The chunk is repaired from the remaining copies, as described in chunk versions and states.
The client spaces failed attempts further and further apart, up to 30 attempts by default (the mfsioretries mount option). After the last attempt, the application receives an I/O error. Errors that retrying cannot fix, such as an exceeded quota, reach the application immediately.
Two things are not on this path
The Master Server never transfers file data. The Master Server grants, records, and unlocks over the client's metadata connection. The data moves only between the client and the Chunkservers, the same split as on the read path.
The client never writes to the copies in parallel. It sends the data once, to one Chunkserver. The chain fans the data out: the client's uplink carries one copy's worth of traffic, regardless of how much redundancy the file's Storage Class requires.