The read path
A read is two conversations. The client asks the Master Server which chunk holds the bytes and where that chunk is stored, then fetches the data from a Chunkserver directly. This page describes one read through both conversations. Client cache and read-ahead covers what the client keeps in memory for both and how it reads ahead.
- Own location first (PRO), then nearer before farther by topology distance.
The client caches the answer per chunk, with no expiry. A failed read, or any Chunkserver that joins or leaves, empties that cache. A change notification corrects the entry in place.
The Master Server never transfers file data: its reply names where the copies are. The bytes are sent only between the client and the Chunkserver.
The Master Server answers with the chunk locations
Each chunk covers 64 MiB of the file, so the client turns the file offset into a chunk index. The client sends the file's inode and that index to the Master Server as a CLTOMA_FUSE_READ_CHUNK message. The answer comes from metadata held in RAM. The answer includes the current file length, the chunk's ID, its version, and the list of Chunkservers that hold a copy. For each Chunkserver the list gives its address, its port, its software version, and its labels.
The Master Server sorts the list before answering. In an instance divided into locations PRO (PRO Edition only), the Chunkservers in the client's own location come first. Each group is ordered by topology distance, nearer before farther, at random among equals. The Master Server does not refuse a request that arrives while another operation runs on the chunk. The Master Server delays the reply until that operation ends.
One Chunkserver sends the data in checksummed blocks
The client reorders the list once more on its own side, which is where preferred labels take effect. The client then connects to the first Chunkserver of the result, or reuses an already open connection to that Chunkserver. The client sends a single CLTOCS_READ command naming the chunk ID, the version, and a byte range within the chunk, on the same port all chunk I/O uses. The command includes an authentication code where the cluster requires it.
A Chunkserver whose copy is of a different version refuses the command. Otherwise the Chunkserver streams the range as blocks of up to 64 KiB, each with its checksum, and closes the exchange with a status message. The client verifies every block's checksum when the block arrives. The full chain of checks along this path is described in scrubbing and CRC verification. While the transfer runs, the client sends a keepalive every second. Five seconds without a byte from the Chunkserver end the attempt.
An erasure-coded chunk arrives as several streams
The Master Server answers differently for a chunk stored in Erasure Coding format when there is no whole copy to serve. The same applies when every whole copy is outside the client's location but a complete set of data parts is inside it. The list then names the Chunkserver of each data part, four or eight of them. The position in the list is the part number. The client connects to all of those Chunkservers and asks each for its part's share of the range. The client reads the streams in parallel and places each arriving block at the position in the chunk that its part number gives. Erasure Coding layout describes how the parts divide a chunk. The client does not read parity parts on this path. For a chunk that is missing a data part, the Master Server answers with an empty list instead. The client retries while the Master Server rebuilds the missing part as its most urgent repair.
The client caches the answer until doubt empties it
The client keeps the answer per file and chunk index, with no expiry time, so the next read of the same chunk skips the Master Server round trip. The client removes an entry when the entry may be wrong, not because of its age. A failed read drops the entry. When a write or a truncation by someone else changes the chunk, the Master Server notifies every client that holds the file open. The notification corrects the cached chunk ID and version in place. When a Chunkserver connects or disconnects, the Master Server tells every client to drop this cache entirely, because any entry may now name a server that has disconnected or omit a server that is available again. The client also empties this cache whenever its connection to the Master Server is lost. The client's other continuous bookkeeping is described in client background work.
A failure sends the client back
An attempt can end early in several ways: a connection that cannot be made, five seconds of silence, a version mismatch, a checksum that does not match, or an error status from the Chunkserver. The client handles each of them the same way: it drops the cached entry, waits, and asks the Master Server again. The next attempt then uses the current answer of the Master Server instead of a list that already failed. The wait increases with the number of attempts. The request as a whole has a budget of mfsioretries attempts, a mount option set to 30 by default. When the budget runs out, the application gets an I/O error. The exact retry schedule is described in client background work. The Master Server declares a chunk definitely lost only when every Chunkserver is connected and none of them holds the chunk. Such a chunk fails with ENXIO when the budget runs out, or immediately on a mount that opted in with mfserroronlostchunk.
A hole is read from no server
An offset that no chunk covers reads as zeros that the client writes into the buffer itself, with no Chunkserver involved. Length, holes, truncation describes where such holes come from.
The Master Server never touches the data
The Master Server's part of a read begins and ends with metadata. File data makes a single network hop, from a Chunkserver to the client. No component relays that data. The Chunkservers carry the traffic. The Master Server only names where the copies are.