Length, holes, and truncation
A file's length is a value that the Master Server keeps in metadata, beside the file's ordered table of chunks. The length and the chunk table do not have to agree. The length can point far past the last chunk that exists. A chunk can also hold less data than the length implies. This page describes how writes change the length, why the gaps read as zeros, what truncation does in each direction, and what appending adds on top.
- Blocks of zeros are never written to disk: the chunk file is sparse (HDD_SPARSIFY_ON_WRITE = 1 by default).
- A checksum that does not match makes the copy report as damaged.
Length is metadata, and writes only raise it
Every byte offset maps to a chunk index: the offset divided by 64 MiB, the size of a chunk. The file's chunk table has one slot per index, up to the furthest chunk that the file reaches. A slot can be empty. The length is stored beside the table, not derived from it.
A write changes the length at two moments. When the Master Server grants a client a write into a chunk index past the file's current end, the Master Server raises the length at once. The length goes up to the start of that chunk, before any data is sent. When the client finishes with a chunk, the client reports the furthest byte it has written into the file. The Master Server accepts the report only when the reported byte is larger than the length already stored. The Master Server records the accepted growth in the metadata changelog. Only truncation lowers the length. Reading, overwriting in place, and closing do not change the length. The write path covers the grant, the chain that the data goes through, and the closing report.
A write past the end creates one chunk
A client may write at any offset, however far past the end. The Master Server lengthens the chunk table up to the index being written and leaves every slot in between empty. The Master Server creates a chunk only for the slot being written. An empty slot is a hole: no Chunkserver holds anything for it. The first write to that slot, whenever it happens, allocates a new chunk there, the same way the file's first chunk was allocated.
The same happens inside one chunk, at the scale of its 64 KiB blocks. A Chunkserver stores a chunk's blocks up to the highest block written. A write that skips ahead within the chunk records the skipped blocks as blocks of zeros. Each skipped block gets the checksum of an all-zero block. No data is written for the skipped blocks.
A hole reads as zeros, from two sources
The zeros come from wherever the read reaches the gap. For a chunk index whose slot is empty, the Master Server answers the client's request with "no chunk". The client fills that stretch of the buffer with zeros itself, without contacting any Chunkserver. Inside an existing chunk, the Chunkserver answers. A read of a block past the highest block that the Chunkserver stores returns zeros with a valid checksum without touching the disk. A skipped block below that highest block reads back as zeros from the part of the chunk file that nothing was ever written to.
None of this costs disk space. Missing chunks are stored nowhere. Skipped blocks take no space in the chunk file. The blocks of zeros dropped by the Chunkserver's HDD_SPARSIFY_ON_WRITE option take no space either. The option is 1 by default. The length is counted in full regardless. Both the length figure that mfsdirinfo reports and the length limit that a quota tests include every hole. The size figure counts only the chunks that exist.
Truncating down rewrites at most one chunk
Cutting a file shorter has at most one physical step. When the new length falls on a chunk boundary, there is no physical step: the operation is pure metadata. When the new length falls inside a chunk that the file holds at that index, the part of that chunk past the new length must be removed. The Master Server raises the chunk's version. The Master Server orders every Chunkserver that holds a copy to cut its chunk file to the new length. The order also includes recomputing the checksum of the block that contains the cut. The truncate is answered only after those copies report back, so the call waits for the disks. A copy that the order does not reach keeps the old version. Such a copy is dealt with later, and chunk versions and states describes how. A chunk shared with a snapshot is not cut in place. The writing file gets a duplicate cut to the new length. The duplicate is made on the same copy-on-write path as a write into a shared chunk.
Every chunk past the new end is removed from the file in the same operation. The Master Server goes through the chunk table once, drops each reference at once, and shortens the table. No later pass is needed for the file itself. The chunks stay on the Chunkservers for a while after the last reference to them is dropped. A chunk that no file uses any longer is removed by the mechanisms that deletion describes.
Truncating up allocates nothing
Extending a file by truncation sets the length and creates no chunks. The new stretch is a hole like any other. The only exception works the way the previous section describes. When the new end falls inside a chunk that the file already holds at that index, that chunk is rewritten the same way. The chunk is extended with blocks of zeros up to the block that holds the new end. The rewrite uses a raised version and the same wait for the copies.
The grown length counts at once. mfsdirinfo reports the new length. The growth is tested against the length quota before the operation is applied, so a truncate that writes no data can still be refused. The test is described in quota enforcement.
Appending reserves a range, and mfsappendchunks appends no data
A write through a descriptor opened with O_APPEND gets its offset from the Master Server. The client asks for the range before sending anything. The Master Server keeps a per-file reservation length that starts at the file's length and grows by each request. The Master Server gives each asker the range between the previous reservation length and the new reservation length. Concurrent appenders, through one mount or many, therefore receive ranges that do not overlap. Their records are written whole, one after another. The reservation promises position, not length. The file's length still grows only when the writes complete and report. An unfinished appender leaves a hole where its range was.
mfsappendchunks grows a file without writing at all. The target's length is first rounded up to a chunk boundary. The padding becomes a hole. The source file's chunk slots are then appended to the target's chunk table, empty slots included. Each referenced chunk is shared between the two files from that moment. The first later write into a shared chunk costs a copy. The copy is described in snapshot copy-on-write semantics. A slice of the source's chunks can be appended instead of all of them. The syntax is in mfssnapshots(1), which is listed under snapshot tools.