Scrubbing and CRC verification
A disk can return bytes that differ from the bytes that it was given, without reporting any error. To detect this, MooseFS stores a checksum with every block of every chunk. A set of paths compares the checksum with the data. MooseFS also runs a background pass (the chunk scrubbing of this page's title) that keeps reading the data that nothing else reads.
The header holds a four-byte checksum per block
A chunk file on a Chunkserver's disk starts with an 8 KiB header. The chunk data follows the header. The header consists of two halves of 4 KiB. (A chunk written by an old enough MooseFS has a 1 KiB identity half instead. The Chunkserver reads both layouts. This page describes the current layout.) The first half holds the file's identity: a format signature, the chunk ID, and the chunk version. The second half is the checksum table: a four-byte value for every block that a chunk can hold, in block order.
The table is always written in full, whatever the chunk contains. A chunk that reaches only one block therefore has the same 4 KiB of checksums as a full chunk. Entries past the last block that the chunk reaches hold the checksum of a block of zeros. When the Chunkserver loads the table of a current-format chunk, the Chunkserver checks that those entries still hold that checksum. The Chunkserver also accepts an all-zero entry. A read of one of those blocks is answered with zeros from memory, without reading the disk.
Each value covers one whole block, padding included, and nothing else. Neither half of the header, the table included, has a checksum of its own. A corrupted table entry therefore looks exactly like corrupted data and produces the same result. The identity half is checked differently: it has to match the chunk file's name.
These checksums belong to the Chunkserver and cover chunk data on disk. A Master Server keeps a number over its own metadata. That number is a separate mechanism with a separate purpose, described in Metadata ID, version, delay, and checksum.
Nearly every full-block read checks the checksum
| Path | What happens to the checksum |
|---|---|
| A client reads | The Chunkserver reads the whole block, computes its checksum, and compares it with the table. The Chunkserver also returns the checksum of the range that the client asked for. The client compares that against what arrived, so the network hop is covered as well. |
| A chunk is replicated | The source Chunkserver serves each block through the same read path, so every block is compared against its own table before it goes out. The receiving Chunkserver checks the block against its checksum before writing it and once more afterwards, from the same buffer, so memory that changed in between does not pass. |
| A client writes part of a block | To merge the write into the block, the Chunkserver reads the whole block and compares it against the table first. The entry is then recomputed from the merged result. |
| A chunk is split into Erasure Coding parts | Every block is read and compared against the table before parity is computed from that block. |
| A chunk moves between disks of one Chunkserver | The header and checksum table go to the new disk first. Every block is then compared against the table as the block is read, and written only if it agrees. A mismatch abandons the half-written file, which is removed. |
| The background test runs | Every block that the chunk has is read and compared. Nothing else is done with the data. |
One operation moves chunk data without comparing anything. A chunk can be shared by a file and a snapshot of that file. On the first write to such a chunk, the Chunkserver makes a local copy of the chunk file. The data and the checksum table are copied unchanged. The Chunkserver neither verifies nor recomputes anything at that moment. A block that was already wrong therefore stays wrong. Its checksum still describes what the block should have been. Truncating a chunk to a length that falls inside a block is different. Since 5.2.0, the Chunkserver first reads that whole block and compares it with its expected checksum. If they do not match, the truncation fails with an I/O error, the file keeps its length, and that copy is reported as damaged. Only after a match does the Chunkserver pad the surviving part with zeros and store the checksum of the result. A duplication with a truncation applies the same check to the block where the cut falls. The blocks before it are copied unverified. Up to 5.1.0, the block is not compared, and whatever the read returned becomes the new expected value for that block.
The background test reads at a fixed rate
A Chunkserver runs one thread that only reads chunks and checks them. On Linux, the thread runs at the lowest CPU priority and in the idle I/O class. On other systems, the thread runs at the priorities that it was started with. HDD_TEST_SPEED sets how fast the thread reads, in MiB/s per disk. The default is 1.0. The option is in the Chunkserver configuration file. Setting HDD_TEST_SPEED to 0 switches the test off. The Chunkserver logs that it does not recommend this setting. If a configuration still has the obsolete HDD_TEST_FREQ and no explicit HDD_TEST_SPEED, the rate is derived from that old option instead. The thread takes one chunk at a time from one disk and reads every block in that chunk. The thread then does not return to that disk until the time that those bytes take at the configured rate has passed. That wait is counted from the start of the read, so the read itself uses part of the interval. Each disk has its own next turn, so the rate applies per disk even though a single thread does the work.
The pace is therefore measured in bytes rather than in chunks. A chunk that holds a quarter of its blocks takes a quarter of the time that a full chunk takes. Nothing else sets the pace. A full pass over a disk takes the data that the disk holds divided by that rate. At the default, that is eleven to twelve days per terabyte. Corruption in data that nothing else reads is therefore found on that timescale, not on the day that it appears. To shorten that timescale, raise the rate and give the test more of the disk's time.
The test skips unhealthy disks and recent chunks
The test skips a disk entirely while the disk is damaged, marked for removal, being removed, or still scanning its contents after a start. A disk on a read-only file system counts as marked for removal.
HDD_MIN_TEST_INTERVAL keeps the test away from chunks that are already being read or written. A chunk whose last I/O, the last test included, is more recent than that interval is skipped and counted as done for this pass. The default is one day. A chunk locked by another operation when its turn comes is left in place and tried again. After five misses in a row on that disk, the chunk is skipped as well. A chunk already reported damaged is skipped and stays skipped for as long as the Chunkserver runs. The mark is kept in memory, so after a restart the chunk is tested and reported again if it is still there and still damaged.
Each disk rotates, and a busy chunk waits
Each disk keeps its chunks in two lists: a list of chunks still waiting for this pass, and a list of chunks that have already been tested in this pass. The test takes the first chunk in the waiting list and moves that chunk to the other list. When the waiting list is empty, the two lists swap roles. That starts the next pass. The order of the very first pass after a disk's scan is shuffled on purpose. Opening a chunk for any read or write moves the chunk to the back of its list and records the time of that access.
A pass therefore follows neither chunk ID order nor the order of age. A pass is a rotation. Chunks that are in use keep moving towards the end. Chunks that nothing accesses move towards the front. That is the order that the test needs. Data that is read often is checked by being read. The test exists for data that is never read.
A mismatch invalidates the copy and reports it
When a comparison fails, the Chunkserver logs the file and the block number, and writes the failure into that disk's error record. The Chunkserver marks the chunk as damaged so that the test stops returning to that chunk. The Chunkserver also puts the chunk ID on a list for the Master Server. The Chunkserver sends that list on its next turn through its event loop, not on a schedule, so the report is immediate. The error record does not bring the disk closer to being taken out of use. The tolerance threshold counts only errors that the operating system itself returned. The page on disk errors and disk removal describes that boundary.
The Chunkserver neither deletes the file nor tries to repair it. A four-byte checksum shows that the block is not what was stored. The checksum does not show what was stored, so nothing on that disk can rebuild the block.
The Master Server records that copy as INVALID at version 0 and re-grades the chunk. The new grade puts the chunk in the queue to be rebuilt from a copy or a set of parts that is still good. Replication priorities covers how that queue is ordered and what limits how fast the queue is emptied. The damaged file stays on the disk until the Master Server orders it deleted. Chunk versions and states describes when that order comes and what happens to a chunk whose copies have all been rejected.