Chunk versions and states

A chunk stored in several places is only useful if the cluster can tell which of those places holds the current data. For this, MooseFS keeps one number per chunk, the version. MooseFS also records a state for every stored copy, measured against that number.

The word "state" is used for two different things. The states on this page describe whether a stored copy can be used at this moment. The storage states CREATE, KEEP, ARCHIVE, and TRASH answer a different question: which definition of a Storage Class applies to a chunk. The two sets never mix.

The file name and header both carry the version

The chunk itself explains where the version comes from and why the Master Server raises the version before a modification. On the storage side, that raise costs a small transaction on each Chunkserver that holds the chunk. A stored copy is an ordinary file on the Chunkserver's local file system. The page on the Chunkserver covers that file. The version is written twice: as the second half of the chunk file's name, and as four bytes inside the file header, next to the chunk ID. To change the version, the Chunkserver writes the header first and then renames the file. If the rename fails, the Chunkserver writes the old value back into the header, so the name and the header never hold different versions on purpose. If the name and the header still hold different versions, the Chunkserver refuses to read the chunk. The Chunkserver logs which file has a header that does not match its name.

The Master Server raises the version only when a raise is due. The Master Server keeps a flag for each chunk. The flag is set as soon as anything happens to the set of stored copies: a Chunkserver that holds the chunk disconnects, a Chunkserver reports one of the copies as damaged or lost, a new copy appears, or a surplus copy is deleted. The flag is cleared when a version raise completes cleanly. A client that writes to the same chunk repeatedly therefore keeps writing at one version, as long as nothing happens to the copies in between. The raise marks the difference between a copy that took part in the recent writes and a copy that did not. A chunk stored in Erasure Coding is the exception: a modification of that chunk always triggers a raise. The modification returns the chunk to copies. The existing parts are then no longer current, by definition.

A copy is a report at the current version

Every chunk that a Chunkserver reports comes with the version stored on disk. The Master Server compares that number with its own and classifies the report accordingly. The comparison checks for an exact match, not for age. A copy one version behind and a copy a hundred versions behind both have the wrong version. Neither counts toward the chunk's Redundancy Level.

mfsfileinfo shows the result per copy and per Erasure Coding (EC) part, next to the address of the Chunkserver that holds the copy. mfsfileinfo -w shows only the chunks in which the tool finds a problem.

Status	What the Master Server recorded
`VALID`	The Chunkserver reports this chunk at the current version. The copy counts.
`MARKED FOR REMOVAL`	The same, on a disk that its Chunkserver marked for removal. The copy is readable. It is a replication source of last resort, taken only when no valid copy is available. But it is not a place for the data to stay. It no longer counts toward meeting the class's goal.
`WRONG VERSION`	The Chunkserver holds this chunk at a version other than the current one. The data may be perfectly readable and is still not this chunk.
`WRONG VERSION, MARKED FOR REMOVAL`	Both at once.
`INVALID`	The Chunkserver reported this copy unusable. A checksum or a length that does not match, or a read error on the disk, produces this.

The Master Server also tracks conditions that never appear in that list. A copy that is taking part in an operation right now is busy. The operation can be a version raise, a truncation, a duplication, or the creation of the copy by a replication. A busy copy is reported as valid, because the operation is expected to finish. A copy that is still being created is reported as valid before all of its data has arrived. A copy that is already ordered for deletion is left out of the report altogether. The word BUSY that you may see in the GUI or in mfscli marks a Master Server that did not answer a query. The word has nothing to do with chunks.

States of one stored copy. Per-copy states, as shown by mfsfileinfo. — Fig.
**States of one stored copy.** Per-copy states, as shown by `mfsfileinfo`. These are the states of one copy on one disk, not of the chunk. The next figure shows what the chunk as a whole is worth.

Disk errors and disk removal describes what being marked for removal does to a disk, and how a Chunkserver decides that a disk is damaged. Scrubbing and block checksums covers what the Chunkserver checks before the Chunkserver marks a copy as invalid.

A rejected copy is kept while needed

A copy with a wrong version, or an invalid copy, is a candidate for deletion. Such a copy is not deleted at once. The Master Server deletes such copies only when the chunk no longer depends on them. This happens in one of three situations. In the first, the chunk already has as many valid copies or parts as its Storage Class requires. In the second, no file references the chunk any more. In the third, the chunk has something to recover from, but every Chunkserver that could hold a new copy is already occupied. In that case, the Master Server deletes exactly one copy per pass, to free a place for a good copy.

The repair described below depends on keeping those copies. Deleting a rejected copy as soon as it is rejected would turn a recoverable chunk into a lost chunk.

Wrong-version copies alone can still be recovered

When every remaining copy of a chunk has a wrong version, the Master Server tries one repair on its own. The Master Server takes the highest version among those copies and accepts it as the chunk's version, but only when two conditions hold. First, the number of wrong-version copies, whatever their versions, is at least what the Storage Class requires. Second, the highest version is exactly one step away from the version that the Master Server was holding. Copies at that highest version become valid. The remaining copies stay rejected, so the repaired chunk can end up below its goal. The chunk is then replicated back to the goal like any other chunk below its goal. Once the change is written into the metadata, the chunk is readable again. An equivalent repair exists for Erasure Coding. That repair works on the full set of parts that the class defines, all sharing one version. The Master Server logs that it is fixing the chunk. The log names every Chunkserver whose copy the Master Server accepted, and every Chunkserver whose copy the Master Server did not accept.

The narrow condition is deliberate. A distance of one version means that a version raise was interrupted. That raise is a step that the cluster took itself and can undo. A larger gap means that the copies missed writes that really happened. Counting copies cannot tell the Master Server what those writes contained. A related undo covers the case where the interruption is detected during the raise. If a version raise ends with no copy at the new version, and copies exactly one version below remain, the Master Server lowers its own version to match those copies rather than declaring them stale.

Outside those cases, you must repair the chunk. The Master Server logs that the chunk has only copies with wrong versions, or only invalid copies, and asks for a manual repair. mfsfilerepair is the tool for this repair. The tool sets the chunk version to the highest version found on the Chunkservers. Where no copy is available at all, mfsfilerepair marks the chunk as readable as zeros. The rest of the file can then be read, but writes to that chunk keep failing. Reads return the zeros only when every registered Chunkserver is connected. While a registered Chunkserver is absent, the Master Server holds the answer back instead of declaring the chunk lost. A read of that chunk waits in the meantime. mfsfileinfo shows a chunk marked that way. If a Chunkserver that still holds the real data reconnects, reads return the real data again automatically. That reconnection does not clear the mark. The mark clears only with the next version change, so the notice can stay after the condition that it warns about has ended.

Every chunk is graded against its class

The per-copy states describe whether one stored object can be used. The grade of the chunk compares its actual redundancy with the redundancy that its Storage Class requires. The cluster uses the same names throughout its own reporting. Self-healing covers what each grade means for the data, and which grades the cluster repairs:

Grade	What it means
`missing`	No valid copy and no readable set of parts.
`endangered`	Below the required level and down to Redundancy Level 0: one copy, or exactly the number of parts a read needs.
`undergoal`	Below the required level, with redundancy left.
`stable`	Exactly what the class asks for.
`overgoal`	More than the class asks for.
`pending deletion`	No file references the chunk any more. The copies are being removed.

Chunk health: the grades. The redundancy a chunk has, against the redundancy its class asks for. — Fig.
**Chunk health: the grades.** The redundancy a chunk has, against the redundancy its class asks for. The right column says what each grade costs you: only `missing` cannot be read. Only `overgoal` is a bill for space.

The cluster reports these grades as a matrix rather than as a row of totals. The rows of the matrix are the Redundancy Level that each chunk should have. The columns are the level that the chunk actually has. In the combined view, every registered chunk is counted in exactly one cell. The diagonal holds the healthy chunks. Everything below the diagonal is short of redundancy. The leftmost column is before level zero and is labeled missing. That column holds the chunks that are lost. Everything above the diagonal is surplus. The row for chunks that no file references is separate. Both axes stop at Redundancy Level 9. A chunk whose class requires more, or which has more, is counted in the last row or the last column. Such a chunk does not get a row or a column of its own. The last row and the last column therefore represent level 9 and every level above it. The last column is labeled RL >= 9 in both the GUI and mfscli. The last row is labeled RL = 9 in both, but still covers the higher levels. Copy-based and erasure-coded chunks are counted on the same axis. Counting them on one axis is the purpose of the Redundancy Level. The matrix can be narrowed to copies only, to one Erasure Coding width, to one Storage Class, or to one location PRO (PRO Edition only). A second view of the same numbers leaves out the copies on disks marked for removal. Use that view when you are about to remove those disks. If a chunk is stable in one view and undergoal in the other view, its redundancy depends on a disk that you are removing. The matrix is in the chunk matrix table, on the Info tab of the GUI, and in mfscli -SIC.

The matrix is updated continuously. The Master Server updates the counters when reports arrive, so a cell changes as soon as a Chunkserver connects or disconnects. This continuous update is also the only caveat of the matrix. The cluster states the caveat rather than hiding it. While Chunkservers are connecting or disconnecting, the counters describe a set of copies that is still changing. Both the GUI and mfscli mark the matrix as possibly invalid for as long as that lasts.

Missing chunks get a report of their own

Missing chunks get a second, separate report, because the matrix shows how many there are but not which files they belong to. The Master Server produces that report in the pass that goes through the file structure rather than the chunk table. As a result, the report names the file and the position of the chunk inside that file. The report also starts from the last completed pass, not from the current counters. The report can be out of date in one direction only. Each time the list is fetched, the Master Server removes from the list every chunk that has regained a valid copy or no longer exists. A chunk repaired since that pass is therefore no longer shown. The report is in the missing files table, on the Info tab of the GUI, and in mfscli -SMF.

The type of each entry depends on what the cluster still holds:

Type	What is left
`No copy`	Nothing. No connected Chunkserver reports this chunk at all.
`Invalid copies`	Copies exist. Every one of them is unreadable.
`Wrong versions`	Copies exist. They are internally consistent and readable. All of them carry a version other than the current one.
`Missing EC parts`	Some Erasure Coding parts remain. They are too few to reconstruct the chunk.

Where a chunk qualifies for more than one type, the order matters. Wrong versions comes first, then invalid copies, then missing EC parts, and no copy last. The type therefore names the most recoverable thing left rather than the worst. Each type has its own message from the cluster. The alerts reference lists those messages.

The cluster repairs some grades and leaves the rest to you

Restoring redundancy is self-healing. The cluster does this without being asked. Chunks that are undergoal or endangered are queued and replicated. Surplus copies of overgoal chunks are removed. Replication priorities describes the order and the pace of that work.

Three things are outside what the cluster can decide. A missing chunk has no valid source anywhere, so waiting does not produce a source. The data returns when a disk or a server that still holds the data is available again. The data can also come from outside the cluster. Copies whose version is more than one step away from the current version are not adopted automatically. When you run mfsfilerepair, you decide that those copies are the best data available. A chunk that is being written to is locked. A locked chunk is the only case in which an undergoal count is not a symptom of a failure at all.

Learn to recognize that last case, because it looks exactly like a fault. A chunk with a write in progress cannot be replicated. A client that keeps a file open and keeps writing to it keeps renewing the lock. The chunk stays at the number of copies that its CREATE definition produced and is reported undergoal the whole time. The Master Server logs that it cannot replicate the chunk because the chunk is being modified. The log entry also shows how long the lock still has to run. A client that finishes writing releases the lock explicitly. The Master Server then works on the chunk at once instead of waiting for the next pass. If a client stops without releasing the lock, the lock expires by itself two minutes after the last modification. Counts that fall without intervention are normal. Counts that stay the same while an application writes continuously are caused by that application. They do not indicate a broken disk.

The file name and header both carry the version​

A copy is a report at the current version​

A rejected copy is kept while needed​

Wrong-version copies alone can still be recovered​

Every chunk is graded against its class​

Missing chunks get a report of their own​

The cluster repairs some grades and leaves the rest to you​