Skip to main content

Self-healing

Self-healing means that the cluster returns to a healthy state on its own after a disk, a Chunkserver, or a Master Server fails. Self-healing covers both data and metadata, but not equally far.

What it is​

Redundancy drops by itself. It drops when a disk fails, when a Chunkserver is switched off, or when a stored copy turns out to be unreadable. A chunk that satisfied its Storage Class an hour ago no longer does. Metadata has two failures of its own. A Master Server stops without writing down what it held, or the copy that a Follower PRO (PRO Edition only) keeps stops matching the Leader's. Self-healing covers both. For data, MooseFS compares what each chunk should hold with what the Chunkservers report holding, and corrects the difference on its own. MooseFS names the results of that comparison. The same names appear in the GUI, in mfscli, and throughout this documentation:

  • undergoal: fewer copies or Erasure Coding parts than the class requires. The chunk is fully readable. The cluster creates the missing copies or parts.
  • endangered: no redundancy left (a single copy, or exactly as many parts as reading needs). The chunk is readable. One more failure loses it.
  • overgoal: more copies or parts than the class requires. The surplus costs disk space. The cluster deletes the surplus.
  • missing: no valid copy and no readable set of parts left. This is data loss, not a state that the cluster repairs.

Why it matters​

Self-healing splits the work between you and the cluster, and splits it differently for data and for metadata. Hardware failure is expected. The cluster handles the consequence for the data without an operator. Your part is capacity and hardware. The cluster decides which chunk to copy where and in what order. The states above are therefore a health reading rather than a task list. Normally, the counts of undergoal and endangered chunks fall on their own. Counts that stay the same mean that the cluster is unable to repair, not that it is not trying. For metadata, the cluster repairs only part of the damage on its own. A Follower PRO (PRO Edition only) that is not synchronized with the Leader resynchronizes on its own. A lost Master Server does not. Neither does metadata that no Metalogger or Follower kept a copy of. Which components are redundant and which are not tells you which failure you still have to plan a procedure for.

How MooseFS implements it​

The Master Server makes every decision about data. The Master Server holds the list of copies and parts per chunk. It keeps that list current when Chunkservers register, report a damaged copy, and disconnect. A chunk whose list stops matching its Storage Class goes into a queue of chunks that need work. Independently of that queue, the Master Server checks every registered chunk. A chunk that nobody reported anything about is therefore reviewed in its turn as well. The Master Server orders repairs by what is at stake, not by the order in which the problems arrived. Work that a client is waiting for comes first, chunks one failure away from being lost next, undergoal chunks after them, and reclaiming space nearly last, ahead only of fixing copy placement. Replication priorities covers the whole order and what changes it. For each repair, the Master Server sends an order to a Chunkserver (fetch this chunk from a server that has it, or delete this copy). The data itself moves between Chunkservers.

Metadata is repaired between Master Servers rather than inside one Master Server. A Master Server writes the whole structure to its own disk and records every change in between in a changelog. At a normal start, the Master Server reads back the image that it last wrote. Rebuilding the current state from an older image plus the changelogs is a separate starting mode. A Master Server that finds its image missing or unreadable stops and reports which starting mode would rebuild the current state. The step after a crash is therefore left to an operator. Between Master Servers, the repair is automatic. Each Master Server stores a checksum beside every metadata image that it saves. The Leader PRO (PRO Edition only) compares the Followers' checksums against its own on a schedule. A Follower whose checksum differs, or that can no longer follow the changelog, discards everything it holds and downloads the metadata from the Leader again. A Follower reports how far it is from the Leader as part of its state, in the labels that leader election lists. Community Edition has none of this comparison, because the edition runs a single Master Server.

Defaults and limits​

Self-healing restores redundancy. Self-healing does not create the resources that redundancy needs. Three conditions stop self-healing on the data side. The cluster clears none of them on its own:

  • Free space. A Chunkserver with almost no free space left is not chosen to receive a copy.
  • Enough Chunkservers. Every copy and every part needs a Chunkserver of its own. A class requiring more than the cluster has, or more than there are Chunkservers matching its labels, therefore stays unmet however long you wait.
  • A valid source. A missing chunk has no valid source anywhere in the cluster. To get the data back, you have to get back a disk or a server that still holds the data, or restore the data from outside the cluster.

One case of a lasting undergoal count comes from none of these conditions. A chunk that is being written to is locked and is not replicated. Chunk versions and states covers this case among the cases that repair does not touch. Metadata has one boundary of the same kind. A Master Server that cannot write its metadata to disk does not carry on without a saved copy. The Master Server tries a list of fallback locations (this is Emergency mode) and then stops, so that a failing disk that holds the metadata cannot pass unnoticed.

  • Chunk: the object that is repaired.
  • Copy: the redundancy format restored by transferring an existing copy.
  • Erasure Coding: the redundancy format restored by recomputing a part.
  • Redundancy Level: the target that the current state is measured against.
  • Storage Class: where that target comes from.
  • Maintenance mode: how you stop the cluster from repairing after a Chunkserver you took down yourself.
  • No single point of failure: the same property stated for components rather than for data.