Skip to main content

The chunk loop and the file loop

Setting a policy in MooseFS moves no data at the moment you set it. Deleting a file does not free its space at that moment. Redefining a Storage Class does not move a chunk. An archive rule has no clock of its own. The Master Server does all of this work later, in two background scans. The file loop is the periodic scan of all files. The chunk loop is the periodic scan of all chunks. This page describes what each loop does in one pass and how long a pass takes, because the pass time is the delay between setting a policy and seeing its effect. In the PRO Edition PRO (PRO Edition only) both loops run only on the Leader.

The file loop decides, the chunk loop acts​

The two loops divide the work by what a decision is based on. The archive rule of a Storage Class measures a file: its length, its timestamps, and its class. The file loop applies that rule. The file loop records the result as a flag on the chunks of the file. The chunk loop measures a chunk against its class: which copies and Erasure Coding parts exist, on which Chunkservers, in which versions. The file loop itself sends nothing to a Chunkserver. The orders for this background work come from the chunk loop and from the repair queues that both loops add chunks to. An operation started by a client sends its own orders, outside either loop. The file loop marks. The chunk loop moves.

An archive transition shows the whole chain. The file loop reaches the file, finds the rule satisfied, and sets the archive flag on the chunks of the file. The changed flag adds each chunk to the repair queues at once. The queues are processed at a much faster rate than either loop. The wait is therefore set by the file loop. The chunk loop finds the same difference on its next pass only where the queues have not already resolved it.

One file-loop pass marks, finds, and audits​

The file loop goes through every inode. For each file, including files waiting in the trash and deleted files that a client still holds open, the loop does the following:

  • It applies the archive rule of the file's class. The rule gives one result for the file, except in per-chunk mode, where each chunk is measured against its own last write. The archive flag is then set on the chunks that the result covers. The rule, its modes, and its edge cases are described in Storage Class evaluation.
  • It clears the trash flag from a chunk that should not have it: a chunk shared with another file, or a chunk whose file is no longer in the trash. What sets the flag is described in trash retention.
  • It records every chunk left without a usable copy, together with the reason. The reason is one of these: no copy at all, only invalid copies, only copies with a wrong version, or an incomplete set of Erasure Coding parts. The missing files report is rebuilt from these records on every pass. A file is counted as missing whether it is a regular file, a file in the trash, or a file held open after deletion.
  • It removes a file's reference to a chunk that is past the end of the file and has no valid copy anywhere. This is the only structural repair that the loop performs itself.
  • It recounts the totals that the loop reports: files and chunks, files and chunks below their class (undergoal), and the missing ones. It also updates the raw space charged to a file whose chunks changed state. For this reason, per-directory raw usage shows an archive transition with a delay.
  • It cross-checks the links between files, directories, and directory entries. It reports every mismatch as a structure error. The report of the loop is titled "Filesystem self-check" because of this audit.

The file loop does not empty the trash. A separate sweep, covered in trash retention, purges files whose retention has expired. The file loop also does not free the space of deleted files. The chunk loop does that work. Deletion describes the whole chain from unlink to reclaimed space.

One chunk-loop pass compares and schedules the work​

The chunk loop goes through the chunk registry. The chunk loop schedules replication, deletion, format conversion, and state transitions. The loop orders the work. Chunkservers perform it. For each chunk the loop does the following:

  • It discards records of copies held by Chunkservers that have disconnected. It counts what remains: valid copies, invalid copies, copies with wrong versions, Erasure Coding parts, and the Redundancy Level that they add up to.
  • It recalculates which class governs a chunk shared by files with different classes, so a class change applies to existing shared chunks within one pass. How the governing class is chosen is described in class priority.
  • It deletes the chunk's record once no file refers to it and no copy remains.
  • It repairs a chunk whose valid copies no longer exist but whose wrong-version copies still hold the data. It also orders the deletion of invalid and wrong-version copies and parts that a healthy chunk no longer needs. What the states mean is described in chunk versions and states.
  • It cancels a replication that has stalled past its deadline, as replication priorities describes.
  • It compares the copies and parts that exist against the definition governing the chunk. Where the stored state and the definition differ, the loop schedules the work that removes the difference. It schedules a replication when the chunk is below its class. It schedules a conversion between copies and Erasure Coding parts when the format is wrong (the conversion operations are described in Erasure Coding layout). It schedules a deletion when there is a surplus. It schedules a move when copies are on Chunkservers that the labels of the class exclude. A chunk that needs urgent repair is added to the priority queues. These queues are processed at a much faster rate than the loop. The queues and their order are described in replication priorities.
  • It deletes copies and parts from a location whose goal in the governing class is zero PRO (PRO Edition only).
  • Last, when no repair is pending anywhere, it considers the chunk for rebalancing.

Every sixteenth pass, the Master Server adjusts the deletion cap to the backlog (covered in replication priorities). It also re-evaluates the readiness of every disk marked for removal. This is how a marked disk becomes removal ready (disk errors and removal).

The pass time is the lag you observe​

The speed of the chunk loop is set by two fixed limits: a minimum pass time and a cap on chunks examined per second. The limit that gives the longer pass applies. The two options, their defaults, and their bounds are described in replication priorities. The size of the backlog does not change them.

The file loop has no pace settings. It advances by a fixed fraction of the file table ten times a second, so a full pass takes about 55 minutes, whatever the number of files. A pass is suspended while the Master Server's information about the cluster is incomplete. While Chunkservers are registering, sending chunk lists, or being disconnected, the loop stays at its current position. It continues when the reports are complete. A pass can therefore take longer than the base time. The chunk loop delays its repair work under the same conditions, as replication priorities describes.

The lag of a policy is therefore up to one pass of the loop that applies the policy. In a chain that crosses both loops, the pass of the file loop takes most of the time. A file that becomes eligible for archiving is flagged within one pass of the file loop. The flagging itself adds the chunks of the file to the repair queues. After that, only the queue time of the conversion remains. Only a redefined class waits for the chunk loop. For each loop, the Master Server keeps the start and end times of the last completed pass and the progress of the current pass. The cluster reports these values.