Snapshot copy-on-write semantics
The snapshot page covers the concept: what a snapshot is, and why a snapshot costs nothing until something is written. This page describes the mechanism underneath. The page covers what the Master Server does at the moment a snapshot is taken and what the first write into a shared chunk starts. The page also covers how a snapshot tree is accounted for and removed. The snapshot tools manpage covers the tools and their options.
A snapshot copies inodes and counts chunks
A snapshot is a walk of the source subtree that takes place entirely inside the Master Server's metadata. Each object visited gets a new inode in the destination. For a file, the new inode receives a copy of the source's ordered list of chunk IDs. Each of those chunks has its file counter raised by one. From that moment the chunk is shared. No Chunkserver takes part. The walk moves no data. The time that the walk takes depends on the number of objects and chunk references, never on the number of bytes that they refer to.
The copy comes last. The Master Server first walks the tree to check that the destination can legally receive that tree. The Master Server then walks the tree again and checks that the destination's quota can accept the charge. The snapshot is charged as a full copy of everything that the destination does not already hold. A failure in either pass fails the whole operation before anything is created, so a snapshot either completes whole or leaves no trace.
The whole subtree is one Master Server operation and one record in the changelog. Followers and Metaloggers replay this record as one operation. When one command names several sources, each source is a separate operation. While an operation runs, the Master Server answers no other request. Every ten thousand objects visited, the Master Server lets its network layer exchange no-op packets. These packets keep clients, Chunkservers, and Metaloggers from timing out. Real work resumes only when the operation ends.
A snapshot taken onto a destination that already holds an earlier snapshot is merged into the destination. The Master Server does not start over. A destination file whose chunk list still matches the source is not recreated. The list and the chunk counters stay unchanged. The Master Server refreshes the destination file's ownership, mode, and times, and sets the snapshot flag. A file that differs is recreated to share the source's current chunks. Directories are merged. A repeated snapshot therefore costs work in proportion to what changed since the previous snapshot.
The first write duplicates the shared chunk everywhere
When a client asks to write into a chunk, the Master Server reads the chunk's file counter. If exactly one file uses the chunk, the write goes into that chunk in place, raising the version when a raise is due. If more than one file uses the chunk, the writing file gets a new chunk ID. The version of that new chunk starts again at 1. Every Chunkserver that holds a valid copy of the shared chunk is told to duplicate that copy onto one of its own disks. The slot in the writing file's chunk list now points to the new chunk ID. The old chunk's file counter drops by one. The files that did not write keep the old chunk unchanged. Truncating a shared chunk works the same way. The duplicate is written out at the new length.
The duplication is local. Each Chunkserver reads its own copy and writes the new chunk file beside that copy, so no data crosses the network. The duplication costs disk I/O on every Chunkserver that holds a copy. The I/O is up to the whole chunk on each Chunkserver, however few bytes the client's write touches. Blocks that contain only zeros are skipped and left sparse, unless the Chunkserver's HDD_SPARSIFY_ON_WRITE option is turned off. The option is 1 by default. The client's write is answered only after every copy has finished duplicating, so the first write into a shared chunk is also the slow one. A shared chunk stored only as Erasure Coding parts is first rebuilt into whole copies. The duplicate is then made from those copies. The client waits through both steps.
The write is answered only once every copy has finished duplicating: the first write into a shared chunk is the slow one, a whole chunk of disk I/O per copy.
Reads trigger nothing. The second and every later write goes in place.
A shared chunk held only in EC is joined back into whole copies first, and the duplicate is made from those.
Every other operation runs without duplicating the chunk. Reading a shared chunk triggers nothing. Metadata operations, renames, and class changes duplicate nothing. The second and every later write to the same chunk finds a file counter of one and goes in place. So does a write to a chunk whose other users have since been deleted, because the counter has dropped back to one. A write into a hole has no chunk to duplicate and allocates a new chunk. This case is described in length, holes, and truncation.
A shared chunk has one class and is counted for every file
The files that share a chunk can have different Storage Classes. The chunk is then kept under the class with the highest priority. The choice is described in class priority and export groups.
On the Chunkservers, a shared chunk exists once per copy, so a new snapshot occupies no chunk space. The accounting does not match the disks. The Master Server's per-directory counters charge a chunk to every file that uses the chunk. mfsdirinfo therefore reports a snapshot at the size of a full copy, unless the precise mode is used. The precise mode counts every chunk once. Quotas charge the destination for a full copy at the moment the snapshot is taken. The counters show the space that the tree will occupy once writes have duplicated its chunks, not the space that the tree occupies today.
Removing a snapshot bypasses the trash
mfsrmsnapshot maps to a single Master Server operation per named object. The operation makes two passes. The first pass checks that every object in the tree has the snapshot extra attribute, and that none is protected by the undeletable, immutable, or append-only attribute. If any object fails the check, nothing is removed. The forced variant skips that pass and removes only the flagged objects. The other objects stay in place. The second pass unlinks every qualifying object with its trash retention forced to zero. The files therefore skip the trash entirely. The file counters of their chunks drop at once. A file that a client still holds open is kept, the same as after any unlink. Deletion covers what happens when a chunk loses its last file.
The attribute that the first pass looks for is easy to lose. The Master Server clears snapshot on a directory whenever an entry is added to that directory or removed from it. On a file, the Master Server clears the attribute whenever one of the file's chunks is written. No change to the data sets the attribute again: undoing the edit that cleared the attribute does not restore the attribute. The attribute can, however, be restored by hand with mfsseteattr -f snapshot. The tree then passes the first check again.
The removal pass stalls the Master Server in the same way as taking a snapshot: one operation, no other requests answered until the operation ends, keepalive packets in between. The length of that pass depends on the number of objects. Removing a very large snapshot in one call therefore stalls the instance for the whole removal. Removing the snapshot subtree by subtree limits each stall to the size of the named subtree.
Removing a snapshot with an ordinary recursive delete behaves differently on both counts. Each object is its own unlink operation, so the Master Server interleaves these operations with other work. Each file also keeps the trash retention that it was created with. By default, a snapshot inherits that retention from its source. A large snapshot deleted this way therefore does not stall the instance. The removal is slower in total. The removal moves every file with a nonzero retention into the trash instead of freeing its space.