Skip to main content

Snapshot

A snapshot is a copy of a file or a directory tree as the source was at one moment. The snapshot takes no extra disk space until one of the two sides is written to.

What it is​

The snapshot of a file is an ordinary object in the namespace, with its own inode, name, and permissions. You read, write, rename, and delete a snapshot with the same operations as any other object. A snapshot is not read-only and is not kept apart in a system area. Neither side is marked as the original. After the operation, MooseFS treats the two sides as independent objects that happen to hold the same content.

The operation copies metadata, not data. The new file has the same ordered list of chunk identifiers as the source, so no chunk is read, written, or transferred. The snapshot of a directory repeats this for every file in the subtree, under a copy of the directory tree. The work happens in the Master Server alone. The Chunkservers take no part in it. The time that the operation takes therefore depends on the number of objects rather than on the amount of data that those objects reference.

You take a snapshot with mfsmakesnapshot. The source can be a file, a directory, a symbolic link, a FIFO, a socket, or a device node. A directory is taken as a whole subtree, in one Master Server operation. That operation is atomic. The Master Server completes the whole operation before answering anything else. A snapshot of a subtree with many objects therefore blocks the whole instance for as long as the operation takes. The destination can be a name that does not exist yet, if its parent directory exists. If the destination is an existing directory, the snapshot is created inside that directory. An existing file as the destination is an error, unless you give -o to overwrite that file. With more than one source, the destination must be an existing directory. In every case, the destination must not be inside the source.

The Master Server copies the source as the source is recorded in the Master Server's own metadata. The Master Server does not check whether a file is open. The operation flushes nothing first. The length that the snapshot records is therefore the last length that the Master Server has recorded.

A write that is already under way is not stopped either. The Master Server counts a whole write session as one operation on the chunk. That operation began before the snapshot, so the copies of the chunk are not separated. The bytes that the client sends after the snapshot are still written into the chunk that both sides now share. A snapshot of a file that somebody is writing to is therefore not a consistent point in time for the content of that file. To get a consistent point in time, stop the writer first yourself. MooseFS does not stop the writer for you.

Why it matters​

A snapshot gives you a state of your data that stops changing while work on the original goes on. You can back the snapshot up while the writers keep writing, because the data that the backup reads no longer changes. You can give a team its own copy of a large data set without waiting for the data to be copied. The team gets that copy without access to the original. You can take a snapshot before a risky change. If the change fails, then the earlier state is still in the file system, under its own name.

The cost of taking a snapshot depends on the number of objects, not on the amount of data. Until something is written, the snapshot uses no chunk space at all. The snapshot occupies one inode and one chunk list per object, in the Master Server's memory.

Storage is used later, and in whole chunks. The first write into a chunk that two files share duplicates that chunk. Every Chunkserver that holds a copy of the chunk makes the duplicate, however few bytes the write touched. When the data is later rewritten in many scattered places, keeping the snapshot therefore costs almost as much as a full copy. The cost grows with the number of chunks touched, not with the number of bytes written. The saving remains when the data that the snapshot references is mostly read.

How MooseFS implements it​

Every chunk counts the files that use it. A snapshot adds the new object to that count for each chunk of the source. From that moment, those chunks are shared.

The Master Server checks that count when a client asks to write to a chunk. A chunk that one file uses is written in place. A chunk that more than one file uses is duplicated first. Only the writing file moves onto the duplicate, so nothing changes for the files that did not write. Truncating a shared chunk works the same way. The glossary calls this copy-on-write: per chunk, and only on the first modification. Snapshots and copy-on-write covers how the duplication is ordered, and what a second snapshot onto the same destination costs.

Which side writes makes no difference. If you write to the original, then the original moves to a new chunk. The snapshot stays on the old chunk. If you write to the snapshot, then the snapshot moves instead. The side that did not write keeps what it had. A snapshot that nobody writes to therefore keeps showing the moment when it was taken. Writing into a snapshot is an ordinary thing to do. The snapshot then stops showing that moment.

By default, the new object takes its attributes from the source. The new object gets the same Storage Class, the same trash retention, and the same permission bits. The owner and group carry over only when you run the command as root or when you own the source object. Otherwise, the new object gets your user and group. The setuid and setgid bits of the new object are cleared. The new object also has the same extended and extra attributes, the same POSIX ACLs, and the same three timestamps. The tools also offer the opposite. With one option, the new object gets the attributes that an ordinary copy into the destination directory would get. These are you as the owner, the current time, and the class and retention of the destination directory. Either way, the two objects are independent afterwards, so their classes can end up different. A chunk that the two objects still share is kept under one class, chosen as Storage Class describes.

These rules describe objects that the operation creates. With -o, an object that already exists at the destination is overwritten the way cp overwrites: the source's extended attributes, extra attributes, and POSIX ACLs are not copied onto it. An existing directory stays in place and keeps its own Storage Class, trash retention, extended and extra attributes, and ACLs. Only its permission bits, owner and group, and timestamps follow the source, under the same rule as above. An existing file is replaced by a new object. That object takes the source's chunk list, Storage Class, trash retention, permission bits, owner and group, and timestamps. Its extended attributes, extra attributes, and ACLs are the ones that any new file in that directory gets.

Every object that the operation creates has the snapshot extra attribute, which means "made by a snapshot and unchanged since". The Master Server clears the attribute on the first change to the object (a write, a truncation, or, for a directory, an added or removed entry). An object created inside a snapshot directory does not inherit the attribute. mfsrmsnapshot removes a whole snapshot in one operation. By default, the command checks the whole tree before removing anything. If one object lacks the attribute, then nothing at all is removed. The -f option skips that check and removes the objects that have the attribute. The other objects stay in place.

Hard links inside the source are not preserved by default. In the destination, two names of one file become two independent objects. The -p option of mfsmakesnapshot keeps the names as hard links to one object instead.

Two limits apply when the snapshot is taken. A file that you may not read is left out. Nothing reports the omission. A directory that you may not read produces an empty directory. The destination is also charged against its quota as though the snapshot were a full copy. Shared chunks count toward the size. The operation fails if that size does not fit.

The same sharing also happens without a snapshot. mfsappendchunks appends the chunks of one file to a second file at chunk boundaries. Those chunks are then shared in exactly the same way.

  • Take a snapshot: the commands that make one, remove one, and verify the result.
  • Chunk: what a snapshot shares instead of copying.
  • Copy: what a shared chunk is duplicated into when it is first written.
  • Storage Class: what the new object inherits, and what governs a chunk that two files share.
  • Metadata and namespace: the structure a snapshot duplicates.
  • Master Server: the component that performs the operation and counts the files that use each chunk.