Backup target

This page describes one deployment: a landing zone for nightly backups on a single site. The zone holds about 1 PB of backup data whose oldest sets are 90 days old. Backup software writes the data through a mount. The data is read again only when something has to be restored. The figures are notional and round. They show which decision follows from which, so that you can repeat the same reasoning with your own figures.

Large files are written once and read only on restore

Backup software writes streams: a few large files per protected system per run. Each file is appended from start to end and then left alone. The files are never rewritten and are not read until a restore. This pattern decides the design. The data is cold within hours of being written. Erasure Coding is for data that is cold. The object count is low for the capacity: a petabyte of backup streams comes to roughly fifteen million chunks in a few thousand files. The load on the cluster therefore comes from the length of the files, not from their number. Check your own counts with Master Server RAM from the object count. Under Erasure Coding, every stored part needs a record of its own, so ten parts per chunk cost ten records.

The backup window matters as much as the volume. This example assumes runs between 22:00 and 06:00 that write 20 TB a night. The retention policy expires most sets within weeks and keeps only a few to the full age. That leaves about 1 PB stored. The landing tier is sized for the nightly figure. The capacity tier is sized for the stored petabyte.

The cluster has a landing tier and a capacity tier

Tier	Chunkservers	Disks per server	Raw space	What it holds
Landing, labeled L	4	4 × 8 TB	128 TB	the newest sets, in two copies
Capacity, labeled C	14	10 × 12 TB	1.68 PB	everything older, as @8+2 parts

The Chunkserver count of the capacity tier comes from the Erasure Coding (EC) thresholds rather than from the terabytes. @8+2 needs 12 Chunkservers to convert copies into parts and 10 to keep parts once they exist (Chunkservers needed for an Erasure Coding scheme). The archive definition below names a label expression, so under the STANDARD labels mode only the servers that the expression selects count toward the conversion threshold. Fourteen capacity servers are therefore two more than the count that keeps conversion running. With two servers lost, the archive still converts. With four lost, the existing parts still stay parts.

The space follows from that count and has to be checked against it. A petabyte held at @8+2 occupies 1.25 PB of raw space, because eight of the ten parts hold data. Choosing between copies and Erasure Coding compares the cost of the two formats. Raw space and user space covers what else is deducted from the disks. The 0.43 PB left over is not slack. One capacity server holds 120 TB, so that margin leaves room to re-create the parts of three lost servers without the cluster filling up.

The landing tier holds 40 TB a night at two copies, so 128 TB holds about three nights. Sizing the landing tier for one night would mean a run that exceeds its window, or a conversion that falls behind and then stalls on a full tier. Growth goes to the capacity tier, in batches rather than one machine at a time, for the reason given in mixing hardware.

One Storage Class decides what happens to the data from write to expiry

Label the Chunkservers of each tier in mfschunkserver.cfg, then define one Storage Class with mfsscadmin and give it to the backup tree with mfssetsclass:

mfsscadmin create -K '2L/[RACK]' -A '@8+2,C' -d 12h -o m -s 8Mi nightly
mfssetsclass -r nightly /mnt/mfs/backup

That one definition decides everything that happens to the data from then on. Each night's directory is created inside the tree and inherits the class, so nothing has to be assigned again. The order of precedence between inheritance, patterns, and manual assignment is described in how policies are applied.

-K '2L/[RACK]' puts two copies on the landing servers. A CREATE definition would gain nothing, so the class has none. An Erasure Coding expression is accepted only in ARCHIVE and TRASH, so the night's data is written as copies whatever the class says. A class without CREATE falls back to KEEP.

-A '@8+2,C' PRO (PRO Edition only) puts the parts on the capacity servers. Two parity parts need PRO. The Community Edition accepts one parity part. That changes both Chunkserver thresholds and the raw space above. Conversion starts as a split inside the Chunkserver that already holds a copy. Afterwards, the parts are moved to servers that match the archive expression. The landing servers therefore spend CPU on the split as well as disk on the copies (Erasure Coding: part layout).

-d 12h -o m measures the file's mtime and waits half a day. The choice of timestamp is not cosmetic. Renaming a finished file, changing its attributes, and running the maintenance commands that change metadata all change ctime. The archive rule measures ctime unless it is told otherwise. A ctime rule therefore restarts the delay for reasons that have nothing to do with the data. Mtime changes on a data write and on nothing else (timestamps). The 12-hour delay puts the conversion into the hours when the disks are otherwise idle. A file finished at 05:00 becomes eligible after 17:00. A file finished at 22:30 becomes eligible after 10:30 the next morning. Both times are outside the write window. The delay is a minimum, not a schedule, because the flag is set when the file loop reaches the file.

Fast mode ignores the delay and flags every chunk as soon as the file loop reaches the file. Fast mode loses that timing of the work. Fast mode also converts the tail chunk of a stream while the writer is still appending. The next append then turns the chunk back into copies. Storage Class evaluation covers what each archive mode does.

-s 8Mi keeps small files in copies. Under Erasure Coding, every part is a chunk file with 8 KiB of its own metadata. A small enough chunk therefore takes more space as parts than as copies (small files). Backup catalogs and index files are small and are read far more often than the streams are. The cutoff keeps those files out of a format meant for data that nobody touches.

There is no TRASH definition. The definition is left out on purpose: a chunk with the trash flag set and no TRASH definition is evaluated against ARCHIVE. An expiring set therefore keeps the format it already had and needs no conversion back to copies as it expires.

Below the EC thresholds, copies replace parts

The class above assumes that the cluster has enough Chunkservers for @8+2. A smaller backup target may not have enough. The least demanding scheme does not lower the requirement far: @4+1 still needs six Chunkservers to convert copies into parts and five to keep the parts it has. A cluster that never reaches the higher count converts nothing. A cluster that drops below the lower count turns the parts it holds back into copies.

The archive definition is still useful with copies on both sides of it. Hold a set in three copies while the set is current, and in two once the set has aged.

mfsscadmin create -K '3*' -A '2*' -o m -d 7d smallsite

This saves a third of the raw space that the aged sets occupied. In exchange, the part of the stored data that nobody reads tolerates one failure fewer: three copies survive two lost Chunkservers, where two copies survive one. Damage to a set that nobody opens is found by the background test rather than by a reader. The week of delay keeps the newest set at the higher count. A restore is most likely to need that set. Weigh the saving against what parts would have saved. At @8+2, the same data occupies a little over four tenths of the three-copy figure, where two copies leave two thirds of it. Defining and assigning classes is the same work either way. Data placement tasks covers that work.

Racks are the failure domain EC cannot express

The /[RACK] extension on the KEEP expression tells the Master Server to keep the two copies in different racks. The Master Server reads the racks from mfstopology.cfg. For Erasure Coding, that extension is ignored. An EC expression takes at most two label expressions, one for the data parts and one for the parity parts, so no class can say "spread ten parts over seven racks".

The cluster does keep one rule: the parts of one chunk are stored on separate Chunkservers. Rack tolerance therefore becomes a layout decision instead of a policy decision. With no more than two capacity Chunkservers in a rack, no rack holds more than two parts of any chunk. @8+2 survives losing two parts. Here, that means seven racks of two capacity servers. The four landing servers are placed in four of those racks, so that the two copies of a newly written chunk cannot share a rack.

Retention stays with the backup software

MooseFS does not expire files by age. MooseFS is a POSIX file system. A file stays until something removes it. The backup software therefore keeps the policy of which sets expire and when. The file system decides what an expiry costs and what a mistake costs.

A deleted set does not free its disk space at once. The set waits in the trash for its retention, at full cost in disk space and in Master Server memory. While the set waits, it is not charged to any quota, so a subtree can hold far more disk space than its quota reports. On a target sized to its stored data, that space decides whether the data fits. Setting the retention of the backup tree to 24 hours leaves a wrongly deleted set recoverable for a day and returns its space inside the same day. The range that clients may set is bounded on the export entry, not left to the clients. Both controls are in space management.

Give each source system a subtree with a quota on it. A client that stops expiring anything then runs out of its own share instead of the whole target's. Somebody can see that shortage in advance.

Snapshots and protective flags limit the client

A snapshot of a finished set costs no chunk space and no data movement. On a backup target, the snapshot keeps the set when a client later deletes or overwrites the set: the snapshot still references the chunks.

A snapshot is not a write-once area, though. It is an ordinary object in the namespace, so whoever can delete the set can delete the snapshot as well. The export and the protective flags prevent that. The backup client's entry in mfsexports.cfg exposes only its own subtree. The snapshots are taken outside that subtree. The disable= option of the same entry removes the operations that the client never needs, among them snapshot, setsclass, and settrash. The Master Server enforces the protective extra attributes undeletable, appendonly, and immutable, so no mount and no mount option can bypass them (access control). The attributes are not inherited, so a finished set gets them in a separate pass. Expiring that set then needs an administrator to clear the flag first.

The background test exists for data like this

Nobody reads a backup set until they need it. Corruption that a disk returns without reporting an error is therefore found by the background chunk test or not at all. At the default HDD_TEST_SPEED of 1.0 MiB/s per disk, a 12 TB drive takes about four and a half months to be read through once. That is longer than a set is kept here. A set could expire without the test ever having read it. Raise the rate on the capacity servers. At 4.0 MiB/s, the same drive is read through in about five weeks, well inside the 90 days. The test reads in the idle I/O class on Linux and skips chunks touched in the last day. The test therefore competes with restores rather than with the night's writes. The pace, the reading order, and what a mismatch triggers are in scrubbing and CRC verification.

A restore reads parts where they lie

A read of a chunk held as @8+2 reaches up to eight Chunkservers, one per 256 KiB region of the range that the read covers. A chunk held in copies is read from a single Chunkserver whatever the length. A large restore out of the capacity tier therefore reads from 140 disks on 14 machines. The newest sets are still in copies. The landing tier has 16 disks on 4 machines. The most likely restore therefore reads from the fewest spindles. If the target exists to restore last night quickly, size the landing tier for the restore rate and not only for the write rate.

Reading undoes none of the archiving. The class measures mtime. A read changes no timestamp that the class measures. A read converts no chunk, so a petabyte read back stays in parts. When a restore has a deadline, the mount usually limits it rather than the cluster: a single mount has a ceiling of its own (the per-mount throughput ceiling). To exceed that ceiling, split the restore over several mounts or several hosts (multiple mounts and parallel access). Recovering one file that somebody deleted by mistake is not a restore at all. The file is recovered from the trash, as using the file system shows.

Where this design stops

An offsite copy of this target is a second instance rather than a wider single instance. The patch tools PRO (PRO Edition only) transfer the changes in a directory between two snapshots to another MooseFS instance. On that instance, the directory that the tools are applied to is on a MooseFS mount. Master Server operations covers the patch tools. A cluster whose own Chunkservers are in more than one place is a different design, which Two data rooms describes.