Virtualization
Disk images of running guests are large files that are rewritten in scattered places all day. A virtualization platform keeps other files beside them: the template library, the ISO files, and the backups. These files are written once and afterwards only read. The images and those files need opposite things from the storage under them. This page describes one deployment. It lists which part of that workload goes into which Storage Class, how many Chunkservers the choice needs, and what each decision costs. Telling a hypervisor that it may use a MooseFS directory as storage is a separate step. Proxmox and XCP-ng cover that step.
The example holds 40 guests and their backups
The figures are an example. The reasoning that justifies each choice also applies to a cluster of a different size.
| Data | Size in the example | How it is written |
|---|---|---|
| Disk images of running guests | 40 files of 20 GB to 100 GB, 4 TB together | scattered writes across each file, all day |
| Templates and ISO files | 500 GB | written once, then read |
| Guest backups | 20 TB | written once, kept, read during a recovery |
| Log files the guests write to the cluster | 200 GB | appended at the end, never rewritten |
This example has six hypervisor hosts and 14 Chunkservers. Each host mounts the instance once. The Master Server does not limit the size of this deployment. In files this large, 25 TB comes to a few hundred thousand chunks and fewer than ten thousand files. The Master Server's memory depends on that count, not on the volume (Master Server RAM from the object count). Start from the workload lists the questions that decide the rest of the hardware.
The archive scheme sets the Chunkserver count
Both classes below keep recently written data in three copies and archive it to Erasure Coding @8+2 PRO (PRO Edition only). The two formats have the same Redundancy Level: two lost disks or machines. They differ in cost: three copies occupy three times the data, @8+2 1.25 times. For the 20 TB of backups alone, that is 60 TB of raw space against 25 TB. Choosing between copies and Erasure Coding compares the schemes. Raw space and user space calculates a capacity figure from a mix of classes.
The scheme sets a minimum machine count. Converting chunks into @8+2 parts needs 12 Chunkservers. Keeping the parts needs 10. Below the lower figure, the chunks go back to copies and cost what copies cost. The counts for every scheme are in Chunkservers needed for an Erasure Coding scheme. A cluster with exactly 12 Chunkservers stops converting as soon as one machine is down. For this reason the example uses 14. With 14, the archive still forms when two machines are out of service. Chunkserver hardware covers what each machine is built of.
An image keeps copies where it is written
A disk image is one file whose regions behave differently. The guest's system areas and logs are rewritten every few minutes, but most of the image is read and never touched again. Per-chunk archiving is the rule written for that kind of file. The rule checks when each chunk was last written instead of reading the file's timestamps. A chunk that receives no write for the whole delay is therefore archived while its neighbor is still being written. A write to an archived chunk clears the flag and returns the chunk to copies. Storage Class evaluation describes the rule and every flag it accepts.
The alternatives fail on the same file. A rule that reads the file's ctime or mtime gives one result for the whole image. A single region rewritten every few minutes therefore keeps all 100 GB in copies. Fast mode archives every chunk as soon as the file loop reaches the file. The busy regions then go through a conversion, a write, and another conversion, repeatedly. Erasure Coding: part layout covers the cost of a write to a chunk held in parts.
With mfsscadmin and mfssetsclass, whose options are described in data placement tasks:
mfsscadmin create -K '3*' -A '@8+2' -o p -d 4h vmdisks
mfssetsclass -r vmdisks /mnt/mfs/images
The delay is counted in whole hours. It is a minimum, not a schedule. A short delay archives idle regions sooner, so more of them go through a conversion that the next write undoes. A four-hour delay keeps in copies the regions that a guest touches during a working day. The log files that the guests write to the cluster also belong in this class. A file that is appended at the end holds archived chunks at its start and chunks in copies at its tail.
Templates, ISO files, and backups archive early
These files are written once and then read, so waiting gains nothing. Fast mode flags every chunk the first time the file loop reaches the file. The delay does not affect these files.
mfsscadmin create -K '3*' -A '@8+2' -o f -s 4Mi vmarchive
mfssetsclass -r vmarchive /mnt/mfs/templates /mnt/mfs/backups
A clone or an exported snapshot that the platform no longer writes through belongs in this class as well. Anything that the platform still writes through belongs in the image class instead.
-s keeps small files out of the archive. Under Erasure Coding, every part is a chunk file that has its own header. Below a few megabytes, the headers cost more than the parity saves, so the file is cheaper in copies. The figures are in Small files.
One mount carries every guest on its host
Every guest on a hypervisor host reaches its image through the host's single mount. All of that traffic goes through one FUSE channel with a throughput ceiling of its own. The ceiling is a limit of the client machine, so no change on the Chunkservers raises it.
A second mount on the host opens a channel of its own. The images are divided between the mount points (multiple mounts and parallel access). You can also avoid the ceiling by mapping the image as a block device. mfsbdev is built on the client library rather than on FUSE, so its traffic does not go through that channel at all (Clients). The guest then gets a device instead of a file. The mapping commands are in Block device operations.
Whichever access method the images use, each write goes to as many Chunkservers as the class requires: three for recently written data. The bandwidth and latency budget calculates a figure for the network from that.
Separate disk tiers need Chunkservers of their own
One class, not a script, keeps running images on SSDs and their archive on mechanical disks. Give the fast Chunkservers the label F and the slow Chunkservers S in mfschunkserver.cfg. Then name the labels in the class instead of writing *: -K '3F' -A '@8+2,S'. Setting labels and moving data between tiers is covered step by step in data placement tasks.
Separate tiers cost machines. A label expression applies the Erasure Coding counts only to the servers that it selects. An archive limited to S therefore needs 12 S-labeled Chunkservers of its own, in addition to the machines that hold the F tier (Chunkservers needed for an Erasure Coding scheme). Running two Chunkserver processes on one machine to serve both tiers does not avoid that cost. The Master Server registers two servers, but there is only one machine. The rule that keeps copies on separate machines applies to full copies only, never to Erasure Coding parts. Several processes on one machine covers what that does to redundancy and what a second process needs to run at all.
A snapshot of a running guest is not one point in time
A snapshot is cheap at the moment it is taken: mfsmakesnapshot copies chunk lists in the Master Server without reading or writing any chunk. The snapshot does not quiesce anything. The Master Server does not flush the file or ask whether a client has it open. A write already under way still goes into the chunk that both files now share. A snapshot of a running guest's image therefore does not capture the content of that image at one point in time. Stop the guest, or have the hypervisor freeze the guest's file systems, before taking a snapshot. The commands are in take a snapshot.
The snapshot costs space later, in whole chunks. The first write into a chunk that two files share duplicates that chunk on every Chunkserver that holds it, however few bytes the write touched. An image rewritten in scattered places after a snapshot therefore approaches the cost of a second image. Snapshot copy-on-write describes the mechanism.
A deleted image holds space until retention ends
Removing a 100 GB image frees no space at once. The file moves to the trash. It keeps its chunks and its metadata for as long as its trash retention sets. On a directory where guests are built and destroyed all day, the space held in the trash is the gap between the space that the platform counts as freed and the space that the cluster has. Cap the retention on the image directory, or empty single entries by hand, as space management describes.
Guest swap stays off the cluster
A guest's swap area holds nothing that must be kept once the guest stops. If the swap area is on the cluster, every page written to it goes over the network to three Chunkservers, like any other write. Size the guests so that they do not swap. Where a guest needs swap anyway, give that guest a local disk or partition of the hypervisor host instead of a file on the cluster.