Raw space and user space
The disks that you buy decide the raw space of a cluster. The part of that raw space that holds files is the user space of the cluster, also called effective capacity. User space needs a separate calculation. The terms of that calculation are not all of the same kind. Redundancy and the space that a Chunkserver holds back subtract from the raw figure. The size that the cluster counts for a file works the other way, because that size is not always the space that the file occupies on a disk.
The Chunkservers report the raw space
The raw space of a cluster is the sum of the space that its Chunkservers report for their disks. Nothing in MooseFS caps that sum. The width of the counter that holds the sum sets the ceiling. Max cluster capacity covers the figure. The rest of the calculation starts from that sum.
Redundancy is per class, not per cluster
The Storage Class of a file, not the cluster, decides how much raw space a stored byte occupies. A Storage Class that keeps copies multiplies the data by the number of copies. A Storage Class that uses Erasure Coding multiplies the data by the parts that it writes over the parts that hold data. Choosing between copies and Erasure Coding compares the schemes and what each one costs.
An instance that runs several classes therefore has no one factor to divide the raw space by. The user space of that instance is the sum, over the classes, of the data that you expect to keep in each class. The answer changes when data moves between classes. Files that are rewritten to a cheaper scheme after they stop changing free raw space without a byte leaving the cluster. Plan class by class, with a figure for the data that each class is meant to hold. For data that is already stored, the Master Server keeps the answer as realsize. This figure counts a subtree after redundancy. Quota compares that figure with the two figures that are easy to confuse with it.
A chunk costs a floor, whatever it holds
In the accounting of the cluster, every chunk that holds any data costs at least 72 KiB. A file shorter than that still costs the full amount. A Storage Class that keeps copies counts that minimum once per copy. A Storage Class that uses Erasure Coding counts the minimum once and scales it by the parts written over the parts that hold data. @8+2 therefore makes a 72 KiB minimum cost 90 KiB, not ten times 72 KiB.
On a workload of small files, this minimum, not redundancy, decides how much of the cluster the data occupies. The number of chunks sets the figure. The bytes inside the chunks matter far less. Count the chunks that the workload will create and multiply, instead of dividing raw space by a redundancy factor.
A Chunkserver keeps part of every disk back
A Chunkserver holds back a fixed amount of free space on each of its disks and never offers that space for chunks. The amount is HDD_LEAVE_SPACE_DEFAULT in mfschunkserver.cfg, 256MiB unless you set it otherwise. The reserve comes off a path that is listed in mfshdd.cfg with no space option. The reserve also comes off a path whose only space option is the shared indicator =. A path that has a limit, a reserve, or a percentage is measured by that option instead.
The same distinction decides where the reserve appears in the figures that the Chunkserver reports. On a path with no space option, the reserve stays inside the reported total and outside the space reported as free. The reserve therefore appears as used space from the moment the disk is empty. On a path marked =, the reported total is the space that the chunks on that path occupy plus the space still on offer. The reserve is outside both.
The size of this deduction depends on the number of disks, not on their capacity. A server with many small disks leaves you less of the space that you bought than a server with a few large disks of the same total size.
The counted size follows from length, not measurement
The Master Server calculates what a file occupies from the length of the file. Every chunk before the last counts as 64 MiB and 8 KiB. The last chunk is rounded up to whole 64 KiB blocks plus the same 8 KiB. Only chunks that were written count, so a chunk that was never written in a sparse file costs nothing. The length alone does not show how many such gaps a file has. mfsdirinfo prints this figure as size, next to length and realsize. A quota can be set against each of the three separately, as well as against the inode count. None of this involves asking a Chunkserver what it wrote.
A Chunkserver writes a chunk sparsely. When a write reaches or passes the current end of the chunk file, the Chunkserver drops the zeros at the start and at the end of the block. The Chunkserver stores only the range between those zeros. The space used on the disk is what was stored. This behavior is on unless HDD_SPARSIFY_ON_WRITE in mfschunkserver.cfg turns it off. A file with long runs of zeros therefore occupies less disk space than its length suggests. Allocation and space selection describes how the two figures affect placement. The accounting of the cluster and the disks therefore report different numbers for the same data. Both numbers are right. The accounting figure is calculated from length. The disk figure is the space that the disks hold. Size against the accounting figure. Treat the difference as room that you may get, not room that you have.
df reports raw space, not user space
The total that a mount reports through df is the raw space of the cluster. The Master Server adds up what the Chunkservers report and gives that sum to the client. Free space is the remainder of that sum. RESERVE_SPACE in mfsmaster.cfg is 0 until you change it. That reserve comes off the available figure alone. The free figure keeps the whole remainder. The mount reports both. The reserve therefore appears as space that you cannot claim, not as space that something has used. Neither figure accounts for redundancy. Under a Storage Class that keeps three copies, one byte written takes three bytes off both figures.
The space that the Chunkservers keep free on disks listed with no space option counts as used. A cluster that nobody has written to yet therefore reports that reserve as occupied. Quotas change the reading in the opposite direction. Run df on a directory that has a quota on length, size, or realsize, or on a directory under such a quota. The total, the free figure, and the available figure are all reduced to what the quota leaves. The mount then reports a smaller file system and not only a fuller one.
The same disks hold very different amounts
Take 20 Chunkservers and one Storage Class that keeps two copies of everything. Each Chunkserver has 20 disks of 1 TiB, listed with no space options.
| What is counted | Figure | Measure |
|---|---|---|
| Disks bought | 400 | count |
| Raw space | 400 TiB | space on the disks |
| Space the Chunkservers keep free | 100 GiB, before anything is written | space on the disks |
| File data at two copies, files large enough that the per-chunk floor never governs | about 200 TiB | size the cluster accounts for |
| File data at two copies, average file 20 KiB | about 3 billion files, holding about 56 TiB of data | length of the files |
The 200 TiB figure is in the size that the cluster accounts for. Quotas and mfsdirinfo use the same measure. The small-file row shows why the two measures differ. The cluster accounts 72 KiB per copy for the chunk that holds a 20 KiB file. That accounting measure therefore fills the disks at about 3 billion files. The data that those files hold, however, comes to about 56 TiB. On the disk, the chunk file has 8 KiB of its own and then only the range that was written (small files describes the layout). The disks hold more 20 KiB files than the count gives, so treat the count as the conservative answer.
Both figures for file data describe the same disks under the same class. The size of the files makes the difference between them. For this reason, the shape of the workload is decided before the hardware (start from the workload).