Space management

This page covers reading how full a cluster is and controlling what fills it. That includes the figures that df, mfsdirinfo, the GUI, and mfscli report, and quotas on directory subtrees. It also covers the cost of the trash, the space that deletions have not freed yet, and what stops when the cluster runs out of space.

df reads the cluster, mfsdirinfo one subtree

df on a mounted MooseFS reports the raw space of the cluster, whatever directory you run it on. The total is the sum of what the connected Chunkservers report for their disks. Every copy and every parity part of a file counts against that total. Raw space and user space lists the deductions inside those figures: the reserve that the Chunkservers keep on each disk and the RESERVE_SPACE margin from mfsmaster.cfg. A quota is the one exception. Under a directory that has a quota on length, size, or realsize, df reports the space that the quota leaves instead. A subtree past its soft limit reports zero free space for its whole grace period (quota enforcement). df -i reports objects against the identifier space of the instance, which has 2 147 483 646 identifiers (limits). Where an inode quota applies, it reports objects against that quota instead.

mfsdirinfo is the MooseFS counterpart of du -s. It answers from counters that the Master Server keeps per directory, so it returns at once, whatever the size of the tree:

mfsdirinfo -h /mnt/mfs/projects

It prints the subtree's object counts (inodes, directories, files, chunks) and three sizes: length, size, and realsize. Quota explains what each of the three measures, why they rarely agree, and why realsize is an estimate rather than a measurement.

The default figures count every occurrence separately. A file hard-linked into the tree twice counts twice. Chunks shared through snapshots count under every file that refers to them. mfsdirinfo -p walks the tree instead and counts every node and chunk once. It counts the copies and parts that exist right now, not what the Storage Class requires. This is the precise reading. On a large tree, it is slow. Syntax: mfsdiagtools(1).

The GUI and mfscli break the space down

The df total does not show where the space is. The Chunkservers tab of the GUI shows the used space, the total, and the percentage used for each server. The space of disks marked for removal is summed separately from the regular space. The Disks tab shows the same figures per disk. mfscli prints the same two tables with -SCS and -SHD.

The Cluster Summary table on the Info tab (mfscli -SIG) adds the cluster totals: total space, available space, and the percentage used. The trash and the sustained area appear next to them, each with a space figure and a file count. The space figures for the trash and the sustained area count file lengths, before redundancy.

The Cluster summary block shows the totals of the whole instance in one row. The row shows the space, what is in the trash, and what the sustained files still hold.

Quotas keep a subtree inside its share

Setting, changing, or deleting a quota needs a mount whose export entry in mfsexports.cfg grants admin, is not read-only, and leaves root unmapped (maproot=0). Reading a quota needs no privilege. Quota describes what a quota limits: four figures with a soft and a hard limit each. Quota enforcement covers what happens at the limits.

mfssetquota sets the limits. -i, -l, -s, and -r set the soft limits on inodes, length, size, and realsize. The uppercase forms set the hard limits. -p sets the grace period for the directory's soft limits. -P uses the default grace period, QUOTA_DEFAULT_GRACE_PERIOD in mfsmaster.cfg, which is one week unless changed:

mfssetquota -i 1000000 -I 1200000 -p 3d /mnt/mfs/projects/alpha
mfssetquota -S 10Ti /mnt/mfs/projects/alpha

A limit is a plain number of bytes or a number with a suffix. The suffixes k, M, G, T, P, and E mean the decimal multiples. The suffixes Ki, Mi, Gi, Ti, Pi, and Ei mean the binary multiples. Nothing may follow the suffix. A trailing B is rejected with bad size limit, so the binary tebibyte is Ti, not TiB.

mfsgetquota prints the directory's grace period and, for each figure, the current value against the soft and hard limits with percentages. The Quotas tab of the GUI lists every quota of the instance with the same usage figures. mfsdelquota removes limits one kind at a time, or all of them with -a -A. mfscopyquota copies every quota of one directory onto another directory. It replaces limits of the same kind and adds the kinds that the destination did not have. It deletes nothing. A limit that the destination already had, and that the source does not set, still applies after the copy. Syntax of all four: mfsquota(1).

Two things make the quota reading differ from the disk usage. A file counts in full against every quota tree that it is linked into, so hard links and snapshots count once per tree. A file in the trash counts against no quota at all. A subtree can therefore use far more disk space than its quota reports. The space stays in the trash figures until the retention expires.

Retention empties the trash, exports bound it

The trash holds every deleted file for that file's retention, at full cost in disk and in Master Server RAM. Trash retention describes the mechanism and its memory cost. The everyday view of the trash, recovery included, is described in using the file system. The administrative controls:

• Lower the retention where turnover is high. The TIME- form of mfssettrashretention lowers only the values above the given time, so it caps a tree without raising any value:

  mfssettrashretention -r 24h- /mnt/mfs/scratch

  New objects inherit the value of the directory they are created in, so setting the value on the directories affects both existing and future objects. Verify with mfsgettrashretention. With -r on the same directory, it lists the values in the subtree, with one count per value. After the cap, no value is above 24 (1d). Syntax: mfstrashretention(1).

• Bound what users may set. mintrashretention and maxtrashretention in an export entry set the range within which clients on that export may set retention. A request outside the range is refused rather than trimmed. An entry that sets neither leaves the range unbounded.

• Remove entries by hand when the space is needed now. No command empties the whole trash at once. The pace depends on the retention itself and on the sweep that detects expiry. Both are described in trash retention. For a single file, mfsmount with -m mounts the MooseFS Meta file system, which shows the trash. Unlinking an entry there destroys the file at once:

  mfsmount -m -H mfsmaster.example.com /mnt/mfsmeta

  Entries under trash are named by inode number and the recorded path. The mount needs its own export entry. That entry limits who can reach the area. Trash and sustained files covers the trash and the sustained area. If you remove an entry while a client still holds the file open, the entry moves to the sustained area and frees nothing.

Destroyed is not yet freed

rm returns at once. The disk space is freed later. Between the two are the trash, the sustained area, the Master Server's chunk loop, and the deletion orders that the Chunkservers execute. Each of these has its own timing. Deletion: from unlink to space returned describes the chain and how to read its progress. Where to look while waiting:

• Files in the trash and in the sustained area still exist. Their space and file counts appear in the Cluster Summary table on the Info tab (mfscli -SIG). A sustained file frees nothing until the last session that holds it open releases the file.
• Chunks of files that no longer exist anywhere wait for the chunk loop. The chunk matrix summary counts them as pending deletion (the chunk matrix table on the Info tab, mfscli -SIC). The self-check loops report shows how many deletions each pass of the loop placed and how far the current pass has progressed. Read it with mfscli -SIL, or in the Chunks Housekeeping Loop on the Info tab of the GUI.
• A chunk that belongs to no file but is still under a lock is not deleted until the lock expires. The loops report counts these as locked unused. Such chunks come from operations interrupted by a failure. When a Chunkserver reports a chunk that the Master Server does not recognize, the chunk is registered and kept for seven days before the loop deletes it.

Writes stop before the disks are full

A Chunkserver is removed from the pool for new chunks while it still has free space. The Master Server stops choosing that Chunkserver when the Chunkserver's reported free space falls below a threshold. The threshold is one chunk plus a margin for every chunk held open for writing on that Chunkserver. A chosen Chunkserver can still refuse a chunk when none of its disks can accept the chunk. When no server can accept a chunk, the client is told that there is no space. The mount retries for about two minutes by default and then fails the write with ENOSPC ("No space left on device"). The selection, the thresholds, and the retry budget are described in allocation and space selection.

Reads and deletions keep working. RESERVE_SPACE in mfsmaster.cfg is zero until you set it. It makes writes stop earlier for everyone but root. The reserve is subtracted from the available space. When the available space reaches zero, the Master Server refuses new chunks for ordinary writes. Writes from uid 0 may still continue into the reserve.

Deleting data frees space only through this chain. A file whose retention is above zero moves to the trash and frees nothing yet. When space runs short, cap the retention or remove trash entries through the Meta mount. Then follow the loop's progress instead of waiting for a fixed figure. For a durable fix, add raw space. Adding disks and Chunkservers to a running cluster is described in disks and Chunkservers.