Skip to main content

Quota

A quota is a limit on how much of the file system one directory subtree is allowed to use.

What it is​

A quota is always set on a directory. The quota covers everything under that directory, at any depth. The limit therefore belongs to a place in the namespace rather than to a person. You give a user a quota by giving the user a directory. Quotas nest. When several directories on the path from the root to an object have a quota, all of those quotas apply.

Four figures are counted. Each one can be limited on its own:

  • inodes: the number of objects inside the subtree.
  • length: the sum of the file lengths, which is the amount that an application sees as written.
  • size: the sum of the sizes of the chunks that those files use, with redundancy left out.
  • realsize: that same sum scaled by the redundancy of the Storage Class that the file has.

The last three count the same data at different points on its way to the disks, so they rarely agree. size is larger than length for an ordinary file, because a chunk is stored with an 8 KiB header. The data in a chunk is also counted in whole 64 KiB blocks. A file that holds a single byte has a length of 1 and a size of 72 KiB. size is smaller than length for a sparse file, because a chunk index that was never written has no chunk assigned to it and takes no space. A hole inside a chunk that the file did reach is counted all the same. size derives the size of each chunk from the file's length rather than from the bytes stored in the chunk, as length, holes, and truncation shows. realsize is the only one of the three that counts the redundant copies and the parity parts. This figure is therefore the only one that measures the space that the subtree uses on the Chunkserver disks, rather than what the application wrote.

realsize is an estimate rather than a measurement. That is by design. A chunk stored in copies is counted exactly, because every copy is a whole chunk. A chunk split into parts by Erasure Coding is counted closely enough for a large file and least closely for a small file. Small files are also the files that Erasure Coding stores at a higher cost. Both points are covered in Erasure Coding layout.

Each figure takes two limits. The soft one may be exceeded for a while. The hard one may not be exceeded at all. Either limit can be set on its own. The soft limits of a directory share one grace period. That period is the time that you have to get back below the soft limits.

Why it matters​

Chunkserver space is a single pool shared by the whole instance. Nothing else in MooseFS stops one directory from filling that pool. A quota keeps a project, a tenant, or a runaway job inside a share of that pool.

Four figures exist because "how much space" has more than one correct answer. A limit on length charges a user for the bytes written. A limit on realsize also charges for the redundancy that those bytes are stored with. The same file therefore counts more under a class that keeps three copies than under one that keeps two.

A quota is also visible from inside. A client can ask for free space anywhere under a directory with a limit on length, size, or realsize. The Master Server then answers with the space left under the quota instead of the space left in the cluster. An inode limit is reported the same way. Tools that check free space before writing therefore see the limit that actually applies to them.

How MooseFS implements it​

MooseFS enforces a quota before the operation, never after it. MooseFS refuses an operation that would take a directory past one of its limits. The test covers the target directory and every directory above it. The application gets the system's disk-quota-exceeded error. Quota enforcement covers what each kind of operation is charged, and why the test costs the Master Server almost nothing.

The two kinds of limit differ in when they refuse. A hard limit refuses the operation at once, in that test. A soft limit lets the operation through and starts a grace period. After the period runs out, the soft limits refuse exactly as hard limits do, until the subtree is back below the soft limits. Quota enforcement describes how the Master Server tracks those grace periods. That page also gives the grace period that a quota uses when the quota sets none of its own. Setting a limit under the current usage deletes nothing. Such a limit stops the subtree from growing.

The figures are counted per tree, not per object. A file hard-linked into two directories with quotas counts in full against both. A snapshot shares the original chunks, but it counts against the quota of its destination directory as though its data were new. A deleted file is removed from the totals of its directory at once, even though the file can still be recovered.

Quotas are part of the metadata and remain after a restart of the Master Server. Reading one takes no special privilege. Setting, copying, or removing one is the work of the mfsquota tools. These tools need a mount whose export grants administrative rights, is not read-only, and leaves root unmapped (maproot=0).