Quota enforcement
The quota concept page covers what a quota is, which four figures it limits, and how those figures differ. This page describes the enforcement. The page covers at which operations the Master Server tests the limits and what the application sees when a test fails. The page also covers what changes when the grace period of a soft limit runs out.
A quota is tested before the operation
The Master Server updates the four figures of every directory as part of each metadata change. A test therefore compares numbers instead of walking a tree. Every operation that would grow a subtree is tested before it is applied. The growth that the operation would cause is added to the current figures of the target directory and of every directory above it. If one limit anywhere on that path is exceeded, the whole operation is refused. The refusal reaches the client as the quota error. The client delivers the refusal to the application as EDQUOT ("Disk quota exceeded"). A platform that does not define EDQUOT gets ENOSPC instead.
| Operation | What the test charges |
|---|---|
| Creating a file, a directory, or any other object | One inode. |
| Creating a symbolic link | One inode, plus the target path counted as length. |
| Creating a hard link | The file's full figures, against the tree it is linked into. |
| Writing a chunk | The room the chunk could still take (next section). |
| Extending a file by truncation | The growth in length and size. |
| Appending one file's chunks to another | The growth in length and size. |
| Taking a snapshot | The source subtree's full figures, less what the snapshot overwrites. |
| Moving a subtree under other quotas | The subtree's full figures, less what the move replaces. |
| Assigning a Storage Class with higher redundancy | The growth in realsize. |
| Recovering a file from the trash into a chosen directory | The file's full figures, against the receiving tree. |
Since 5.2.0, recovering a file from the trash to its original path (trash/undel in the Meta mount) | The file's full figures, against every quota along that path. Directories that the recovery creates are not charged. |
Where the table charges size, a matching realsize charge is added. That charge is the size growth scaled by the most storage that the file's Storage Class can require across its keep and archive states.
A rename is tested only against quota directories that are above the destination and not above the source. A move that stays inside one quota tree changes no total that a quota counts. Such a move therefore never fails on quota, regardless of the state of the limits. Reading and deleting grow nothing and are never refused. Overwriting data in place can be refused only in the last chunk of the file, when that chunk is not yet full. The test below charges that chunk with the growth that the chunk could still make.
A write is tested chunk by chunk
A client writes through the write path. Each chunk that the client tries to write begins with a request to the Master Server. The Master Server runs the quota test on that request. The charge is conservative. A chunk that does not exist yet is charged at the full size that a chunk can occupy. The last chunk of the file is charged with the growth that the chunk could still make. An existing chunk in the middle of the file is charged nothing. Small files covers what a chunk occupies, and why a small file is charged far more than its length. Close to the limit, this rounding refuses a write of a few bytes into a new chunk unless a whole chunk still fits under the size limit.
A quota refusal of that request is final. The client does not retry the request, unlike after a refusal for lack of cluster space. The client marks the file's write stream as failed. The error appears on the next call that uses the stream: a later write(), an fsync(), or the close(). Because the client buffers writes, the write() call whose bytes went over the limit has usually already returned success. An application that never calls fsync() can therefore learn about the quota only at close().
An open descriptor cannot write past the quota for long. Every chunk that the descriptor touches needs a new grant from the Master Server. The stream therefore stops at the first chunk request after the quota fills. Chunks already granted are written out. The operation's closing step then tests the new file length against the length quota once more. When that final test fails, the file keeps its previous length. The application gets the same error, even though the data reached the Chunkservers.
Hard limits refuse, soft limits start a clock
The test above always enforces hard limits. A soft limit becomes part of that test only after the limit lapses.
Once a second the Master Server sweeps every quota in the instance. In the PRO Edition the sweep runs on the Leader PRO (PRO Edition only). The sweep starts the grace clock of a directory that has gone above a soft limit. The sweep stops and resets the clock of a directory that has dropped below the soft limit again. The sweep marks the quota as lapsed when the clock exceeds the grace period. From then on, every soft limit of the directory refuses operations in the same way as a hard limit. The first sweep that finds the subtree below the soft limits again removes the mark. Between the crossing of the soft limit and the mark, only the hard limits cap the growth.
The grace period belongs to the quota. mfssetquota -p sets the grace period per directory. A quota with no grace period of its own uses QUOTA_DEFAULT_GRACE_PERIOD from mfsmaster.cfg. The default is one week. Every change of the clock or of the mark is written to the metadata changelog, so the state survives a restart of the Master Server.
Over the soft limit, free space reads zero
When a client asks for free space under a quota, the answer is the room left under the limits, not the room left in the cluster. The concept page describes this behavior. The Master Server answers with the room left under the tightest of the length, size, and realsize limits on the path. The Master Server counts a soft limit from the moment the limit is set, not from the moment the limit lapses. A subtree above its soft limit therefore reports zero free space for the whole grace period, even though its writes still pass. A tool that checks free space before writing stops there, before the first EDQUOT.
Trash releases the figures, snapshots recharge them
A file in the trash counts against no directory quota. Recovering the file into a directory is tested as new growth of the receiving tree (see the table). Since 5.2.0, recovering the file to its original path is tested the same way, against every quota along that path. Trash retention covers when the trash releases and uses space. A snapshot is the opposite case. Taking a snapshot is tested against the destination's quotas with the source subtree's full figures. The full charge applies even though copy-on-write means that the snapshot shares the original chunks and adds almost nothing to the disks until the copies diverge.
The tools report the figures the tests use
mfsgetquota prints the limits and the current figures of a directory. The Quotas tab of the GUI lists every quota in the instance with its usage. The syntax of the quota tools is in command line tools.
The Quotas tab lists every quota in the instance at once, with the current value of each figure next to both of its limits and as a percentage of them.