Trash retention and its memory cost
Deleting a file with a trash retention above zero destroys nothing at first. Until the retention runs out, the Master Server keeps the file's metadata and the Chunkservers keep the file's chunks. Trash and sustained files covers what the Global Trash and sustained files are. This page describes the mechanism of the trash and sustained files. The page covers what a trash entry is made of, how its expiry is decided, what the entry costs in RAM until then, and how a file is recovered.
A deleted file becomes a trash entry
When the last name of a file with a retention above zero is removed, the Master Server keeps the file's node. The Master Server changes the type of the node to a trash entry. The edge that held the name in its directory is removed. The entry gets a new edge in the trash tree. That edge is named with the whole path that the file had. Recovery restores that path. Deletion sets the file's change time, so the retention clock starts no earlier than the deletion itself. An empty file goes to the trash only when a session still holds it open or when KEEP_EMPTY_FILES_IN_TRASH is set in mfsmaster.cfg. An empty file has nothing in it to recover, so by default an empty file is destroyed immediately.
A session holds a file for about eleven minutes after the application closed the file, because the mount deliberately reports the inode as open for that long. Client-side background work describes the reporting and explains the delay. An empty file deleted during those eleven minutes therefore does leave a trash entry, regardless of how KEEP_EMPTY_FILES_IN_TRASH is set. The default behavior above applies only after that period has passed.
No message goes to any Chunkserver at that moment. The entry keeps the file's chunk table. Every chunk keeps the copies it had. The trash sweep flags the file's chunks as trash chunks when the sweep reaches a file whose content has not changed for five minutes. The file's retention must also not be below the minimum that the file's Storage Class sets for this. Storage Class evaluation covers which definition of the class the flag then selects.
The retention that a file takes into the trash is the file's own attribute. A pattern can have forced a different value at the moment the name was created. How policies are applied covers when patterns take effect. The mintrashretention and maxtrashretention of an export limit only a retention that a client on that export sets itself. A value inherited from the parent directory or forced by a pattern is not measured against these limits. A file can therefore have a retention that no client on that export could set on the file by hand. The limit applies to what a client may do, not to what an object may hold.
Expiry needs all three timestamps and a sweep
A trash entry is due once its access, modification, and change times are all older than the current time minus the retention. The change time is set at deletion, so the file waits at least the full retention, regardless of its other timestamps. Reading the trash through the Meta mount changes none of the three timestamps. That is by design: browsing cannot delay a removal. Searching the trash for a file to recover does not extend the stay of the entries that you look at during the search.
A due entry is not yet removed. The trash is split by inode number into 4096 parts. The Master Server examines one part per second, on a timer of its own rather than inside the chunk loop or the file loop. A full round over the trash takes about 68 minutes, so an entry can remain in the trash for up to that long after its retention ends.
A due entry is not destroyed when a client still holds its file open. The entry moves to the sustained area and waits there with no time limit. Deletion covers how the destruction itself proceeds: the deletion of the chunks and the freeing of the space. The role of the trash ends when the entry expires.
The entry costs RAM like a live file
A trash entry is the same node that a live file has. The Master Server keeps this node in RAM with attributes, timestamps, Storage Class, and the chunk table. The entry's edge holds the entire former path, but a live file's edge holds one name. The entry therefore costs slightly more than the file did in the namespace. What metadata holds lists those records and their cost per object. Master Server memory explains why that cost is the ceiling on how far a cluster grows.
The number of trash entries scales with turnover rather than with the amount stored. At a steady deletion rate, the trash holds that rate times the retention in entries. A long retention on a file system that deletes often keeps every deleted generation in RAM next to the live generation. The Master Server's memory therefore grows even though the visible file count does not.
The retention value itself is stored on every object as a whole number of hours, in a 16-bit field. That field sets the ceiling on the value. The tools that set retention enforce that ceiling. The tools read the period in hours and refuse a larger period with value too big in time period. No request past the ceiling therefore reaches the Master Server.
The tools read the period in hours, not in seconds. A bare number is a number of hours. The tools accept the suffixes h, d, and w, and refuse s and m. A fraction of an hour is rounded to the nearest full hour, so 0.4h becomes none. A half rounds up, so 1.5h becomes two hours. A period under thirty minutes therefore becomes a retention of zero. That retention gives no trash entry at all, not a short stay in the trash. A file whose retention is zero is destroyed at the moment its last name is removed. The file leaves nothing in the trash to recover.
A client sends the Master Server a number of seconds. The Master Server rounds a value that is not a whole number of hours up to the next full hour. The tools have already made the value whole, so for values from the tools that second rounding changes nothing.
Recovery rebuilds the path, but sustained files are lost
Recovering a trash entry turns the entry's node back into a file. The recovery links that file under the path stored on the entry's edge. Directories missing along that path are created as part of the recovery. They are owned by root and have mode 0755. The recovery is refused when the name at the end of the path is already taken, or when a component of the path exists but is not a directory. The recovery is also refused when a directory along the path is immutable or is under an exceeded quota. Since 5.2.0 it is also refused when the file's figures would take a quota along the path over its hard limit, or over a soft limit whose grace period has run out. The refusal reports Disk quota exceeded. A refused entry stays in the trash. The recovery removes the trash flag from the file's chunks.
Recovery works through a mount of the MooseFS Meta file system. To recover an entry, move the entry from the trash directory into trash/undel. The mount itself is covered in trash and sustained files, including what the mount shows and who may use it. The metafiles that a mount answers with are listed under special files in a mount.
A sustained file is not recoverable. The recovery call accepts only a trash entry and reports anything else as nonexistent, including a sustained file. The sustained directory can therefore be read, but nothing in it can be moved or changed. A sustained entry ends without recovery. The Master Server sweeps the sustained area in 256 parts, one part per second. The Master Server destroys an entry as soon as no session holds the file open.
The session and the sweep both take time, but the session takes longer. The session releases the file about eleven minutes after the last application closed that file or died while holding it. The sweep then needs up to a little over four more minutes to reach the entry's part. A sustained entry therefore outlives its last holder by roughly a quarter of an hour. Most of that time is spent waiting for the session to release the file.