Skip to main content

Trash and sustained files

A deleted file is not removed from MooseFS at once. MooseFS keeps the file in the trash for a while. If a client still holds the file open, then the file stays until that client closes the file.

What it is​

The Global Trash is a single space per cluster that keeps files after they have been deleted. The Master Server holds the Global Trash. A file is kept there for its trash retention (the time that it stays recoverable) and is then removed permanently. Trash retention is an attribute of the object, like its permission bits. Every file and directory has its own value. A new file or directory takes the value of its parent directory. A pattern can override that value when the name is created.

The sustained area is the second holding place. It works on a different principle. Operating systems let you delete a file that is still open. On Unix, you unlink the name while a descriptor stays open. On Windows, the same operation has the names "delete on close" and "mark to delete". Sustained files come from such deletions. A sustained file is one that was deleted while a client still had the file open. It has no name in the file system any more and no time limit. It exists because somebody is still reading or writing it. It is removed when the last such reader or writer stops.

Both areas are visible through a mount of the MooseFS Meta file system (mfsmeta), which shows a trash and a sustained directory instead of your data. That mount is a separate kind of session and has to be permitted by its own export entry. Both directories belong to root and report root-only permission bits, but nothing on this mount tests those bits. stat returns these bits. No check reads them back. The export entry keeps the area out of reach, so permit that entry for administrators alone.

Why it matters​

Deleting the wrong file is the most common way to lose data. Deleting the wrong file is also the one failure that redundancy cannot help with. Every copy of the file is deleted exactly as instructed. With the trash, that mistake stays reversible for a period that you choose per directory or per file. The period can be none at all. A retention of zero means that the file is removed the moment it is deleted.

A file in the trash still holds its data. Files in the trash share disk and metadata resources with live data. A long retention on data that is replaced quickly can therefore take a large share of the cluster. Files in the trash stop counting against the quota of the directory that they were deleted from, so the space that they hold is easy to miss.

Sustained files exist because of a POSIX rule. A program that has a file open continues to read and write it after the file is unlinked. Only the last close destroys the file. MooseFS follows this rule across the network, where the process and the file are on different machines.

How MooseFS implements it​

Only regular files reach either area, and only when the name being removed is their last one. Directories and special files are destroyed at once. A file goes to the trash if its retention is more than zero. The file becomes a trash entry. That entry records the whole path that the file had. That recorded path makes the original name recoverable. Trash retention covers what that entry is made of, and the one case in which an empty file is kept. A file that does not qualify for the trash is destroyed at once. If a client holds the file open, the file goes to the sustained area instead.

The Master Server sweeps one part of the trash every second. A removal therefore happens when the sweep reaches the part that holds the entry, not at the moment the entry's retention timer runs out. Trash retention covers when an entry is ready for removal, and how much the sweep can add to that. A file can also leave the trash earlier, by recovery or by removal from the trash by hand. Recovery puts the file back under the path that it had.

A file whose time in the trash is up is not destroyed either if a client still holds the file open. Such a file moves to the sustained area and stays there. The same happens to a file removed from the trash by hand while the file is open.

Sustained files depend on the record that the Master Server keeps of the inodes that each session holds open. The client keeps that record current for as long as the client is running. The sustained area has a sweep of its own, which completes its round faster than the sweep of the trash. A file leaves the sustained area when no session holds that file any more. A client that crashes or disconnects therefore does not free its sustained files immediately. The files are removed when the Master Server drops the session.

Defaults and limits​

The tools that set trash retention read the period in hours and refuse units smaller than one hour, such as minutes and seconds. The maximum value that a file or directory can have is in Limits. The tools that set retention refuse a larger value rather than reducing it to the maximum. An export can narrow the range at both ends. A request to set a value outside that range is refused in the same way.