Skip to main content

Using the file system

This page covers the everyday tasks on a mounted MooseFS file system: recovering a deleted file from the trash, deciding how long deleted files stay recoverable, taking snapshots, and inspecting files with the mfs* tools. Every task here runs on the machine that has the mount (first mount and verification leaves one) and needs no account on the cluster's servers. Who may mount what, and with which rights, is the subject of access control. Freeing space the cluster holds is the subject of space management.

Recover a deleted file from the trash​

A deleted file does not leave MooseFS at once: it waits in the trash until its trash retention passes. As long as the file is in the trash, you can recover it. Recovery has three steps: mount the meta file system, find the file's trash entry, and move the entry into the undel directory.

Mount the meta file system​

The trash is visible through a separate mount, the MooseFS Meta file system. Mount it with mfsmount. Give your Master Server's address after -H and add the mfsmeta option (-m and --meta mean the same):

mkdir -p /mnt/mfsmeta
mfsmount -H mfsmaster.example.com -o mfsmeta /mnt/mfsmeta

This mount shows a trash directory and a sustained directory instead of your files. The Master Server accepts a meta mount only from a machine that an export entry for the meta file system covers. That entry is a line in mfsexports.cfg whose directory field is .. The configuration file shipped in the packages contains such an entry for all addresses. If the mount is refused, the export list on the Master Server does not cover your machine. The exports are the administrator's side of access control.

Find the file's trash entry​

The trash directory contains the undel directory and 4096 subdirectories named 000 through FFF. All 4096 are always listed, even when they are empty. A deleted file appears in the subdirectory named by the last three hexadecimal digits of its inode number. undel is listed inside every one of those subdirectories as well. undel is one directory that is shown in many places, not 4096 separate directories. Each entry name consists of the file's inode number in hexadecimal, a | character, and the path that the file had. Every / in that path is also replaced by |. A file deleted as /mnt/mfs/projects/report.ods therefore appears as:

/mnt/mfsmeta/trash/A02/00A74A02|projects|report.ods

The path in the entry name is relative to the root of the MooseFS file system, not to your mount point. When you do not know the inode, search by name:

find /mnt/mfsmeta/trash -name '*report.ods'

Looking around the trash does not affect removal. Listing and reading entries does not touch their timestamps, so browsing does not postpone the moment when a file is removed permanently. If you prefer all entries directly in trash with no subdirectories, mount with -o mfsmeta,mfsflattrash instead. With millions of files in the trash, the subdirectories are the workable layout, because each subdirectory contains only a part of the files.

Move the entry into undel​

Recovery is a rename into the undel directory. The undel at the top of the trash and the undel listed beside the entry are the same directory, so either path works. Move the entry that find printed. The block continues the example above. The | characters mean that the shell needs the name quoted:

mv '/mnt/mfsmeta/trash/A02/00A74A02|projects|report.ods' /mnt/mfsmeta/trash/undel/

The file returns to the path recorded in the entry. The recovery recreates the directories missing along that path, owned by root with mode 0755. The recovery is refused when the name at the end of the path is already taken or the path cannot be rebuilt. The entry then stays in the trash. The exact conditions are listed with the recovery mechanism.

Verify by looking at the original path through the regular mount:

ls -l /mnt/mfs/projects/report.ods

When you are done, unmount the meta file system with umount /mnt/mfsmeta.

Recover to a different path​

A trash entry, opened as a file, contains the path that the recovery rebuilds. The entry does not contain the file's data. Reading the entry shows the path. Overwriting the entry replaces the path. To recover the file somewhere else, write the new path into the entry first. Then move the entry into undel, again with the entry name that find printed:

cat '/mnt/mfsmeta/trash/A02/00A74A02|projects|report.ods'
echo 'projects/recovered/report.ods' > '/mnt/mfsmeta/trash/A02/00A74A02|projects|report.ods'
mv '/mnt/mfsmeta/trash/A02/00A74A02|projects|report.ods' /mnt/mfsmeta/trash/undel/

What the trash cannot give back​

Only regular files reach the trash. A directory or a special file is destroyed the moment it is deleted. The same happens to a file whose trash retention was zero. There is no entry to recover. Removing an entry from the trash with rm destroys the file at once, unless a client still holds the file open. In that case the command succeeds, but the entry only moves to sustained. The space is freed when the last client that holds the file open releases it. This is how you release space held by the trash by hand, before the trash retention passes. Removing trash entries at scale is described in space management.

The sustained directory is read-only. It lists files that were deleted while some client still had them open. You cannot recover or remove such a file. The recovery call accepts only trash entries and reports a sustained file as nonexistent. A sustained file is removed automatically when the last client that holds it open releases the file. Trash and sustained files covers what keeps a file sustained.

Decide how long deleted files stay recoverable​

Trash retention is an attribute of every file and directory. It is the time that a deleted file waits in the trash before the file is removed permanently. A new file or directory inherits the current value of the directory it is created in. Changing a directory's value later does not change the children that already exist. To change a whole tree, change the value recursively. Read and set the value with mfsgettrashretention and mfssettrashretention:

mfsgettrashretention /mnt/mfs/projects

prints the value in hours, with a readable form in parentheses, for example /mnt/mfs/projects: 24 (1d).

mfssettrashretention -r 2d /mnt/mfs/projects

sets two days on the directory and everything under it. Read the value back: mfsgettrashretention /mnt/mfs/projects now prints /mnt/mfs/projects: 48 (2d). Each object under the directory prints the same value. The retention is a whole number of hours. Give it as a plain number of hours or as a period such as 12h, 2d, or 1w, as described in mfstrashretention(1). A trailing + raises only objects below the given value. A trailing - lowers only objects above it. Use these suffixes to set a minimum or a maximum on a tree without overwriting values that were set higher or lower on purpose.

To set the value on an object, you have to own the object. Root can set the value anywhere. The export that your mount uses can limit the range with its mintrashretention and maxtrashretention options. A request outside the range is refused, not trimmed. A long retention has a cost. Every file in the trash keeps its metadata in Master Server RAM and its chunks on disk until the file expires. Trash retention and its memory cost calculates that cost. A retention of zero means that deletion is final immediately.

Take a snapshot​

A snapshot is a copy of a file or a directory tree. The snapshot costs little when it is made, because it shares the source's chunks. Data is copied only when either side is written to later, chunk by chunk, as described in the copy-on-write semantics. Take one with mfsmakesnapshot:

mfsmakesnapshot /mnt/mfs/projects /mnt/mfs/backups/projects-2026-08-20

Source and destination must be on the same MooseFS instance. The tool refuses anything else. Each source argument is snapshotted atomically. The destination tree appears in one operation, so the tree reflects one moment of the source. A source directory named without a trailing slash is copied as a whole. With a trailing slash, only the directory's content is copied into the destination. If the destination file already exists, the command reports an error unless -o allows overwriting. By default, hard links inside the source are not preserved. Each name becomes an independent file in the destination. Each of those files costs its own metadata. Pass -p to preserve them. Names that are hard-linked to each other in the source are then hard-linked in the destination as well. mfssnapshots(1) lists the remaining options.

Every object a snapshot creates has the snapshot extra attribute. mfsrmsnapshot removes only objects that have the attribute, so the tool cannot remove a tree that was not made by a snapshot:

mfsrmsnapshot /mnt/mfs/backups/projects-2026-08-20

Using the tree clears the attribute. Adding an entry to a directory or removing one from it clears the attribute on that directory. Writing to a file clears the attribute on that file. No operation on the tree sets the attribute again, not even undoing the change that cleared it. The command therefore no longer treats a snapshot that somebody has written into as a snapshot. It reports Operation not permitted and removes nothing. -f removes every object that still has the attribute and leaves the rest in place: the files somebody wrote to, the directories that lost the attribute, and every directory that still holds something. An ordinary recursive delete removes what is left. The flag returns only when somebody sets it again, with mfsseteattr -f snapshot or with mfscopyeattr from an object that has the flag. Both tools require ownership of the object, not a mount admitted with admin. A new mfsmakesnapshot -o over the same destination also sets the flag again.

The removal is one Master Server operation. The removed files skip the trash. Both consequences are described in the copy-on-write semantics.

Check the attributes and the counts of the destination, not the space that the destination uses. mfsgeteattr with -r counts an attribute over a whole tree instead of printing one object's flags:

mfsgeteattr -r /mnt/mfs/backups/projects-2026-08-20

For every extra attribute, the command prints how many directories and how many non-directory objects under that path have the attribute. For snapshot, both counts have to cover the whole destination, down to the last file. By default, mfsrmsnapshot removes a directory only when everything inside it has the attribute.

The second reading is how much the destination holds. Take it on both sides with mfsdirinfo:

mfsdirinfo /mnt/mfs/projects
mfsdirinfo /mnt/mfs/backups/projects-2026-08-20

Both outputs show the same counts of inodes, directories, files, and chunks, and the same length. Both snapshot lines of mfsgeteattr -r have to match as well. The directory line matches the directory count. The line for non-directory objects matches the inode count minus the directories. size and realsize are also the same on both sides. The repeated values do not mean that the space is used twice. The counters charge a snapshot-shared chunk once per file that refers to it. A new snapshot therefore appears as large as its source, but both trees use the same chunks. To count those chunks once, run mfsdirinfo -p on a directory that contains both trees (check a file and measure a directory).

Check a file and measure a directory​

Three diagnostic tools answer the questions "where is this file", "is it fully stored", and "how big is this tree". All three are described in mfsdiagtools(1). The first two work on trash entries in a meta mount as well as on live files.

mfsfileinfo shows where a file is physically stored:

mfsfileinfo /mnt/mfs/projects/report.ods

For each chunk of the file, the command prints the chunk's identifier and version. Under that line, the command prints one line per copy or erasure-coded part, with the Chunkserver's address and port and the copy's status.

mfscheckfile gives a quick health summary of the same information:

mfscheckfile /mnt/mfs/projects/report.ods

It prints how many chunks the file has with each combination of full copies and erasure-coded parts, for example 2 full copies with the resulting Redundancy Level. It also prints separate lines for empty chunks and for chunks with no copies found.

mfsdirinfo is the MooseFS counterpart of du -s:

mfsdirinfo /mnt/mfs/projects

It prints the tree's counts of inodes, directories, files, and chunks. Then it prints length (the sum of file lengths), size (the sum of chunk lengths), and realsize (the estimated disk usage including redundancy). The output comes from the Master Server's counters, so it is immediate even for a huge tree. But those counters charge every hard link and every snapshot-shared chunk once per file that uses it. The -p option recounts precisely. It walks the tree and counts every node and chunk once, which makes it slower.

Read a file's Storage Class and extra attributes​

mfsgetsclass shows which Storage Class a file belongs to, and so how the file is stored:

mfsgetsclass /mnt/mfs/projects/report.ods

prints the class name, for example /mnt/mfs/projects/report.ods: 2CP. Assigning a different class, and every other way to control where data is stored, is an administrator's task. It is described in data placement tasks.

mfsgeteattr reads the MooseFS-specific per-object flags, the extra attributes:

mfsgeteattr /mnt/mfs/projects/report.ods

prints the flags set on the object as a comma-separated list, or - when none is set. mfseattr(1) lists the individual flags, including snapshot.