Skip to main content

Frequently Asked Questions

This page lists the questions asked most often, with short answers. Where a subject has a page of its own, the answer links to that page.

Find your question by topic​

Speed and resources

Disks and Chunkservers

Files, space, and limits

Keeping the data

Access, network, and platform

Checking on the cluster

What average write/read speeds can we expect?​

As in most file systems, the speed depends on the block size and on the type of access (sequential or random). In MooseFS, it also depends on the hardware. The main factors are the performance of the hard drives, and the capacity and topology of the network (network latency). Faster hard drives and higher network throughput give higher performance of the whole system. Bandwidth and latency covers which of the two your own workload depends on.

Does the Storage Class influence writing/reading speeds?​

Generally speaking, it does not. For reads, more than one copy can sometimes make the operation faster. When two clients access a file kept in two or more copies, they can read from different copies. Each client then has all the available throughput for itself. On average, though, the Storage Class does not change the speed of reads. Erasure Coding is the exception: the client reads the parts of a chunk in parallel, one connection for each part.

Similarly, the Storage Class has a negligible effect on the write speed. Every write goes down a chain, whatever the number of copies. The client sends the data to one Chunkserver. That Chunkserver stores the data and at the same time passes it to the next Chunkserver, which does the same, until the chain ends. The client therefore sends the data once, however many copies the chunk has. The copies are written almost at the same time. The mechanism is described on copy. In tests, a write can use all the available bandwidth on the client's side of a 1 Gbps network.

Are concurrent read and write operations supported?​

All read operations are parallel: several clients can read the same data at the same moment. Write operations are parallel too, except operations on the same chunk (fragment of a file). The Master Server synchronizes those operations, so they must be sequential. Several mounts at once describes what one mount sees of the writes made by another mount.

How much CPU/RAM resources are used?​

Take one installation: about 1 PiB of total space and 36 million files in 6 million folders, stored in 38 million chunks on 100 machines. There, the Chunkserver CPU usage under constant file transfer is about 15-30%. Chunkserver RAM usage usually stays between 100 MiB and 1 GiB. That range depends on the number of chunks on each Chunkserver. The Master Server uses about 50% of a modern 3.3 GHz CPU (about 5000 file system operations per second, of which about 1500 are modifications) and 12 GiB of RAM. CPU load depends on the number of operations. RAM usage depends on the number of objects and on the number of chunks. The length of a file decides how many chunks the file uses, one for each 64 MiB. The RAM usage is proportional to the number of entries in the file system, because the Master Server process keeps all metadata in memory for performance. HDD usage on that Master Server is about 22 GB.

Those figures are readings taken from one installation of that shape, not a specification. A cluster with a different object count and a different workload uses different amounts. The readings do not scale by any single ratio. Sizing the cluster calculates the figures from what your own cluster will store. Master Server RAM from the object count calculates the memory part record by record.

Is it possible to add/remove Chunkservers and disks on the fly?​

You can add and remove Chunkservers on the fly. Avoid disconnecting a Chunkserver if it holds the only copy of a chunk in the file system (see Chunk versions and states). You can also disconnect (change) an individual hard drive. The procedure is:

  1. Mark the disks for removal (see How to mark a disk for removal?)
  2. Reload the Chunkserver process
  3. Wait until the cluster reports the disk as removal ready on the Disks tab of the GUI. That state means that a full pass over the chunk table found no chunk that still depends on the disk. See Disk errors and removal
  4. Delete the entries of the disconnected disks in mfshdd.cfg
  5. Reload the Chunkserver process
  6. Unmount the disks
  7. Stop the Chunkserver machine
  8. Remove the hard drives
  9. Start the machine

Steps 7 and 9 are needed by the hardware, not by MooseFS. They are in the list because a drive that cannot be removed from a running machine has to be removed with the machine off. MooseFS does not need either of them. MooseFS also never needs the Chunkserver process stopped. On reload, the Chunkserver reads mfshdd.cfg again, stops using every disk whose line was removed from the file, and keeps serving the disks that are still listed.

For hot-swap disks, follow these steps instead:

  1. Mark the disks for removal (see How to mark a disk for removal?)
  2. Reload the Chunkserver process
  3. Wait until the cluster reports the disk as removal ready on the Disks tab of the GUI
  4. Delete the entries of the disconnected disks in mfshdd.cfg
  5. Reload the Chunkserver process
  6. Unmount the disks
  7. Remove the hard drives

If you follow the steps above, the work of client computers is not interrupted, so MooseFS users do not notice the operation.

How to mark a disk for removal?​

To mark a disk for removal from a Chunkserver, edit the Chunkserver's mfshdd.cfg configuration file. Put an asterisk * at the start of the line for the disk that you want to remove. In this example, /mnt/hdd is marked for removal:

/mnt/hda
/mnt/hdb
/mnt/hdc
*/mnt/hdd
/mnt/hde

After changing mfshdd.cfg, reload the Chunkserver with mfschunkserver reload. Starting and stopping the cluster covers the systemd units that wrap the same action.

After the disk is marked for removal and the Chunkserver has reloaded its configuration, the system makes more copies of the chunks stored on this disk. This keeps the required number of copies. A restart is not needed. The Chunkserver acts on a reload.

Finally, before you disconnect the disk, wait until the cluster reports the disk as removal ready. The Disks tab of the GUI shows that status. The status is a direct answer. The Master Server sets that status after a full pass over its chunk table in which no chunk depended on the marked disks. The Master Server computes the status per Chunkserver, not per disk, so two disks marked on the same server become ready together. The chunk state matrix gives the same answer indirectly. That indirect reading is misleading when redundancy is missing somewhere else in the cluster at the same time.

My experience with clustered file systems is that metadata operations are quite slow. How is this solved in MooseFS?​

Metadata operations are fast in MooseFS because the whole file system structure is kept in RAM on the Master Server. The Master Server therefore needs more memory. The Master Server frequently saves the metadata to files.

A Metalogger receives every metadata change immediately and writes the changes to its own file system. Both editions include a Metalogger.

In MooseFS PRO, Metaloggers are optional, because the Follower Master Servers keep synchronized with the Leader. The Follower Master Servers also save metadata to their hard disks.

What does the value of directory size mean in MooseFS? It is different from standard Linux ls -l output. Why?​

A directory's size has no fixed meaning in any file system, so MooseFS uses it for extra information. The number represents the total length of all files inside the directory (as in mfsdirinfo -h -l), displayed in an exponential notation.

You can convert the directory size as follows:

The number has 7 digits: xAAAABB. To convert this notation to a number of bytes, use the following expression:

AAAA.BB xBytes

Where x selects the prefix:

0 = (none: plain bytes)
1 = kibi
2 = mebi
3 = gibi
4 = tebi
5 = pebi
6 = exbi

Example:

To convert the following entry:

drwxr-xr-x 164 root root 2010616 May 24 11:47 test
                         xAAAABB

Read the folder size 2010616 as 106.16 MiB. When x = 0, the number can be shorter:

Example:

Folder size 10200 means 102 bytes.

When I perform df -h on a file system the results are different from what I would expect from the actual sizes of written files. Why?​

Every Chunkserver sends its own disk usage, increased by 256 MiB for each partition or disk in use. The Master Server sends the sum of these values to the client as the total disk usage. With 3 Chunkservers of 7 disks each, the reported disk usage grows by 3 × 7 × 256 MiB (about 5 GiB). That figure covers the paths in mfshdd.cfg that have no space option of their own. A path with a space option of its own (a limit, a reserve, a percentage, or the shared indicator =) is measured according to that option. Raw space and user space explains this.

The other source of differences is that df counts everything on the disk. When the Chunkserver disks are used only for MooseFS, df shows the correct disk usage. If you keep other data on them, df counts your own files too.

To see the actual space usage of your MooseFS files, use the mfsdirinfo command.

Can I keep source code on MooseFS? Why do small files occupy more space than I would have expected?​

MooseFS was initially designed to store large numbers (for example, several thousand) of very big files (tens of gigabytes). It has a hard-coded chunk size of 64 MiB and a block size of 64 KiB. A consistent block size helps network performance and efficiency, because all nodes in the system work with a single 'bucket' size. Even a small file therefore occupies 64 KiB plus 4 KiB of checksums and 4 KiB for the header. Small files explains this, including the effect of Erasure Coding and of several copies.

The space that a small file occupies inside a MooseFS chunk is noticeable, but still negligible. Take 25 million files kept in two copies. Those files hold 25 million chunks, stored as 50 million copies of 72 KiB each. Most of those copies are not completely used, because the file is smaller than the block that each copy uses. Those 50 million copies take about 3.35 TiB in all. By modern standards, this amount is not a significant concern. A more typical, medium to large project with 100,000 small files in two copies takes about 13.7 GiB. This is again because MooseFS uses a whole 64 KiB block for each copy of each file.

You can therefore store source code files on MooseFS, either for active use during development or for reliable long-term storage or archiving.

The comfort of developing the code may matter more. That comfort depends on the performance of a network file system. For a project under active development, a network file system can perform file IO operations more slowly than a directly attached hard drive. The same applies to any other network-based file system, such as NFS or CIFS.

Some modern integrated development environments (IDE), such as Eclipse, make frequent IO requests on several small workspace metadata files. If you run Eclipse with the workspace folder on a MooseFS file system (or any other network file system), the user interface is slightly slower than with a workspace on a local hard drive.

Evaluate for yourself whether MooseFS is right for the working copy that you actively develop in an IDE.

A typical text editor with a version control system, such as Subversion, is a different case. Checking out project files into a MooseFS file system usually causes no loss of performance. The larger IO latency of the remote Subversion repository outweighs the IO overhead that MooseFS adds as a network file system. Individual file operations (open, save) have no observable latency when you use simple text editors (outside of complicated IDE products).

A more likely setup keeps the Subversion repository files on a MooseFS file system. In that setup, svnserve or Apache + mod_svn serves requests to the Subversion repository. Users check out working sandboxes onto their local hard drives.

Do Chunkservers and the Master Server do their own checksumming?​

Chunkservers do their own checksumming. Every chunk file has a checksum table of 4 KiB. The table holds a 4-byte checksum for each 64 KiB block. The Chunkserver writes the whole table when it creates the file. The size of the table is fixed, not proportional: a chunk with a single block has the same 4 KiB table as a chunk with all 1024 blocks.

The Master Server checksums metadata as well. It writes a checksum with every metadata image that it saves. In MooseFS PRO, the Leader compares the checksums reported by the Followers with its own PRO (PRO Edition only). Metadata ID, version, delay, and checksum describes what those numbers decide.

Run the Master Server on ECC RAM. Requirements for Master Servers explains why: the whole file system metadata is kept in the memory of that machine. The checksum of a saved copy covers whatever the memory held at that time. No checksum can detect a bit that was already flipped in RAM.

What resources are required for the Master Server?​

RAM on the Master Server machine matters most, because the whole file system structure is held in RAM, not read from disk. Master Server RAM from the object count calculates the memory record by record. Besides RAM, the Master Server machine needs some space on its hard disk for the metadata images and the changelogs. Requirements for Master Servers gives the formula for that space.

The size of a metadata image depends on the number of objects and on the number of chunks, so a longer file counts for more than a shorter file. The size of the changelogs depends on the number of operations per hour, but you can configure how long (in hours) the changelog is kept.

When I delete files or directories, the MooseFS size doesn't change. Why?​

MooseFS does not erase files immediately on deletion, so that you can undo the deletion. Deleted files are kept in the trash for the configured amount of time before they are erased.

You can configure how long files are kept in the trash and empty the trash manually (to free the space). Deletion describes the mechanism. Recovering a deleted file from the trash describes the procedure.

In short, check how long a deleted file is kept with the mfsgettrashretention command, and change it with mfssettrashretention.

When I added a third server as an extra Chunkserver, it looked like the system started replicating data to the 3rd server even though the file was still kept in 2 copies.​

Yes. Rebalancing moves each chunk independently, so the chunks of one file can be spread across all of your Chunkservers. Rebalancing covers both mechanisms that move chunks and what limits each of them.

Is MooseFS 64-bit compatible?​

Yes. Operating systems covers which word sizes the packages are built for, and why to choose 64-bit.

Can I modify the chunk size?​

No. File data is divided into fragments (chunks) with a maximum of 64 MiB each. The value of 64 MiB is hard-coded into the system, so you cannot modify it. Chunk covers both sizes of the storage format. The chunk size was chosen from real-world data. It is a very good compromise between the number of chunks and the speed of rebalancing and updating the file system. If a file is smaller than 64 MiB, it occupies less space.

In production systems, individual file sizes significantly exceed 100 GB with no noticeable chunk size penalty.

How do I know if a file has been successfully written to MooseFS?​

Writing to the file system goes through a buffer. This affects programs. What the client does in the background covers when the buffer is emptied and which call returns a write error.

In all current file systems, files are written through a buffer (write cache). Therefore, the write command itself only transfers the data to the buffer (cache) and does not write it to disk. A successful write command does not mean that the data has been correctly written to a disk. All the data in the buffers (cache) is physically written out only when the fsync (or close) command is called and completes. If an error occurs during that write, the fsync (or close) command can return an error.

However, the vast majority of programmers do not check the status of the close command, which is a very common mistake. Therefore, a program can "assume" from a successful write that the data has been written correctly. In fact, the write can fail during the later close command.

Because of how network file systems (like MooseFS) work, more data is, on average, "left over" in the buffers (cache) than in regular file systems. Therefore, the close or fsync command often processes a significant amount of data. If an error occurs during that write, the close or fsync command returns that error. So, especially on MooseFS, run fsync after writing to a file and before running close. Then check the result of fsync. Also check the return status of close.

Note: when stdio is used, the fflush function only executes the write command. A successful fflush is therefore not enough to be sure that all the data has been written successfully. Check the status of fclose too.

This problem can also occur when you redirect the standard output of a program to a file in a shell. Bash (and many other programs) do not check the status of close. So the application > outcome.txt syntax can finish successfully in the shell even though writing the outcome.txt file failed. Avoid this syntax for redirecting output in the shell when you write to a MooseFS mount point. If necessary, create a simple program that reads the standard input, writes everything to a chosen file, and correctly checks the result status of fsync. Then use application | mysaver outcome.txt, where mysaver is that program, instead of application > outcome.txt.

This problem is common and does not come directly from MooseFS itself. It can affect any file system. Network file systems are more exposed to it. Follow the recommendations above at all times, also on classic file systems.

What are limits in MooseFS (e.g. file size limit, file system size limit, max number of files that can be stored on the file system)?​

  • The maximum file size in MooseFS is 257 bytes = 128 PiB.
  • The largest capacity MooseFS can represent is 264 bytes: 16 EiB, less one byte. That number is the width of the counter that holds the figure, not a limit that the product enforces. Nothing caps total capacity. Every disk is accepted, however large it makes the cluster. Max cluster capacity describes the counter along its whole path.
  • The maximum number of objects that can be stored on one MooseFS instance is 231 - 2, over 2.1 billion. Files, directories, and links all count toward that one limit.
  • Limits lists the other limits of MooseFS, and names what sets the maximum where MooseFS sets none.

Can I set up HTTP basic authentication for the mfscgiserv?​

mfscgiserv serves nothing any more. The web interface has its own daemon, mfsgui. mfscgiserv installs only as a stub that prints where the interface moved to. The page on installing the server components covers both. The GUI server has no authentication of its own and serves plain HTTP, so you add authentication in front of it. Bind it to a local address. Put an authenticating reverse proxy, or at least a firewall, between it and everyone else. The procedure is described on access control.

Can I run a mail server application on MooseFS? Mail server is a very busy application with a large number of small files - will I not lose any files?​

You can run a mail server on MooseFS. You won't lose any files under a large system load. When the file system is busy, it blocks until its operations are complete. This slows the mail server down.

Are there any suggestions for the network, MTU, or bandwidth?​

Use jumbo frames (MTU 9000) when you control every device on the network path and can set the same MTU on all of them. Use 1500 when you cannot. See MTU. With more Chunkservers, connect the switches through optical fiber or use aggregated links.

Does MooseFS support supplementary groups?​

Yes. POSIX compliance covers how the client collects the list and what a mount without its group cache sends instead.

Does MooseFS support file locking?​

Yes, in MooseFS 3.0 or newer. File locking covers both families of lock and what a mount can turn off.

Is it possible to assign IP addresses to Chunkservers via DHCP?​

Yes. A Chunkserver that reconnects with a different address keeps its place, because the Master Server identifies the Chunkserver by an identifier, not by its address. Reserve the addresses anyway: export entries, the address map, and the topology file all use addresses. In an instance divided into locations, an address from the wrong range disconnects the Chunkserver PRO (PRO Edition only). Addresses and ports covers all of this.

Some of my Chunkservers utilize 90% of space, but others only 10%. Why does the rebalancing process take so long?​

Production experience has shown that aggressive replication is not desirable, because it can substantially slow down the whole system. The overall performance of the system is more important than equal use of hard drives across all of the Chunkservers. By default, replication is not aggressive. In practice, a new Chunkserver usually needs about a week to reach the standard disk usage. Aggressive replication would make the whole system considerably slower for several days.

These two options set the replication speeds. A reload of the Master Server applies them:

  • CHUNKS_WRITE_REP_LIMIT

    How much replication one Chunkserver may receive at once (default is 2,1,1,4,4).

    The caps count redundancy rather than transfers. A whole copy counts as one against the cap, and an Erasure Coding part as half of one. A Chunkserver under a cap of 1 can therefore receive two parts at once, but only one whole copy. Replication priorities describes the five classes.

    The option accepts five limits separated by commas. One number sets all five limits to that value.

    1. The first limit is for endangered chunks: a single remaining copy, or exactly as many Erasure Coding parts as a read needs
    2. The second limit is for every other repair in the queues, including undergoal chunks
    3. The third limit is for rebalance between servers whose space usage differs little
    4. The fourth limit is for rebalance between other servers (space usage differs a lot)
    5. The fifth limit is for replications triggered by client read and write operations, such as recovering missing Erasure Coding parts

    Usually, the first number should be greater than or equal to the second, the second greater than or equal to the third, and the fourth greater than or equal to the third (1st >= 2nd >= 3rd <= 4th). The fifth number should be equal to or greater than any of the other four numbers. The Master Server does not verify these relations.

  • CHUNKS_READ_REP_LIMIT

    How much replication one Chunkserver may send out at once (default is 10,5,2,5,10), counted the same way.

    The option accepts five limits separated by commas. One number sets all five limits to that value. Limit groups are the same as in the write limit. Relations between numbers should be the same as in write limits (1st >= 2nd >= 3rd <= 4th, fifth at least as high as the other limits).

Tuning these options for your environment requires some experimenting.

I have a Metalogger running - should I make additional backup of the metadata file on the Master Server?​

Yes. Make an additional backup of the metadata file. This backup is a worst-case recovery option if the Metalogger data cannot be used to restore the Master Server (for example, because the Metalogger machine is also destroyed).

The Master Server writes the metadata kept in RAM to the metadata.mfs.back binary file every hour on the hour (xx:00). Copy the metadata file every hour on the half hour (30 minutes after the dump). This limits data loss to about 1.5 hours of data. You can back up the file with any usual copy tool, for example cp, scp, or rsync.

After you restore the system from this backed-up metadata file, the most recently created files are lost. Files that were appended to have the size that they had at the time of the metadata backup. Files that were deleted exist again. Files that were renamed or moved have their previous names (and locations) again. You still have all the data of every file created before the metadata backup.

In MooseFS PRO, the Leader does not write the hourly dump itself. At each save interval, the Leader has its synchronized Followers store a dump. The Leader writes its own dump on the consistency-check schedule, once a day by default. Downloading a dump from a Follower is a separate setting, off by default. Both schedules, and their cost in changelog files, are described on changelogs and dumps. The newest dump in the Leader's own directory can therefore be a day old. Take the copy from a Follower, or force a save first: Master Server operations gives the command.

I think one of my disks is slower / damaged. How should I find it?​

On the Disks tab of the GUI, switch the "I/O stats" columns to "Last hour", choose "Max time", and sort the results by the "write" column. Then look for disks that have a significantly longer write time. You can also sort by the "fsync" column and check the results. Find the individual disks that are slower, because they can be a bottleneck for the system.

To get enough load on the system for visible statistics in the GUI, you can run a test operation that continuously copies some data. In that case, switch the "I/O stats" columns to "Last minute" instead of "Last hour".

MooseFS can also mark the slow operations for you. HDD_IO_SECONDS_TO_ERROR in mfschunkserver.cfg records an operation on a chunk file that took longer than the given number of seconds, even though the operation succeeded. The record adds the hang to the error list of the disk. There, a disk that is slow but not broken becomes visible. The option is off by default. Since 5.2.0, such a record counts toward the error tolerance that takes a disk out of service, described in Disk errors and removal. Up to 5.1.0, it does not.

After you find a "bad" disk, replace it with the usual procedure. Mark the disk for removal, reload the Chunkserver, and wait until the Disks tab reports the disk as removal ready. That status means that a full pass over the chunk table found no chunk that still depends on the disk, so everything the disk holds is stored elsewhere as well. The statuses that come before it, removal not ready (validating) and removal not ready (in progress), are listed on the same page.

How can I find the Master Server PID?​

Run the following command:

mfsmaster test

The command prints mfsmaster pid: PID for a running process, or says that the Master Server is not running. Starting and stopping the cluster covers the rest of the process controls.

Web interface shows chunks awaiting deletion. What does it mean?​

This is how MooseFS counts chunks that belong to files that no longer exist, that is, deleted files. MooseFS deletes files asynchronously. First, a file is removed from metadata. Its chunks are counted as pending deletion. The chunks themselves are removed later, when the periodic pass of the Master Server over every chunk reaches them. The Master Server also caps how many deletions it orders at once. This is less disruptive than erasing everything at the exact moment the file was deleted. The GUI and mfscli both name this count pending deletion. Deletion describes the path of a chunk from the unlink to the freed space.

Such chunks may also appear after a recovery of the Master Server, if they were created shortly before the failure and were not available in the restored metadata file.

Is every error message reported by mfsmount a serious problem?​

No. Most of them report a retry, not a lost operation. mfsmount does not write every failure in communication with Chunkservers to the syslog. It starts logging a request only after that request has already failed mfslogretry times, 5 by default. Attempts below that count are not logged. Transient network problems can cause IO errors to be displayed. This does not mean data loss, and it does not mean that mfsmount returns an error code to the application. The client (mfsmount) retries each operation several times. The error is returned to the application only after the number of failures (reported as try counter) reaches mfsioretries, which is 30 by default.

Monitor these messages. When messages appear more often from one Chunkserver than from other Chunkservers, that Chunkserver may have a problem. A hard drive may be broken. A network card may have problems. In the GUI, check the charts of that Chunkserver and the operation times of its hard disks.

Note: XXXXXXXX in these examples means the IP address of the Chunkserver, written in human-readable form.

Example 1:

file: NNN, index: NNN, chunk: NNN, version: NNN - writeworker: connection with (XXXXXXXX:PPPP) was timed out (unfinished writes: Y; try counter: Z)

This means that the Zth try to write the chunk was not successful. The Chunkserver did not confirm the write of Y blocks that were sent to it. After the client reconnects, it sends these blocks again. The default retry limit is 30.

This message is informational and doesn't mean data loss.

Example 2:

file: NNN, index: NNN, chunk: NNN, version: NNN, part: P - readworker: connection with (XXXXXXXX:PPPP) was timed out (lastrcvd:TTT,now:TTT,lrdiff:DDD received: R/T, try counter: Z)

This means that the Zth try to read the chunk was not successful. The client tries to read the chunk again. received: R/T shows how much of the requested data had arrived before the connection stopped sending data. A single such message means a transient problem and needs no action. The read fails for the application only when Z reaches the retry limit. Read failures are reported the same way for other causes, for example readworker: connection with (XXXXXXXX:PPPP) was reset by peer / POLLHUP. The default retry limit is 30.

How do I verify that the MooseFS cluster is online? What happens with mfsmount when the Master Server goes down?​

If the Master Server stops while mfsmount is already running, mfsmount does not disconnect the mounted resource. Files waiting to be saved stay in the queue for quite a long time, while the client tries to reconnect to the Master Server. After a set number of tries, those files return EIO, "input/output error". However, mfsmount does not start when the Master Server is offline. mfsmount registers with the Master Server before the command returns. If the registration fails, the mount ends. -o mfsdelayedinit moves that registration into the background, so the mount command succeeds without a Master Server. The background registration makes an /etc/fstab entry independent of the order in which services start at boot. The option does not make the file system usable immediately. Until the registration succeeds, operations on the mount point return an error instead of waiting for that registration.

There are several ways to make sure that the Master Server is online. A few of them:

Check if you can connect to the TCP port of the Master Server (for example, with a socket connection test).

To make sure that a MooseFS resource is mounted, it is enough to check the inode number: the MooseFS root always has inode 1. For example, with MooseFS mounted in /mnt/mfs, the stat /mnt/mfs command (on Linux) shows:

stat /mnt/mfs
File: `/mnt/mfs'
Size: xxxxxx Blocks: xxx IO Block: 65536 directory
Device: 13h/19d Inode: 1 Links: xx
(...)

mfsmount also creates a virtual hidden file .stats in the root of the mounted folder. To get the statistics of mfsmount when MooseFS is mounted, read this .stats file. A client built against FUSE 7.31 returns a set of rows like these:

cat /mnt/mfs/.stats
fuse_ops: 12
fuse_ops.statfs: 1
fuse_ops.access: 1
fuse_ops.lookup: 1
fuse_ops.lookup.internal: 1
fuse_ops.lookup.master: 0
fuse_ops.lookup.master.error: 0
fuse_ops.lookup.master.negative: 0
fuse_ops.lookup.master.positive: 0
fuse_ops.lookup.cached: 0
fuse_ops.lookup.cached.negative: 0
fuse_ops.lookup.cached.readdir: 0
fuse_ops.getattr-cached: 0
fuse_ops.getattr: 4
fuse_ops.setattr: 0
fuse_ops.mknod: 0
fuse_ops.unlink: 0
fuse_ops.mkdir: 0
fuse_ops.rmdir: 0
fuse_ops.symlink: 0
fuse_ops.readlink: 0
fuse_ops.readlink.cached: 0
fuse_ops.readlink.master: 0
fuse_ops.rename: 0
fuse_ops.link: 0
fuse_ops.opendir: 1
fuse_ops.readdirplus: 2
fuse_ops.readdir: 0
fuse_ops.readdir.with_attrs+: 0
fuse_ops.readdir.without_attrs: 0
fuse_ops.readdir.with_attrs: 0
fuse_ops.releasedir: 1
fuse_ops.create: 0
fuse_ops.open: 1
fuse_ops.release: 0
fuse_ops.read: 0
fuse_ops.write: 0
fuse_ops.flush: 0
fuse_ops.fsync: 0
fuse_ops.setlk: 0
fuse_ops.getlk: 0
fuse_ops.flock: 0
fuse_ops.removexattr: 0
fuse_ops.listxattr: 0
fuse_ops.getxattr: 0
fuse_ops.setxattr: 0
symlink_cache.#links: [0]
symlink_cache.search_misses: 0
symlink_cache.search_hits: 0
symlink_cache.inserts: 0
master.usec_timediff: [134]
master.usec_ping: [121]
master.reconnects: 19
master.bytes_sent: 851282
master.bytes_received: 837863
master.packets_sent: 87082
master.packets_received: 64823

A counter in square brackets is a current value. The other counters count up from the moment the mount started. Writing to the file resets those counters to zero. The mount provides other special files as well: Special files in a mount lists them.

Your own mount shows a similar set of rows, not the same set. The counters that a mount registers depend on the FUSE version that the client was built against, and on the mount options. fuse_ops.readdirplus and fuse_ops.readdir.with_attrs+ need FUSE 3. A section tied to a cache appears only when that cache is on. For that reason, the output above has no negentry_cache section: mfsnegentrycacheto is 0.0 by default.

To check that the Master Server responds correctly, read the Storage Class of any object, for example of the root folder:

mfsgetsclass /mnt/mfs
/mnt/mfs: 2CP

The command prints the Storage Class currently assigned to the root folder. 2CP is the class assigned to / on a new installation. If the command returns a class name, the Master Server is running.

I found that MooseFS trash takes too much of my disk space. How to clean it up manually?​

To empty the MooseFS trash, mount the special directory called "MooseFS Meta".

Create a mount point for the MooseFS Meta directory first:

mkdir /mnt/mfsmeta

and mount mfsmeta:

mfsmount -o mfsmeta /mnt/mfsmeta

If your Master Server host name differs from the default mfsmaster, or the port differs from the default 9421, use the matching option, for example:

mfsmount -H master.host.name -P PORT -o mfsmeta /mnt/mfsmeta

Then you can find your deleted files in the /mnt/mfsmeta/trash/SUBTRASH directory. A subtrash is a directory inside /mnt/mfsmeta/trash named with three uppercase hexadecimal digits. There are 4096 of them, 000 through FFF. All of them are listed whether or not they hold any files. Subtrashes help when you have many files in the trash (for example, millions), because you can run Unix tools such as find on each subtrash. With all the files in one directory, such tools may fail.

If you do not have many files in the trash, mount Meta with the mfsflattrash parameter:

mfsmount -o mfsmeta,mfsflattrash /mnt/mfsmeta

or if you use a Master Server host name or port other than the default:

mfsmount -H master.host.name -P PORT -o mfsmeta,mfsflattrash /mnt/mfsmeta

In this case, your deleted files are directly in /mnt/mfsmeta/trash (without a subtrash).

In both cases, remove files with rm file. To undelete files, move them to the undel directory in the trash or subtrash (mv file undel).

If you do not want certain files moved to the trash at all, set their "trash retention" (in hours) to 0 before deletion. If you set a specific trash retention for a directory, all the files created in this directory inherit that trash retention from the directory, for example:

mfssettrashretention 0 /mnt/mfs/directory

You can also set the trash retention to a different value, for example 1 hour:

mfssettrashretention 1 /mnt/mfs/directory

For more information on the parameters of mfsmount or mfssettrashretention, see man mfsmount and man mfstrashretention.

What is the minimal number of Chunkservers in MooseFS?​

In the Community Edition (CE), the minimum number of Chunkservers equals the highest number of copies that any Storage Class configured in the system requires.

Erasure Coding sets its own minimum, which is two numbers, not one. Chunks are converted into an @N+X format only where N+2X Chunkservers are present. Chunks already in that format are kept in it where N+X Chunkservers are present. For the two schemes that the Community Edition accepts, these numbers are 6 and 5 for @4+1, and 10 and 9 for @8+1. Below the lower count, no data is lost: the chunks are converted back into whole copies. Each location counts its own Chunkservers PRO (PRO Edition only). Chunkservers needed for an Erasure Coding scheme lists both figures for every scheme.

In MooseFS PRO, the election that picks the Leader sets a second minimum. This minimum does not depend on Storage Classes. A Master Server keeps the Leader role only as long as the disconnected Chunkservers stay below half of those registered. With two registered Chunkservers, the loss of one leaves the instance with no Leader at all. Three Chunkservers is the smallest number that survives losing one. Leader election and quorum covers the count.