Alerts

What a MooseFS cluster tells you about its own state. The GUI marks the object that an alert concerns and shows its message together with the explanation and the fix given here. The sections say which part of the cluster an alert is about. Within a section the errors come first.

Cluster

Severity	Message	What it means	Suggested fix
Error	Licence doesn't allow using this version of software.	Licence is granted for the previous version of the software. You are not entitled using this version of MooseFS any longer. Please downgrade or contact support.
Error	The amount of data in the cluster exceeds the allocated license. New data cannot be written to the cluster.	The license you are using is limited in size. The total size of data stored in the cluster now exceeds the granted license. This is a critical situation. No new data may be written. The cluster resiliency is degraded as data management routines have no space to work with. Note that license size refers to so-called raw data size, i.e., data after required duplications. It is usually larger than the user data size as most of the data is redundant.	It is urgent to extend the license size or replace it. It is also possible to remove unnecessary data (snapshots included) or reduce the level of data redundancy. Additionally, data can be removed from the system trash, or the retention time for data in the trash can be shortened (see mfssettrashretention). Contact support for further assistance.
Error	The amount of data in the cluster may exceed the available license any moment. New data will not be able to be saved then.	The license you are using is limited in size. The total size of data stored in the cluster now exceeds 95% of the granted license. This is a critical situation. This means that the available space in the cluster may be exhausted at any moment. As a result, some cluster functionality will become unavailable. In particular, it will not be possible to add new data, and the cluster resiliency may be degraded. Note that license size refers to so-called raw data size, i.e., data after required duplications. It is usually larger than the user data size as most of the data is redundant.	It is urgent to extend the license size or replace it. After the stored data size hits the license limit, the cluster will not work as intended: no new data may be written. It is also possible to remove unnecessary data (snapshots included) or reduce the level of data redundancy. Additionally, data can be removed from the system trash, or the retention time for data in the trash can be shortened (see mfssettrashretention). Contact support for further assistance.
Error	The cluster has only one metadata server, so no backup metadata records are available.	The cluster has only one metadata server. Additional Metaloggers or Followers (for the PRO version) should be used to maintain additional metadata records. A single metadata server is a single point of failure in the cluster. If it fails, the entire cluster becomes unavailable.	Install additional Metaloggers or install additional Followers if you use a PRO version. Follower's advantage is that it can automatically become the new Leader Master server if the previous one fails. Metalogger keeps track of metadata changes but needs to be upgraded to the leading Master server manually. Always avoid having just a single metadata Master server. Contact MooseFS support if you need assistance.
Error	The license expired.	The license you are using is time-limited and has already expired. Cluster is shut down.	It is urgent to extend the license validity or replace it. The cluster doesn't work after the license expires. Alternatively, you can switch to the community version. In this case, you will lose some PRO functions. Contact support for further assistance.
Error	The PRO cluster has only one Master server, so no backup metadata server is available.	The cluster has only one Master server. Additional Followers should be used to maintain additional metadata records. A single Master server is a single point of failure in the cluster. If it fails, the entire cluster becomes unavailable.	Install additional Followers. Follower can automatically become the new Leader Master server if the previous one fails. Always avoid having just a single Master server. Contact MooseFS support if you need assistance.
Error	There are too many leading Master servers	There is more than one Leader (and/or Elect and/or Usurper) Master server in the cluster. This is a critical situation. The Leader Master server is the most important server in the cluster. It is responsible for managing metadata and ensuring the cluster's proper operation. Having more than one Leader Master server in the cluster may lead to data corruption, loss, or inconsistency. It is a serious error that should be immediately corrected.	If the Leader Master server is not available, it is necessary to pick a new Leader. If the Leader Master server is available, it is necessary to investigate why the second Leader (or Elect or Usurper) Master server appeared in the cluster. This is severe internal inconsistency that may lead to data loss. In case of manually picking a new Leader use the appropriate (usually the most recent) metadata records. Contact MooseFS support if necessary.
Warning	Implicit assignment to default location	The server is not explicitly assigned to any location by its IP address. Therefore, it is assigned to the default location implicitly. This is working but strongly discouraged configuration. Implicit assignment should be avoided. Note: as long as there are chunkservers in the cluster that are not explicitly assigned to any location, it will not be possible to switch the default location (see mfslocadmin setdefault) because it may cause problems with voting, implementation of the refund policy (sclass) and others.	We suggest always assigning servers to default location explicitly. Use the mfslocadmin map tool to explicitly assign servers to locations by defining appropriate IP address ranges for each location.
Warning	Location is set to I/O only mode	The location is currently in I/O only mode. The cluster is operational. Data is being send/received to servers assigned to this location and clients can connect to this location. However some features are disabled, namely: chunk replications, rebalance, and chunk deletions. This state is intended for maintenance purposes.	The location should be set to online as soon as possible. If the location is not intended to be used anymore, it should be removed from the cluster configuration. See the mfslocadmin tool for more info.
Warning	Location is set to offline	The location is currently set to offline. The cluster is operational but no data is being send or received to servers assigned to this location. Clients can't connect to this location. This state is intended for maintenance purposes, e.g., when the location is being prepared for removal, due to connectivity issues or when it is temporarily unavailable.	The location should be set to online as soon as possible. If the location is not intended to be used anymore, it should be removed from the cluster configuration. See the mfslocadmin tool for more info.
Warning	The amount of data in the cluster will soon exceed the available license. New data will not be able to be saved then.	The license you are using is limited in size. The total size of data stored in the cluster now exceeds 90% of the granted license. This means that the available space in the cluster may be finished soon. As a result, some cluster functionality will become unavailable. In particular, it will not be possible to add new data, and the cluster resiliency may be degraded. Note that license size refers to so-called raw data size, i.e., data after required duplications. It is usually larger than the user data size as most of the data is redundant.	It is urgent to extend the license size or replace it. After the stored data size hits the license limit, the cluster will not work as intended: no new data may be written. It is also possible to remove unnecessary data (snapshots included) or reduce the level of data redundancy. Additionally, data can be removed from the system trash, or the retention time for data in the trash can be shortened (see mfssettrashretention). Contact support for further assistance.
Warning	The license will expire soon.	The license you are using is time-limited and will expire in less than a month. After the license expires, the cluster will shut down.	It is urgent to extend the license validity or replace it. The cluster will not work after the license expires. Alternatively, you can switch to the community version. In this case, you will lose some PRO functions. Contact support for further assistance.
Info	No Leader Master server in the cluster, election in process.	There is currently no Leader Master server in the cluster. The process of electing a new leader is underway. One of the Follower Master servers is being elected and should become the Leader soon.	The process of electing a new leader is underway. It is started if the Leader is unavailable, e.g., after starting the cluster or after the failure of the previous Leader. There is no need for immediate action. The election process should soon end automatically, and the new Leader should be elected. However, if the election does not end within one minute, the reasons for this situation should be sought. If the election process started during regular cluster operations, why the old Leader became unavailable should be examined. It may turn out that it has permanently failed and should be repaired and connected back to the cluster to maintain the assumed level of resilience of the entire cluster. Electing a new leader is possible only when a quorum of Chunkservers is available. That is, half plus one Chunkservers are connected in the cluster. Contact MooseFS support if necessary.
Info	No Leader Master server in the cluster, leader switching (via usurper) in process.	There is currently no Leader Master server in the cluster. The process of switching to a new leader is underway. One of the Follower Master servers is designated by user for leadership (usurper) and should become the Leader soon.	The process of switching to a new leader is underway. It is started because the user has demanded the switch with mfssupervisor tool. There is no need for immediate action. The process should soon end automatically, and the new Leader should be functional. However, if the switching process does not end within one minute, the reasons for this situation should be sought. Electing a new leader is possible only when a quorum of Chunkservers is available. That is, half plus one Chunkservers are connected in the cluster. Contact MooseFS support if necessary.

Master Server

Severity	Message	What it means	Suggested fix
Error	The Follower Master server is unreachable.	The Follower Master server is unreachable, so the desired level of metadata protection is not fulfilled. All followers should be up and running in the cluster. It should be brought back online as soon as possible.	Check network configuration: ensure the Follower, the Leader, and management tools (GUI/CLI) are able to reach each other, and check firewalls. Check if the mfsmaster process is running on this server. Check log files on those servers. Contact MooseFS support if the problem persists.
Error	The Follower server version is older than the Leader Master server's.	The Follower should not be in an older version than the Leader, as this may result in protocol incompatibility leading to incorrect metadata being written. As a result, data in the cluster may become inconsistent or may be lost.	This situation must be immediately addressed by upgrading the Follower to at least the version of the Leader (or a newer one). Alternatively, it is possible to replace this Follower it with another (newer) one. The number of Followers depends on the desired level of cluster redundancy. It is recommended that all cluster components be at the same version. Contact MooseFS support if you need assistance.
Error	The follower's clock differs too much from the Leader Master server clock.	The Follower's clock differs by more than 2 seconds from the Leader Master server clock. This is a severe, intolerable difference. Ideally, there should be no difference. Incorrect server clock settings may cause severe metadata synchronization issues, degrading cluster resilience. Among others, auto-healing and data redundancy routines may not work correctly. All clocks in the cluster should be accurate and thus synchronized.	Check and adjust all clocks in the cluster. They should be accurate and thus synchronized. If possible, use an NTP server to synchronize them with a centralized time source. Contact MooseFS support if you need assistance.
Error	The Follower's exports checksum does not match the Leader's exports checksum.	Export checksums are one way to ensure exports configuration is the same across different Master servers. A mismatch indicates a serious error, and the cause of the mismatch should be pinpointed and eliminated.	Review the content of mfsexports.cfg file on all Master servers and make sure the entries in this file on all servers are identical. Contact MooseFS support if you need assistance.
Error	The Follower's metadata id doesn't match the Leader Master server metadata id.	The id of the metadata on this server doesn't match the Leader's metadata id. This is critical and should not happen. This indicates a serious error in the synchronization of metadata between the individual servers. It looks as if the Follower's metadata comes from another cluster, but the actual cause of this behavior should be immediately found and eliminated. Desynchronization of the servers will likely result in corruption or loss of metadata and/or data. Switching a Leader in such a situation will likely cause unexpected cluster behavior.	Check which id is the correct one. It is probably the one used by the Leader Master server. You may force downloading a completely new set of metadata by a Follower by stopping its mfsmaster process, removing metadata files stored on its hdd, and reloading fresh metadata from the Leader (see mfsmaster -e). Investigate why the ID is mismatched to avoid it in the future. However, this is a critical situation, and you should contact MooseFS support immediately.
Error	The Follower's metadata timestamp is delayed from the Leader.	The metadata timestamp differs from the Leader's by more than 6 seconds. This is a severe difference and should not be tolerated. It means that metadata is getting out-of-sync. Ideally, there should be no delay. Incorrect server clock settings may cause such delays. Or a general (Follower) server slowdown causing problems with handling requests in real-time by the metadata management process (mfsmaster). Out-of-sync metadata degrades cluster resilience and may lead to data loss or corruption, especially in case of other concurrent failures.	Check all server clock settings. They should be accurate and thus synchronized. If possible, use an NTP server to synchronize them with a centralized time source. Also, check the servers' performance and load, including RAM, swap, and I/O performance. If necessary, refer to the MooseFS Hardware Guide for hints on hardware requirements. Contact MooseFS support if the delay doesn't decrease immediately.
Error	The last saved metadata checksum doesn't match other Master servers' metadata checksums.	The checksum of the metadata written by this server is different from the checksum written by other metadata servers. This is critical and should not happen. This indicates a serious error in the synchronization of metadata between the individual servers. The cause of this behavior should be found and eliminated. Desynchronization of the servers will likely result in corruption or loss of metadata and/or data.	Review the server's log files, for both OS and MooseFS related messages. Contact support immediately.
Error	This Follower is in the PRO version, while the other Master is in the community edition (CE) version.	The PRO version of this server is installed, while the Leader (and perhaps other parts of the cluster) is installed in the community edition (CE).	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Error	This Master is in the community edition (CE) version, while the Leader is in the PRO version.	The community edition (CE) version of this server is installed, while the Leader (and perhaps other parts of the cluster) is installed in the PRO version.	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Warning	The follower's clock differs from the Leader Master server clock.	The Follower's clock differs by more than 1 second from the Leader Master server clock. Ideally, there should be no difference. Incorrect server clock settings may cause severe metadata synchronization issues, degrading cluster resilience. All clocks in the cluster should be accurate and thus synchronized.	Check and adjust all clocks in the cluster. They should be accurate and thus synchronized. If possible, use an NTP server to synchronize them with a centralized time source. Contact MooseFS support if you need assistance.
Warning	The Follower's metadata is not synchronized with the Leader Master server's metadata.	The Follower's metadata may not be synchronized with the Leader Master server's metadata for a while, especially right after the cluster restart or electing a new Leader. It should re-sync within a minute or so. Syncing Followers is a part of the MooseFS auto-rehealing routines. However, you should find a reason if it is not in sync for longer.	Check network configuration: ensure the Follower and the Leader are able to reach each other, and check firewalls. Check log files of both. Contact MooseFS support if the problem persists.
Warning	The Follower's metadata timestamp is slightly delayed from the Leader.	The metadata timestamp differs from the Leader's by more than 1 second. This is a sign that metadata servers' synchronization is not entirely correct. Ideally, there should be no delay. Incorrect server clock settings may cause such delays. Or a general (Follower) server slowdown causing problems with handling requests in real-time by the metadata management process (mfsmaster). Out-of-sync metadata degrades cluster resilience and may lead to data loss or corruption, especially in case of other concurrent failures.	Check all server clock settings. They should be accurate and thus synchronized. If possible, use an NTP server to synchronize them with a centralized time source. Also, check the servers' performance and load, including RAM, swap, and I/O performance. If necessary, refer to the MooseFS Hardware Guide for hints on hardware requirements. Contact MooseFS support if the delay doesn't decrease.
Warning	The last Master server metadata save took more than 45 minutes.	The last Master server metadata save took more than 45 minutes. Each Master server (leader or follower) 's metadata saving process helps keep a backup copy of cluster metadata. Basically, it is a metadata memory dump to a server's persistent storage. In case of server failure or restart, such a copy is read back to the server's memory and used for normal server operations. If the saving takes too long (an hour or longer), it may downgrade the cluster resiliency. Moreover, by default, the saving process on Followers is scheduled every hour, so it may be impossible to dump metadata fast enough between consequent requested dumps. Note that this period may be adjusted in the server configuration (see METADATA_SAVE_FREQ).	Saving time depends both on the metadata size (number of objects in the cluster and its additional attributes) and on the speed of internal server's hdd I/O. To shorten this time, use faster persistent storage to dump metadata, e.g., consider using NVMe or SSD. You may also consider limiting the size of the metadata by limiting global trash size, number of objects, number of additional attributes, or number of stored snapshots. Check if this time is not longer than METADATA_SAVE_FREQ config parameter. Contact MooseFS support if necessary.
Warning	This server dumped a consistent metadata file in the foreground.	This server dumped a consistent metadata file. However, it did it in the foreground instead of the background. This is not the ideal way for the Master server to work. The server probably did not have enough resources to make a persistent copy of the metadata to its hard drive in the background.	You should check the server configuration, load from other processes, RAM availability, and swap configuration. If you use Linux make sure that overcommit memory is turned on in your system (see if /etc/sysctl.conf contains this entry: vm.overcommit_memory=1). You can check the server logs and MooseFS logs on this server. If the situation does not repeat itself, there is probably no reason to worry, but you should check the status of this server more carefully for some time. Contact MooseFS support if necessary.
Warning	Unknown status of the last metadata save operation.	Unable to recognize the status code received from the server.	Check if the version of this GUI management console is up to date and compatible with this Master server version. Contact MooseFS support if necessary.
Info	CRC background save
Info	No CPU info available	Obtaining CPU usage is not supported by this OS or can't be obtained from the server.
Info	No memory info available	Obtaining memory usage is not supported by this OS or can't be obtained from the server.
Info	The copy of metadata on disk is older than on other Master servers.	This server has backed up its metadata to disk earlier than other servers. This is normal if this server is (or recently was) a Leader Master server. The Leader usually writes a copy of metadata to disk less often than the Followers. The difference in save times between Leader and Followers should never be longer than 24 hours. Its the Leader job to order Followers to save their metadata.	No action is required, but you should check whether the metadata was saved too long ago. In particular, earlier than the one set in the setup parameters. By default, the Leader's metadata is saved once a day at 12am UTC, and the Follower's metadata is saved every full hour. See the Master server configuration file parameters: METADATA_SAVE_FREQ, METADATA_SAVE_OFFSET, METADATA_CRCCHECK_FREQ, METADATA_DOWNLOAD_FREQ.
Info	The Follower server version is newer than the Leader Master server's.	The Follower is in a newer version than the Leader. This situation is permissible, especially during the cluster upgrade process. However, it is recommended that all cluster components eventually be at the same version. Generally, the difference in major version numbers between individual components should not be greater than 1. For example, components in version 4.x.x can work with components in version 3.x.x, but not with those in version 2.x.x. When mixing different versions of the cluster components, always pay attention to the guidelines concerning possible combinations of individual component types found in the documentation.	Upgrade all cluster components to the same version. If possible, use the latest available version.
Info	This server dumped a consistent metadata file in the background.	This is the desired behavior. The server had enough resources to make a persistent copy of the metadata to its hard drive in the background.
Info	This server lastly downloaded its copy of metadata from another Master server.

Metalogger

Severity	Message	What it means	Suggested fix
Error	The Metalogger version is older than the Leader (or Follower) Master server's.	The Metalogger is older than the Leader (or one of the Followers). This situation should not occur, as this may result in protocol incompatibility leading to incorrect metadata being logged. As a result, data in the cluster may become inconsistent or may be lost.	This situation must be immediately addressed by upgrading the Metalogger to at least the version of the newest Master server (Followers or Leader). Alternatively, it is possible to replace this Metalogger it with another (newer) one. The number of Metaloggers depends on the desired level of cluster redundancy. It is recommended that all cluster components be at the same version. Contact MooseFS support if you need assistance.
Error	This Metalogger is in the community edition (CE) version, while the Leader is in the PRO version.	The community edition (CE) version of this server is installed, while the Leader (and perhaps other parts of the cluster) is installed in the PRO version.	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Error	This Metalogger is in the PRO version, while the Master is in the community edition (CE) version.	The PRO version of this server is installed, while the Leader (and perhaps other parts of the cluster) is installed in the community edition (CE).	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Info	The Metalogger version is newer than the Leader Master server's.	The Metalogger is in a newer version than the Leader. This situation is permissible, especially during the cluster upgrade process. However, it is recommended that all cluster components eventually be at the same version. Generally, the difference in major version numbers between individual components should not be greater than 1. For example, components in version 4.x.x can work with components in version 3.x.x, but not with those in version 2.x.x. When mixing different versions of the cluster components, always pay attention to the guidelines concerning possible combinations of individual component types found in the documentation.	Upgrade all cluster components to the same version. If possible, use the latest available version.

Chunkserver

Severity	Message	What it means	Suggested fix
Error	The Chunkserver is unreachable	Can't get disk info because the Chunkserver is unreachable. It is either turned off, or there is a networking problem. If the server is undergoing planned maintenance, it should be put into maintenance mode.	Check network configuration: ensure the Master servers, management tools (GUI/CLI), and Chunkservers are able to reach each other. Check any firewalls in between. Check if the mfschunkserver process is running on the Chunkserver. Check log files on servers. All Chunkservers should be brought online as soon as possible. Contact MooseFS support if the problem persists.
Error	The Chunkserver is unreachable.	The Chunkserver is unreachable, so the desired level of data protection is not fulfilled. All Chunkservers should be up and running in the cluster, and this server should be brought back online as soon as possible. It is either turned off, or there is a networking problem. If too many Chunkservers are unreachable, the cluster may lose the Chunkserver quorum required to elect the Leader Master server. If the server is undergoing planned maintenance, it should be put into maintenance mode.	Check network configuration: ensure the Master servers, management tools (GUI/CLI), and Chunkservers are able to reach each other. Check any firewalls in between. Check if the mfschunkserver process is running on the Chunkserver. Check log files on servers. All Chunkservers should be brought online as soon as possible. Contact MooseFS support if the problem persists.
Error	The Chunkserver version is newer than the Leader Master server's or one of other cluster components.	The Chunkserver should not be in an newer version than the Leader (or any of cluster components), as this may result in protocol incompatibility leading to incorrect data being written. As a result, data in the cluster may become inconsistent or may be lost.	This situation must be immediately addressed by upgrading the Leader Master server or other cluster components. Chunkserver can not be downgraded. It is recommended that all cluster components be at the same version. Contact MooseFS support if you need assistance.
Error	The Chunkserver's disk is damaged.	The Chunkserver's disk is damaged. Disk damage usually requires recovering the copies of data stored on it from other disks in the cluster. Recovery process is automatic. Data should not be lost, but during recovery, the cluster's fault tolerance and performance may be reduced. In particular, another simultaneous failure could exceed the assumed number of concurrent failures and lead to data loss. The number of allowable simultaneous failures depends on the cluster configuration, particularly the storage class settings.	It is possible to attempt to repair the disk, but a large number of errors that result in the disk being marked as damaged usually indicate that the disk should be replaced. The number of allowed disk errors before marking disk as damaged is configurable.
Error	The Chunkserver's disk is invalid.	A disk is invalid if it is present in the Chunkserver configuration but for some reason a Chunkserver cannot write to or read from it. For example, its path may be misspelled, the disk may be unmounted, it may be physically damaged, it may have incorrect access rights, etc.	Check if the disk is properly mounted, accessible for writing and reading from the operating system, if there is no spelling error in the path, if the disk is not damaged. You can use any system disk checking tools.
Error	The Chunkserver's software version is too old	The Chunkserver's software version is too old to provide information on its disks. The Chunkserver should be updated to the latest version.	Update the Chunkserver to the latest version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Error	There is actually no space left on the disk	The disk is 99% full or completely full. Actually, no new data can be written to this disk. Lack of space on the Chunkserver may block the achievement of the set data replication goals. This can lead to a reduction in the cluster's fault tolerance.	It is necessary to add new disks to the server or to add new Chunkservers to the cluster. It is also possible to reduce the level of data redundancy. If using the community version, consider upgrading to the PRO version and implementing redundancy with erasure coding. Erasure coding significantly reduces storage space requirements compared to standard data replication (copying) while maintaining the same level of redundancy. Additionally, data can be removed from the system trash or the retention time for data in the trash can be shortened (see mfssettrashretention).
Error	There is actually no space left on the disks	All disks of the Chunkserver are full or 99% full. Actually, no new data can be written. Lack of space on the Chunkserver may block the achievement of the set data replication goals. This can lead to a reduction in the cluster's fault tolerance.	It is necessary to add new disks to the server or to add new Chunkservers to the cluster. It is also possible to reduce the level of data redundancy. If using the community version, consider upgrading to the PRO version and implementing redundancy with erasure coding. Erasure coding significantly reduces storage space requirements compared to standard data replication (copying) while maintaining the same level of redundancy. Additionally, data can be removed from the system trash or the retention time for data in the trash can be shortened (see mfssettrashretention).
Error	This Chunkserver is in the community edition (CE) version, while the Leader is in the PRO version.	The community edition (CE) version of this server is installed, while the Leader (and perhaps other parts of the cluster) is installed in the PRO version.	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Error	This Chunkserver is in the PRO version, while the Leader is in the community edition (CE) version.	The PRO version of this server is installed, while the Leader (and perhaps other parts of the cluster) is installed in the community edition (CE).	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Warning	'Marked for removal' but detailed status is unrecognized.	It is not clear if the disk is ready for removal or not. Check the cluster integrity and versions of the software components (GUI and Chunkserver). If the situation persists, contact MooseFS support.
Warning	'Marked for removal' but its status is unrecognized	It is not clear if the disk is ready for removal or not. Check the cluster integrity and versions of the software components (GUI and Chunkserver). If the situation persists, contact MooseFS support.
Warning	Chunkserver's location is offline.	The Chunkserver's location is offline (OFF), which makes this server not available. As a consequence the desired level of data protection is not fulfilled. All Chunkservers should be up and running in the cluster, and this server should be brought back online as soon as possible. If too many Chunkservers are unreachable, the cluster may lose the Chunkserver quorum required to elect the Leader Master server.	If there are no other errors (networking etc.) this server should be back online as soon as its location is turned online again. See mfslocadmin tool. All Chunkservers should be brought online as soon as possible. Contact MooseFS support if the problem persists.
Warning	Marked for removal, but NOT ready yet	There are disks 'Marked for removal' on this Chunkserver but they are NOT ready for removal yet. The cluster validates if they require any maintenance tasks (mainly copying data) before they can be safely removed from the Chunkserver's configuration, unmounted, and, subsequently, physically removed.	There is nothing you have to do now. Just wait until the disks are ready for removal. The cluster will secure their data automatically.
Warning	Marked for removal, but NOT ready yet	There are disks 'Marked for removal' on this Chunkserver but they are NOT ready for removal yet. They are being prepared for removal: Chunkservers are currently copying data among themselves to ensure those disks' data stays fully redundant after disk removal.	There is nothing you have to do now. Just wait until the disks are ready for removal. The cluster will secure their data automatically.
Warning	Marked for removal, but NOT ready yet	The disk is 'Marked for removal' but it is NOT ready for removal yet. The cluster validates if it requires any maintenance tasks (mainly copying data) before this disk can be safely removed from the Chunkserver's configuration, unmounted, and, subsequently, physically removed.	There is nothing you have to do now. Just wait until the disk is ready for removal. The cluster will secure its data automatically.
Warning	Marked for removal, but NOT ready yet	The disk is 'Marked for removal' but it is NOT ready for removal yet. It is being prepared for removal: Chunkservers are currently copying data among themselves to ensure this disk's data stays fully redundant after disk removal.	There is nothing you have to do now. Just wait until the disk is ready for removal. The cluster will secure its data automatically.
Warning	The Chunkserver is in maintenance mode	The Chunkserver is in maintenance mode while connected, meaning that the cluster avoids using it, however, the stored data can still be read or written. During this time, the cluster has reduced fault tolerance. The maintenance mode is designed to temporarily switch off the Chunkserver due to its scheduled service, such as upgrading software or replacing hardware. Cluster is not attempting to re-replicate data from the maintained Chunkserver because it assumes it will be back online soon with untouched data. Note, if you want to remove the Chunkserver's hdd during maintenance, mark it for removal first and wait until its data is replicated to other disks in the cluster (according to the desired redundancy level). Removing a disk without marking it for removal in the first place may lead to data loss.	Finish all maintenance work on this server. If necessary, restart it and its processes, bring it back online and switch off maintenance mode as soon as possible.
Warning	The Chunkserver is temporarily in maintenance mode	The Chunkserver is in "temporary maintenance mode", meaning that the cluster avoids using it, however, the stored data can still be read or written. During this time, the cluster has reduced fault tolerance. The maintenance mode is designed to temporarily switch off the Chunkserver due to its scheduled service, such as upgrading software or replacing hardware. Cluster is not attempting to re-replicate data from the maintained Chunkserver because it assumes it will be back online soon with untouched data. The "temporary maintenance mode" turns on automatically whenever the server is stopped gracefully (e.g., mfschunkserver stop or systemctl stop moosefs-pro-chunkserver.service). After restarting and reconnection, the server is automatically switched back to normal mode. If you don't reconnect the server within the specified timeout (default 30m), the Chunkserver will be considered unreachable, and replication routines in the cluster will start re-healing its data. See CS_TEMP_MAINTENANCE_MODE_TIMEOUT for timeout definition. Note, if you want to remove the Chunkserver's hdd during maintenance, mark it for removal first and wait until its data is replicated to other disks in the cluster (according to the desired redundancy level). Removing a disk without marking it for removal in the first place may lead to data loss.	Finish all maintenance work on this server. If necessary, restart it and its processes, bring it back online. The temporary maintenance mode will turn off automatically upon server reconnection.
Warning	The Chunkserver is unreachable but in maintenance mode	The Chunkserver is unreachable but in maintenance mode. During this time, the cluster has reduced fault tolerance. The maintenance mode is designed to temporarily switch off the Chunkserver due to its scheduled service, such as upgrading software or replacing hardware. Despite the Chunkserver being unreachable, the cluster is not attempting to re-replicate data from it. It is assumed this server will be back online soon with untouched data. Note, if you want to remove the Chunkserver's hdd during maintenance, mark it for removal first and wait until its data is replicated to other disks in the cluster (according to the desired redundancy level). Removing a disk without marking it for removal in the first place may lead to data loss.	Finish all maintenance work on this server. If necessary, restart it and its processes, bring it back online and switch off maintenance mode as soon as possible.
Warning	The Chunkserver version is older than the Leader Master server's.	The Chunkserver is in an older version than the Leader. This situation is permissible, especially during the cluster upgrade process. However, it is recommended that all cluster components eventually be at the same version. Generally, the difference in major version numbers between individual components should not be greater than 1. For example, components in version 4.x.x can work with components in version 3.x.x, but not with those in version 2.x.x. When mixing different versions of the cluster components, always pay attention to the guidelines concerning possible combinations of individual component types found in the documentation.	Upgrade all cluster components to the same version. If possible, use the latest available version.
Warning	The Chunkserver working queue is heavy loaded.	The Chunkserver working queue is heavy loaded. It receives more requests (e.g. read, write, replication) than it can handle without delay. Overloading may be temporary, but if it persists, the overall performance of the server should be checked: memory availability, load from other processes, the status of the hard disks, etc. An overloaded server can cause delays in client access to data and in the operation of the cluster's self-healing processes.	It is necessary to check the server's status, system software updates, and load from other processes. It should be verified whether the server is not too old or significantly slower compared to other Chunkservers. A common cause is the occurrence of errors on disks or disproportionately long I/O operation times on disks - which, by the way, is often a symptom of (approaching) physical damage to the disk. Additionally, it should be generally assessed whether the cluster load and redundancy definitions are appropriately adjusted to the servers' throughput.
Warning	The Chunkserver's disk is being scanned.	The Chunkserver's disk is being scanned. The scanning process is used to restore any missing internal Chunkserver's data structures (about its chunks). E.g., to restore the .chukdb file. The scanning process may take some time, depending on the disk's size and speed. The server is still operational during the scanning process. It may read and write user data but rebalancing, replication and similar health related processes are suspended.	There is nothing you have to do now. Just wait until the Chunkserver is done with scanning. However, it is worth to investigate why Chunkserver lost its internal (cached) data. Check what happened to the .chunkdb file: is it present and r/w accessible? It should not happen under normal circumstances, this is an extraordinary procedure. Scanning should take place only when Chunkservers is starting.
Warning	The disk is reporting read/write errors	The Chunkserver's disk is reporting read/write errors. E.g., chunk is inconsistent or missing. A large number of errors may result in the disk being automatically taken out of use (and marked as damaged). A sudden increase in errors may lead to data loss on that disk (undergoal or missing chunks). Taking the disk out of use without replacing it may reduce the cluster's fault tolerance.	It is necessary to check how frequently errors occur on this disk. The disk should be checked with an appropriate tool for assessing disk quality and replaced if necessary. The error notification can be dismissed by clicking the appropriate link in the Disks tab of the GUI management console.
Warning	There is very little available disk space remaining	There is very little space left on this Chunkserver. It may run out soon. Currently, some data redundancy and self-healing processes may be slowed down or at risk. Soon, after writing additional data or in the event of another simultaneous failure, the cluster's fault tolerance may decrease.	It is necessary to add new disks to the server or to add new Chunkservers to the cluster. It is also possible to reduce the level of data redundancy. If using the community version, consider upgrading to the PRO version and implementing redundancy with erasure coding. Erasure coding significantly reduces storage space requirements compared to standard data replication (copying) while maintaining the same level of redundancy. Additionally, data can be removed from the system trash or the retention time for data in the trash can be shortened (see mfssettrashretention).
Warning	There is very little available disk space remaining on the Chunkserver	There is very little space left on the hard disks. It may run out soon. Currently, some data redundancy and self-healing processes may be slowed down or at risk. Soon, after writing additional data or in the event of another simultaneous failure, the cluster's fault tolerance may decrease.	It is necessary to add new disks to the server or to add new chunkservers to the cluster. It is also possible to reduce the level of data redundancy. If using the community version, consider upgrading to the PRO version and implementing redundancy with erasure coding. Erasure coding significantly reduces storage space requirements compared to standard data replication (copying) while maintaining the same level of redundancy. Additionally, data can be removed from the system trash or the retention time for data in the trash can be shortened (see mfssettrashretention).
Info	Ready for removal	There are disks 'Marked for removal' on this Chunkserver and they are ready for removal. Their data is already safely stored elsewhere on the cluster. Those disks can be removed from the Chunkserver's configuration, unmounted, and, subsequently, physically removed.
Info	Ready for removal	The disk is 'Marked for removal' and it is ready for removal. Its data is already safely stored elsewhere on the cluster. The disk can be removed from the Chunkserver's configuration, unmounted, and, subsequently, physically removed.

Mount

Severity	Message	What it means	Suggested fix
Error	The Mount (client) version is newer than the Leader Master server's.	The Mount software (client side) should not be in an newer version than the Leader (or any of cluster components), as this may result in protocol incompatibility leading to incorrect data being read or written. As a result, data in the cluster may become inconsistent or may be lost.	This situation must be immediately addressed either by downgrading client-side mount software or by upgrading older cluster components. It is recommended that all cluster components and client-side software be at the same version. Contact MooseFS support if troubled.
Error	This client uses the community edition (CE) version MFS mount package, while the Leader is in the PRO version.	The community edition (CE) version of client's side MooseFS software is used, while the Leader (and perhaps other parts of the cluster) is installed in the PRO version. It may lead to unpredicted behavior (including data loss) as the client software doesn't implement a part of the MFS network protocol.	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster and client-side components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Warning	This client uses the PRO version MFS mount package, while the Master is in the community edition (CE) version.	The PRO version of client's side MooseFS software is installed, while the Leader (and perhaps other parts of the cluster) is installed in the community edition (CE). It should not lead to any unexpected behavior, however we insist on the cluster consistency.	Keep the cluster version consistent: all components (servers and client-side drivers) of a MooseFS cluster should be either PRO or community edition. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Info	The client (mount point) is inactive.	Inactive mount is a situation where a client was connecting to the cluster some time ago but for some reason lost the connection before ending it gracefully. This is not a dangerous situation from the cluster's point of view but such a client cannot use the cluster's resources. As a rule, clients should connect and disconnect from the cluster gracefully. If the client does not connect back to the cluster soon, this mount will be permanently abandoned.	From the cluster's perspective, there is no need to perform specific service actions. However, it is worth checking whether the client should not be connected and why the unexpected disconnection occurred. An inactive mount may indicate some network problems with client access to the cluster, e.g. packet loss, high latency, incorrect configuration, too restrictive firewalls, etc.
Info	The Mount (client) version is older than the Leader Master server's.	The Mount software (client side) is in an older version than the Leader. This situation is permissible, especially during the cluster upgrade process. However, it is recommended that all cluster components and its clients eventually be at the same version. Generally, the difference in major version numbers between individual components should not be greater than 1. For example, components in version 4.x.x can work with components in version 3.x.x, but not with those in version 2.x.x. When mixing different versions of the cluster components, always pay attention to the guidelines concerning possible combinations of individual component types found in the documentation.	Upgrade client-side mount software and all cluster components to the same version. If possible, use the latest available version.

Storage class

Severity	Message	What it means	Suggested fix
Error	State is not achievable	There are not enough Chunkservers available (or not enough Chunkservers have been assigned to the appropriate labels) to write data at the redundancy level indicated in this state. No data will be written to the desired redundancy level in this state.	Add (or assign to the appropriate labels) new Chunkservers or consider reducing the target redundancy level of this state.
Warning	State is not achievable	The cluster cannot store data in this state because the disks of the Chunkservers (designated to store data in this state) are full.	You may change designated Chunkservers (labels) or remove some data from the cluster. Do not leave this warning unaddressed as your data integrity may be at risk.
Warning	State is not fully achievable	The cluster can keep chunks in this state provided that Chunkservers that are in maintenance mode or overloaded are used. The cluster will not normally create new chunks or rebalance on such servers.
Warning	State is not fully achievable	There are enough Chunkservers available (>x+n) to hold existing data in the specified format EC x+n, but not enough to store new data in this format. Creating new EC x+n chunks requires at least x+2*n available Chunkservers.	Add (or assign to appropriate labels) new Chunkservers or consider reducing the target redundancy level of this state. Do not leave this warning unaddressed as the copies will not be converted to EC format, leading to faster space utilization on the cluster.
Warning	Unnecessary min trash retention time definition	This parameter is a number that tells what the minimum trash retention time a file must have so that the system converts its chunks to TRASH state after deleting this file. Introduced to avoid unnecessary changing of chunk states (and therefore physical copying of data) that are to be kept in trash for a very short time. If the TRASH state is not defined, this parameter has no effect.	If you want to use this parameter, you should explicitly define the TRASH storage class. If you do not want to use the min trash retention parameter, we suggest to set it to 0.
Info	Chunks requested in EC format are still stored as copies	It takes some time to calculate chunks' parity parts (EC) from copies - where they were kept in the KEEP state. Under normal circumstances, this number should be relatively low. Large numbers may appear if there is a lot of data waiting to be converted from copy to EC, e.g. after adding new data or after changing the storage class definition. Wait until the system automatically converts the copies to EC format.
Info	Chunks requested to be stored as copies are still in EC format	It takes some time to calculate chunks' copies from EC. It should not take place unless this class is 'reversible' or the storage class definition changed. Wait until the system automatically converts the EC to copy format.
Info	Redundancy level is lower than requested	The redundancy level of the chunks is lower than the one requested in the storage class (state) definition. It may be normal behavior if chunks just changed their state (e.g., KEEP->ARCHIVE) and the latter state has a higher redundancy level: it takes some time to create additional redundant data. However, this may be due to the lack of available Chunkservers or the lack of Chunkservers assigned to the appropriate labels. Note: deficiency may be undergoal, endangered, or missing chunks. Check the data health status.	If this is not due to the ordinary state transition: add (or assign to the appropriate labels) new Chunkservers or consider reducing the target redundancy level of this state. Do not leave this warning unaddressed as the data may be at risk.
Info	Redundancy level is lower than requested and chunks requested in EC format are still stored as copies	The redundancy level of the chunks is lower than the one requested in the storage class (state) definition. However, it may be normal behavior as the chunks requested in EC format are still stored as copies. This is because it takes some time to calculate chunks' parity parts (EC) from copies - where they were kept in the KEEP state. Under normal circumstances, this number should be relatively low. Large numbers may appear if there is a lot of data waiting to be converted from copy to EC, e.g. after adding new data or after changing the storage class definition. Wait until the system automatically converts the copies to EC format.	Wait for the system to calculate parity parts if it comes to simply moving copies to EC. However, we recommend checking the health of the data (see the Info tab).
Info	Redundancy level is lower than requested and chunks requested to be stored as copies are still in EC format	The redundancy level of the chunks is lower than the one requested in the storage class (state) definition. Usually, it should not take place unless this class is 'reversible'. The other reason may be: changing the storage class definition, possible Chunkserver failure or wrong label assignment.	Wait until the system automatically converts the EC to copy format. Regardless, we recommend checking the health of the data (see the Info tab).
Info	Storage class absent from this location	The storage class is not configured for use at this location. As a result, no objects (files, folders) assigned to this storage class will be stored on servers at this location. This is perfectly acceptable if you do not intend to use this storage class here.	Use the mfsscadmin create|modify|copy tool with -l option to assign storage class to a location.
Info	This parameter is not applicable here because the TRASH state is not defined for this storage class.
Info	Transition to TRASH state will be performed regardless of the file's trash retention time

Data health

Severity	Message	What it means	Suggested fix
Error	Some data chunks are endangered: there is no redundancy to them.	Endangered indicates that there is only one replica available. The data is still accessible, but it's at risk because it does not meet the redundancy goal. If another chunk copy (disk or server) fails, data loss could occur. The system is attempting to create additional copies (or data/parity parts) to meet the specified redundancy goal.	The data is at immediate risk. Wait until the system automatically restores the desired level of data redundancy. Monitor the cluster for any further simultaneous failures. Refrain from performing any other maintenance tasks. If possible, limit the use of the system.
Error	Some data chunks are missing: either no copies or not enough parts can be found in the cluster.	Missing refers to a situation where all copies of a chunk or too many chunk's data/parity parts are unavailable. This means that the data contained in that chunk cannot be accessed or recovered because there are no existing replicas to retrieve data from. This is a critical condition because it indicates permanent data loss for the affected chunks unless other backups or external recovery methods are available.	This is a serious situation. Try to recover the missing chunks, for example, by recovering damaged disks or servers from the cluster. Often, the chunks stored on the disks can be retrieved and reintroduced into the cluster. Additionally, you can locate the appropriate files in other places or use external backups. Tools such as mfsfileinfo and mfsfilerepair can help in partially recovering and/or supplementing the missing data. If possible, disconnect users from the cluster during the repair or at least limit write operations. Contact support for further assistance.
Error	Some files are missing due to missing chunks.	Internally files are composed of chunks, which must be available for the system to reconstruct the entire file. If one or more chunks are missing, the system cannot access or retrieve the necessary data to reconstruct the file, resulting in the file being missing. Consequently, any attempt to open a file that has missing chunks will lead to errors, preventing users or applications from accessing the intended data.	This is a serious situation. Try to recover the missing chunks, for example, by recovering damaged disks or servers from the cluster. Often, the chunks stored on the disks can be retrieved and reintroduced into the cluster to fill in the missing parts of the files. Additionally, you can locate the appropriate files in other places or use external backups. Tools such as mfsfileinfo and mfsfilerepair can help in partially recovering and/or supplementing the missing data in the file. If possible, disconnect users from the cluster during the repair or at least limit write operations. NOTE: The health-check loop is a long running operation that was started some time ago. It is possible that the data is healthy again since the loop was started. Check the current status of any missing chunks in the 'Chunk matrix table' above. Contact support for further assistance.
Error	Some trash files are missing due to missing chunks.	Internally files are composed of chunks, which must be available for the system to reconstruct the entire file. If one or more chunks are missing, the system cannot access or retrieve the necessary data to reconstruct the file, resulting in the file being missing. Consequently, any attempt to revert from the trash a file that has missing chunks will fail.	This is a serious situation. Try to recover the missing chunks, for example, by recovering damaged disks or servers from the cluster. Often, the chunks stored on the disks can be retrieved and reintroduced into the cluster to fill in the missing parts of the files. Additionally, you can locate the appropriate files in other places or use external backups. Tools such as mfsfileinfo and mfsfilerepair can help in partially recovering and/or supplementing the missing data in the file. If possible, disconnect users from the cluster during the repair or at least limit write operations. The situation may seem harmless, because there are missing files that are in the trash anyway and waiting to be finally deleted, but under normal circumstances this should not happen. It may be a symptom that there are some problems in the cluster. NOTE: The health-check loop is a long running operation that was started some time ago. It is possible that the data is healthy again since the loop was started. Check the current status of any missing chunks in the 'Chunk matrix table' above. Contact support for further assistance.
Warning	It is unknown whether all data is stored with the desired redundancy, as the status check is still ongoing.	It is unknown whether all data is stored with the desired redundancy, as the status check is still ongoing. Such a situation usually occurs after a Leader election or after a cluster restart. Please wait until the data check is complete.	Please wait until all Chunkservers reconnect and the data check is complete. Contact MooseFS support if the data check doesn't finish within a few minutes.
Warning	Some data chunks are undergoal: they are below the configured minimum redundancy level.	Undergoal means that the number of replicas is below the desired redundancy goal set by the system's configuration, but not necessarily at a critical level. The system is attempting to create additional copies (or data/parity parts) to meet the specified redundancy goal. The data is not immediately at risk, but efforts are being made to ensure the number of replicas is brought up to the desired level.	The data is not immediately at risk, wait until the system automatically restores the desired level of data redundancy. Monitor the cluster for any further simultaneous failures. Refrain from performing any other maintenance tasks.
Warning	Some files are below their expected redundancy level (undergoal)	Undergoal means that the number of replicas is below the desired redundancy goal set by the system's configuration, but not necessarily at a critical level. The system is attempting to create additional copies (or data/parity parts) to meet the specified redundancy goal. The data is not immediately at risk, but efforts are being made to ensure the number of replicas is brought up to the desired level.	The data is not immediately at risk, wait until the system automatically restores the desired level of data redundancy. Monitor the cluster for any further simultaneous failures. Refrain from performing any other maintenance tasks. NOTE: The health-check loop is a long running operation that was started some time ago. It is possible that the data is healthy again since the loop was started. Check the current status of any missing chunks in the 'Chunk matrix table' above.
Info	Missings possibly caused by unreachable Chunkservers.	Some chunks/files are missing but we also found unreachable Chunkservers. Substantial number of unreachable Chunkservers may lead to (temporal) reporting missing chunks/files. When the Chunkservers become available again, there is a chance data will be restored without permanent loss.	If possible, try to bring the unreachable Chunkservers back online. Check network configuration: ensure the Master servers, management tools (GUI/CLI), and Chunkservers are able to reach each other. Check any firewalls in between. Check if the mfschunkserver process is running on the Chunkserver. Check log files on servers. All Chunkservers should be brought online as soon as possible. Monitor the cluster for any further simultaneous failures. Refrain from performing any other maintenance tasks.
Info	Not all parts of the chunk are available	The chunk is stored using erasure coding, but not all parts are available. The chunk can't be reconstructed.	Locate valid chunks (data) wherever possible. There is a way to bring chunks back to the cluster and restore missing files.
Info	There are copies of chunk but they are in wrong version	The chunk has an incorrect version, but it is internally consistent and readable (unlike an 'invalid' chunk). The chunk may come from some older backup, hard disk, etc.	Locate valid chunks (data) wherever possible. There is a way to bring chunks back to the cluster and restore missing files.
Info	There are copies of chunk but they are invalid	Invalid means that the chunk is present in the cluster, but for some reason it is inconsistent or can't be read. The internal CRC or length does not match, or there is a read error on the disk.	Locate valid chunks (data) wherever possible. There is a way to bring chunks back to the cluster and restore missing files.
Info	There is no single copy of the chunk in the cluster	There is no single copy of this chunk on any disk in any of the connected Chunkservers. This is a critical condition because it indicates permanent data loss for the affected chunks unless other backups or external recovery methods are available. This can happen due to unexpected loss or disconnection of disks or entire Chunkservers.	Locate missing chunks (data) wherever possible. There is a way to bring chunks back to the cluster and restore missing files.

GUI and CLI

Severity	Message	What it means	Suggested fix
Error	GUI/CLI version older than some servers in a cluster	GUI servers should not be older than any server in the cluster because they may not support some of the packages exchanged by the servers, potentially leading to incorrect data being displayed.	If the GUI/CLI tools are too old, update them to a newer version, for example by installing the appropriate package. We recommend that all cluster components be in the same version. If possible, upgrade all cluster components to the same version using the latest available version. Contact MooseFS support if you need assistance.
Error	In the cluster managed by GUI/CLI tools in the community edition, there are servers running the PRO version.	Using community tools (GUI/CLI) to manage a Pro version cluster should be avoided because these tools do not support all the functionalities offered by the PRO cluster. As a result, data in the cluster may become inconsistent or may be lost.	GUI and CLI tools should be used in a version identical to the servers used in the cluster. Use the PRO GUI/CLI components to manage PRO cluster components. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version. Contact MooseFS support if you need assistance.
Info	GUI/CLI version newer than other servers in a cluster	GUI/CLI tools can be in newer versions than other servers in the cluster. However, the difference between different cluster components versions cannot be too large. Generally, GUI/CLI can support servers with a major version number at most one less than the major version number of the GUI/CLI. For example, GUI/CLI version 4.x.x can support servers in versions 3.x.x and 4.x.x. This, among other things, allows for upgrading the cluster to subsequent 'major' versions.	If the tools' version is too new, it is best to use a version that is as new as the newest server in the cluster. Usually, this will require using older installation packages. We recommend that all cluster components be in the same version. If possible, upgrade all cluster components to the same version using the latest available version.
Info	In the cluster managed by GUI/CLI tools in the PRO version, there are servers running the community edition.	Using PRO tools (GUI/CLI) to manage a community version cluster is possible; however, we advise using identical versions of all components in the cluster.	GUI and CLI tools should be used in a version identical to the servers used in the cluster. If possible, do not mix PRO and community components. Use appropriate install packages. Upgrade all cluster components to the same version. Use the latest available MooseFS software version.