Starting and stopping the cluster

This page gives the order for starting and stopping a whole cluster. It also describes how to stop a single process without starting a rebuild. Use it at first start, for a planned outage, and when one machine needs a restart.

The commands come in two equivalent forms. The systemd units installed by the packages wrap the actions of the binaries: systemctl start moosefs-pro-master runs mfsmaster start, and systemctl stop moosefs-pro-master runs mfsmaster stop. On a system without systemd, call the binaries directly. The actions are listed in mfsmaster(8), mfschunkserver(8), and mfsmetalogger(8). The binaries are listed in the command reference. The unit names below are the PRO ones. Community Edition packages name the same units without the pro segment: moosefs-master, moosefs-chunkserver, moosefs-metalogger.

Start the Master Server first, the clients last

Start the layers in the order in which they depend on each other. Check each one before you start the next.

Start the Master Server (in PRO PRO (PRO Edition only), every Master Server):
```
systemctl start moosefs-pro-master
```
Every other component reaches the cluster through the Master Server, so nothing works before the Master Server is up. Verify on the master machine:
```
mfsmaster test
```
The command prints mfsmaster pid: PID for a running process.
Start every Chunkserver:
```
systemctl start moosefs-pro-chunkserver
```
Files become readable only when the Chunkservers that hold their copies are connected, because clients exchange data with Chunkservers directly. A Chunkserver that starts while the Master Server is still unreachable does no harm: it retries the connection every 5 seconds (MASTER_RECONNECTION_DELAY in mfschunkserver.cfg). Run mfscli from any machine to verify that all of them have registered:
```
mfscli -H mfsmaster.example.com -SCS
```
Start the Metalogger:
```
systemctl start moosefs-pro-metalogger
```
The Metalogger connects on the same retry schedule and resumes its copy of the metadata stream. mfscli -SML lists the connected Metaloggers.
Mount the clients:
```
mfsmount -H mfsmaster.example.com /mnt/mfs
```
A mount attempted while the Master Server is unreachable fails, unless the mount uses -o mfsdelayedinit (mfsmount(8)). How long the mount takes to fail depends on how the connection fails. A stopped Master Server refuses the connection. A refused connection ends the attempt at once. Packets that are dropped without an answer usually mean that a firewall is in the way. Such packets put the mount through its retry loop instead: one attempt and up to ten retries, 2 seconds each, so up to about twenty-two seconds. A mount made before the Chunkservers are up exposes files whose data is not yet reachable. Mount options and the checks for a new mount are in first mount and verification.

The Leader returns by election, not start order

In a PRO instance PRO (PRO Edition only), no start command chooses the Leader. Start all Master Servers the same way. Each Master Server starts as a Follower that has not synchronized its metadata yet. This is the expected first reading on every one of them, not a fault (a Master Server starts as a Follower). The two tools report this state in different words. mfscli -SIM prints FOLLOWER (DESYNC). mfssupervisor prints not synchronized for the same moment. The election needs the majority of the Chunkservers that the Master Server already has in its metadata. So on a restart, the Leader appears while the Chunkservers register, not before. On the very first start, no Chunkserver is registered yet. An empty list meets the majority. A Leader therefore appears within seconds after the Master Servers are up. The vote goes to the candidate with the highest metadata version. After a clean shutdown, this candidate is the server that was stopped last (how the candidate is picked).

Watch the states with mfssupervisor. Without an action option, the tool prints the current state of every Master Server that the master name resolves to (mfssupervisor(8)):

mfssupervisor -H mfsmaster.example.com

Stop the cluster from the clients inward

Unmount every client:
```
umount /mnt/mfs
```
An application that writes into a mount outlives every server that you stop while the mount is in place. So unmount first, on every machine that mounts the cluster.

In the reverse order, the mount has nothing to answer with. Every operation on the mount waits for the Master Server. A process that reads the mount goes into uninterruptible sleep, where a termination signal does not reach the process. A plain umount waits as well and prints nothing. The steps to release such a mount point, and the false trail on the way, are in the mount hangs and umount says nothing.
Stop every Chunkserver while the Master Server still runs:
```
systemctl stop moosefs-pro-chunkserver
```
Each Chunkserver announces its departure to the Master Server. The Master Server then suspends rebuilds instead of starting them. The state that does this is maintenance mode.
Stop the Master Server. In PRO PRO (PRO Edition only), stop the Followers first and the Master Server that was the Leader last. That Master Server holds the highest metadata version. The next election promotes the highest version that it finds, so the cluster resumes in the state in which it stopped. Read the roles before step 2, not after it. A Master Server without the majority of its Chunkservers steps down. So once every Chunkserver is stopped, no Master Server reports LEADER any more.
```
systemctl stop moosefs-pro-master
```
mfsmaster stop sends the termination signal and waits for the process to exit. The Master Server writes the metadata image to disk in the foreground before it exits, so on a large instance the command takes minutes. The command waits up to 30 minutes, the default of the -t option of mfsmaster(8). The shipped unit gives the stop the same 30 minutes.

warning
Do not kill the process while the stop waits. The clean image metadata.mfs exists only after a completed stop. Without that image the next mfsmaster start refuses to run. A stop that makes no progress is a save that keeps failing: the Master Server retries it every 10 seconds and says so in the log, while mfsmaster stop gives up on its own after the 30 minutes without ending the process.
Stop the Metalogger:
```
systemctl stop moosefs-pro-metalogger
```
The Metalogger backs up the metadata stream for as long as a Master Server runs, so stop the Metalogger last.

To verify a full stop, run the test action of each binary on its machine. mfsmaster test prints mfsmaster is not running. The actions of the Chunkserver and the Metalogger answer the same way.

Both orders as a checklist

The same steps, condensed into a list to tick off. Each step is described above.

Start:

Master Servers started. mfsmaster test prints a PID on each machine.
PRO PRO (PRO Edition only): mfssupervisor shows a Leader: on a restart once the Chunkservers register, and on a first start even before they register.
Chunkservers started. mfscli -SCS lists every one of them.
Metalogger started. mfscli -SML lists it.
Clients mounted. A file can be written and read back.

Stop:

Every client unmounted.
Chunkservers stopped while the Master Server still runs.
PRO PRO (PRO Edition only): roles read before the Chunkservers were stopped, then Followers stopped first and the Master Server that was the Leader last.
Master Server stop ran to completion. The process was not killed during the metadata write.
Metalogger stopped.
The test action of each binary answers "is not running".

A controlled Chunkserver stop starts no rebuild

When one Chunkserver machine needs a restart, stop the process in a controlled way. The cluster then waits for the Chunkserver instead of rebuilding redundancy. The state in which the cluster waits is temporary maintenance mode.

Stop the process:
```
systemctl stop moosefs-pro-chunkserver
```
The Chunkserver finishes the writes that it has accepted and announces the departure to the Master Server. The Master Server puts the Chunkserver into temporary maintenance mode. While the server is disconnected, the GUI raises the warning "The Chunkserver is temporarily in maintenance mode".
Do the work and start the process again. When the Chunkserver reconnects, the mode clears by itself. The time window is CS_TEMP_MAINTENANCE_MODE_TIMEOUT in mfsmaster.cfg, 30 minutes by default. A server that stays disconnected longer is treated as failed. Rebuilds then start. For work that takes longer than the time window, such as a disk swap or hardware service, use manual maintenance mode and the procedures of disks and Chunkservers.

A killed process, a power cut, or a lost network connection announces nothing, so the Master Server treats such an absence as a failure.

After a power failure the Master Server waits

Chunkserver and Metalogger units that are enabled start again with their machines and reconnect on their own. A client mount set up with -o mfsdelayedinit mounts even before the network is up and connects in the background (mfsmount(8)). Mount setup at boot is part of installing clients.

The Master Server is the exception. The exception is deliberate. A power failure is not a clean stop, so the clean image metadata.mfs is missing. A plain mfsmaster start refuses to run and points at the -a option. For the Master Server, stopping at an error is better than starting on damaged metadata and letting clients modify the file system on top of that metadata. An enabled master unit does not change this. The unit runs the same plain start at boot. The attempt fails the same way, every time. Starting the Master Server automatically at boot is therefore not recommended.

Do not respond to the refusal by putting -a into the unit file or into any start automation. The flag is a recovery tool, not a start flag. It picks the best metadata that the Master Server finds in its own data directory and replays changelogs on top of that metadata. The instance can hold a better copy elsewhere. After a power failure that disabled a whole rack, the most recent usable copy can be the copy on the Metalogger, in another room. An automatic -a start would ignore that copy and roll recent changes back. Needing -a at all means that the Master Server was not stopped correctly. A person has to inspect that state. Check the metadata and the cause of the failure before you start the Master Server. Do not force a start from whatever the disk holds.

In PRO PRO (PRO Edition only), the additional Master Servers exist for this case. When one Master Server stops uncleanly, the remaining Master Servers keep the cluster running. There is time to examine the failed server before you start it again. In CE, the same refusal is a trade-off: the cluster waits for its one Master Server. In exchange, an automatic start cannot make the metadata irreparable.

Start the Master Server first, the clients last​

The Leader returns by election, not start order​

Stop the cluster from the clients inward​

Both orders as a checklist​

A controlled Chunkserver stop starts no rebuild​

After a power failure the Master Server waits​