Skip to main content

Disks and Chunkservers

This page describes the planned hardware tasks on a running cluster: taking a Chunkserver down for service and bringing it back, adding a Chunkserver, adding disks, replacing disks, retiring disks, and removing a Chunkserver permanently. Every procedure here starts from hardware that still works. A disk that has already failed is a repair task. Chunkserver and disk recovery describes that task.

Several Chunkservers at once count against the list​

The Leader Master Server PRO (PRO Edition only) holds its role only while more than half of the cluster's registered Chunkservers are connected to it. Stopping a Chunkserver does not take the server off that list. Maintenance mode does not take the server off the list either: the state means that the absence is planned, not that the server has left.

Batch maintenance has a limit because of this. The chunk matrix gives no warning of it. In a cluster of ten registered Chunkservers, four can be down at once. The Leader keeps its role. A fifth stopped Chunkserver leaves the Leader without its majority, so the instance accepts no metadata change until enough Chunkservers reconnect. Plan against the number of Chunkservers that the cluster has registered, including disconnected ones, not against the number currently running. A permanently retired Chunkserver leaves the count when it is removed from the cluster.

A controlled stop covers a short restart​

For a reboot or an operating system update that fits in half an hour, stop the process in a controlled way:

mfschunkserver stop

or stop the systemd unit that runs this command (see Starting and stopping). No other preparation is needed. A controlled stop finishes pending write operations and announces the disconnection to the Master Server. The Master Server then puts the server into temporary maintenance mode. The cluster waits for the server instead of rebuilding redundancy. CS_TEMP_MAINTENANCE_MODE_TIMEOUT in mfsmaster.cfg sets the length of the wait, 30 minutes by default. A server that reconnects within that time leaves the mode automatically. A server that does not reconnect is treated as failed. The Master Server then starts replicating what that server held. A process that is killed or loses power announces nothing and enters no maintenance mode.

Verification: on the Chunkservers tab of the GUI, the stopped server moves from the upper table to the Disconnected chunkserver details table below it. In the Maintenance column, the stopped server shows a countdown of the time left in temporary maintenance mode, in minutes and seconds. After reconnection, the server appears among the connected servers again, and its Maintenance column shows OFF. The same column in mfscli -SCS reads on (temp ; N seconds left) as long as the server is down, and off after the server reconnects. The GUI page describes what the two maintenance readings mean.

Longer work needs maintenance mode set by hand​

For hardware service, an operating system reinstallation, or moving a machine, enable maintenance mode by hand before stopping the process.

  1. On the Chunkservers tab of the GUI, flip the switch in the Maintenance column next to the server, or use the command line:

    mfscli -H mfsmaster -CM1/192.0.2.21/9422

    Here and in the commands below, substitute the Chunkserver's address and port for 192.0.2.21/9422.

  2. Stop the Chunkserver process and do the planned work.

  3. Start the process again and wait until the Chunkservers tab shows the server as connected.

  4. Switch maintenance mode off with the same GUI switch, or with -CM0 and the same address. Manual maintenance mode does not clear on reconnection. The mode stays on until you switch it off, unless CS_MAINTENANCE_MODE_TIMEOUT gives the mode an expiry (the default 0 means that the mode never expires on its own).

  5. Verify: on the Chunkservers tab, the server appears among the connected servers and its Maintenance column reads OFF. The same column reads off in mfscli -SCS.

  6. Before you service the next server, open the chunk matrix table on the Info tab and switch it to "All chunks". Wait until no undergoal or endangered chunks remain.

warning

While at least one Chunkserver is disconnected and every disconnected Chunkserver is in maintenance mode, the Master Server schedules no chunk repairs of its own. Nothing undergoal or endangered is fixed anywhere in the affected location PRO (PRO Edition only), or anywhere in the cluster when locations are not used. Recovery a client is waiting for still runs, so reads and writes are served throughout. Keep the window short and switch the mode off as soon as the work ends. The maintenance mode page describes what exactly is suspended and why the suspension covers this much.

A new Chunkserver fills up without further orders​

  1. Install and configure the Chunkserver on the new machine (Install server components). List its disks in mfshdd.cfg. Do not give it a version newer than the Master Server's.
  2. Start the process.
  3. Verify: the server appears as connected on the Chunkservers tab of the GUI, or in mfscli -SCS, with the expected total space.

Nothing else is required. The Master Server delays repairs while the new server's chunk lists are still arriving (replication priorities). The Master Server then levels space usage across servers on its own (rebalancing). The rebalancing page covers how fast the leveling runs and which limits shorten it at the cost of client throughput.

Add disks to every Chunkserver at once​

When you expand more than one Chunkserver, add the new disks to all of the servers in parallel rather than one server at a time. If you finish one server first, the cluster levels data onto the expanded server. The cluster then moves the same data off again as the remaining servers grow. Adding the disks everywhere at once saves both moves. You do not need to wait for one server's internal rebalance to finish before you add disks to the next server.

On each server:

  1. Mount the new disk and append its path to mfshdd.cfg.

  2. In mfschunkserver.cfg, raise the internal rebalance pace for the duration of the fill: set HDD_REBALANCE_UTILIZATION = 100. If the projected time is still too long, also add HDD_HIGH_SPEED_REBALANCE_LIMIT = 2. The rebalancing page describes what each option changes.

  3. Reload the process:

    mfschunkserver reload

    A reload is enough: the Chunkserver re-reads mfshdd.cfg and both rebalance options without a restart.

  4. Verify: the new paths appear on the Disks tab of the GUI, or in mfscli -SHD. Each new path is first scanning and then in service. From then on, space usage across the server's disks converges.

  5. After the disks have evened out, set HDD_HIGH_SPEED_REBALANCE_LIMIT back to 0 and reload again. Fast rebalance replaces the slow rebalance. As long as fast rebalance is on, it competes with the cluster's replication work.

Do not expect the new disks to fill from incoming writes. New chunks are assigned in proportion to capacity rather than free space, so the old disks keep receiving their share. The internal rebalance does the leveling.

Mark a disk for removal to retire it​

To retire a working disk, mark it for removal with the * prefix on its line in mfshdd.cfg.

  1. Prefix the disk's path and reload the Chunkserver process:

    */mnt/chunkdisk03

    From that moment, the copies on this disk stop counting toward redundancy. The cluster replicates the affected chunks and prefers their copies on other disks and servers as sources. The disk keeps serving reads the whole time. Nothing is deleted from the disk: marking makes the cluster independent of the disk without emptying it. The disk error handling and disk removal page describes the mechanism, including the one narrow case in which a marked copy is deleted.

  2. Track progress on the Disks tab of the GUI or in mfscli -SHD. The disk's status changes from removal not ready (validating) through removal not ready (in progress) to removal ready. Readiness is computed per Chunkserver, so all disks marked on one server become ready together. The status changes after the replication rather than with it. The status waits for a full pass over the chunk table in which nothing depended on the disk. The pass during which the last copy was made does not qualify (a disk is ready when nothing depends on it). Expect the status to keep showing in progress for a while after the matrix in the next step is already clean.

  3. Cross-check the chunk matrix table, switched to "Regular chunks only" (mfscli -SIC -m 1). This view counts the cluster as if every marked disk were already removed. When the table shows no missing, endangered, or undergoal chunks, you can pull the marked disks without losing redundancy.

  4. Delete or comment out the disk's line, reload the process, unmount the disk, and remove the hardware.

  5. Verify: the path no longer appears on the Disks tab or in mfscli -SHD. The chunk matrix table switched to "Regular chunks only" stays clean.

Marking is reversible at any point. Remove the * prefix and reload, and the disk is a regular disk again. The replication off a marked disk is ordinary repair work. This replication pauses under the same maintenance-mode hold as every other repair. Its pace follows the replication limits described under replication priorities.

The other line prefixes of mfshdd.cfg are not retirement tools. The prefixes < and > control the internal rebalance between the server's own disks. The prefix ~ relaxes one damage check. The mfshdd.cfg reference lists all of them.

Replace a working disk: retire, then add​

  1. Mark the old disk for removal and wait for removal ready and a clean "Regular chunks only" matrix. The steps are in Mark a disk for removal to retire it.
  2. Remove the disk's line, reload, and swap the hardware.
  3. Add the new disk's path to mfshdd.cfg and reload. The internal rebalance fills it.
  4. Verify: the new path appears on the Disks tab of the GUI, or in mfscli -SHD. The path is first scanning and then in service. Its chunk count grows from then on.

This procedure is for a disk that still works. For a disk that no longer mounts, or that the Chunkserver has already dropped as damaged, skip the marking entirely. Chunkserver and disk recovery covers that case.

Empty a Chunkserver's disks before removing it​

Maintenance mode has no part in a permanent removal. The mode is for an absence that ends. Removal starts by making the cluster independent of the server's data.

  1. Mark all of the server's disks for removal at once. The replication works on all of the disks in parallel and takes its sources from other servers where possible. Marking everything at once is therefore not slower than going disk by disk, and can be faster.

  2. Wait until the server's disks report removal ready and the chunk matrix table switched to "Regular chunks only" shows no missing, endangered, or undergoal chunks.

  3. Stop the Chunkserver process.

  4. On the Chunkservers tab of the GUI, use the remove link next to the now-disconnected server, or the command line:

    mfscli -H mfsmaster -CRC/192.0.2.21/9422

    The Master Server accepts the removal only for a disconnected server. The server's maintenance state does not block the removal. The removal clears the state, including the temporary maintenance mode that a controlled stop creates, so repairs elsewhere are not delayed.

  5. Verify: the server no longer appears on the Chunkservers tab. The matrix stays clean.

A disconnected server that nobody removes leaves the list automatically after CS_DAYS_TO_REMOVE_UNUSED days, 7 by default. At 0 the entry never expires and waits for a manual removal. In MooseFS PRO PRO (PRO Edition only), remove retired servers promptly rather than waiting for that timeout. A disconnected server still on the list keeps counting toward the majority that leader election requires. Decommission the machine after every Chunkserver process on it is stopped and removed from the list.