Rebalancing

A disk added to a Chunkserver and a Chunkserver added to the cluster both start empty. Incoming writes alone do not fill either of them. New chunks are assigned in proportion to capacity rather than to free space. The new disk or server therefore receives only its share of new chunks. The disks and servers that are already full keep their own share. Allocation and space selection describes that choice. To even out the usage, MooseFS has to move chunks that already exist.

MooseFS moves existing chunks with two mechanisms. One mechanism runs inside a single Chunkserver, between its own disks. The other mechanism runs across the cluster, between Chunkservers. The Master Server controls that mechanism. The two mechanisms measure different things and are paced by different settings. The two mechanisms run independently of each other.

Both mechanisms move a copy that is already there. Replication restores a copy that is missing. Replication is a different job, with its own order and its own limits, described in replication priorities.

Chunkservers balance disks, the Master Server balances servers

Inside one Chunkserver, a rebalance is a file copy between two of its disks. The Chunkserver picks a chunk at random on the fuller disk and writes the chunk to the emptier disk. The Chunkserver unlinks the original once the new file is complete and named. The chunk itself stays the same: same identifier, same version, same copy count in the cluster, same used space on the server that holds it. The Master Server is never informed, because from the Master Server's side nothing has changed.

Between Chunkservers, the Master Server cannot move anything, because the only orders it has are "fetch this chunk" and "delete this chunk". A rebalance between Chunkservers uses both orders, in sequence. First, the Master Server orders an emptier Chunkserver to fetch the chunk from a fuller Chunkserver. The chunk then has one copy more than its goal requires. Next, the chunk goes into the queue for surplus copies. The Master Server removes the surplus copy moments later. Where the class leaves the choice open, the Master Server removes the surplus copy from the server that has the highest space usage among the servers that hold a copy. Together, the two orders move the chunk. The move costs one full chunk transfer plus one deletion.

A disk joins at one point, leaves at half

The Chunkserver compares each of its disks with the average space usage of the disks that it counts. The Chunkserver counts only the disks that hold more than 100 chunks. A disk more than one percentage point below that average becomes a destination. A disk more than one percentage point above the average becomes a source. A single pair, one of each, is chosen for each move. A new, empty disk is far below the average. The disk is also too empty to be counted into that average, so it is a destination from the start. When every counted disk is on one side of the average, the round still runs. When no disk is below the threshold, any disk no more than a point above the average can be a destination. The mirror rule applies to sources, so a single outlying disk still gets a pair.

Once moving begins, the threshold halves. The Chunkserver continues until no disk is further than half a percentage point from the average. The gap between the two thresholds keeps the Chunkserver from oscillating around the average. The gap also makes the balance that the Chunkserver reaches tighter than the imbalance that started the work. When the Chunkserver stops, it waits a minute before it checks again.

Not every disk is eligible. A disk takes no part in a round when it has been marked damaged, when it is being emptied, or when it has not finished its startup scan. In a round driven by the usage comparison, a disk smaller than a gigabyte also takes no part. A disk that still holds duplicates to remove cannot be a destination, but can still be a source. A disk at 99 percent or more is never a destination. Emptying a disk that is being removed from the server is a different mechanism. Disk errors and removal covers that mechanism. The two mechanisms overlap: prefixing a disk's path in mfshdd.cfg with < or > forces that disk to be a source or a destination. The internal rebalance then moves the chunks. A forced direction overrides the usage comparison entirely.

The default rebalance moves one chunk, then sleeps

One thread does the work. The thread copies a chunk and measures how long the copy took. The thread then sleeps long enough that copying takes up the share of total time that HDD_REBALANCE_UTILIZATION sets, in percent. At the default, the thread is active a fifth of the time and idle the other four fifths. Raising the value to 100 removes the sleep. The thread then copies chunks continuously, but still one chunk at a time. The thread also runs at low I/O priority, so the operating system gives priority to everything else that the server is doing.

That is all the pacing. At the default, the thread copies far less than the disks could handle. A copy between two disks of the same server uses both disks and no network. The copy therefore competes with client reads and writes for the one resource that is scarce: the disks.

Fast rebalance replaces the slow thread

The second implementation has two names. The setting, the release notes, and the status that the Chunkserver sends to the Master Server call it the high speed rebalance. The GUI and mfscli label the same state Fast rebalance. This page uses the label that the screens use. Both names mean the mode described below.

Setting HDD_HIGH_SPEED_REBALANCE_LIMIT to anything above 0 switches the Chunkserver to the fast mode and stops the slow rebalance. The mode is off until you turn it on. Turning the fast mode on does not speed up the slow thread. The slow thread stays idle for as long as the setting is above zero.

The fast implementation starts and stops on the same thresholds and skips the same disks. Unlike the slow thread, the fast implementation does not sleep and does not do the copying itself. The fast implementation passes each move to the pool of worker threads that already serves the replications ordered by the Master Server. The Chunkserver keeps passing moves to that pool until every destination disk has the configured number of moves running at the same time. The setting counts simultaneous moves per disk. A move uses its source disk and its destination disk alike.

The fast moves share a queue with the cluster's work. They make that queue longer and appear in the load that the Chunkserver reports. The fast mode has one further effect, on chunk placement rather than on rebalancing. For as long as the mode is on, the server drops out of the pool that the cluster picks new chunks from. Allocation and space selection describes that rule in full.

The Master Server sees which mode a Chunkserver runs

A Chunkserver that is rebalancing states this in its load report. The two modes report different values. The slow mode reports a state that the interfaces show as Rebalance. A server in the slow mode keeps all its roles. The server still receives replications and still serves as a replication source. It is also still a candidate to receive a rebalance from a fuller server. Only the placement of new chunks holds such a server back. The page on allocation and space selection describes this rule for a server that is busy rather than dropped. The fast mode reports a state shown as Fast rebalance. In that mode, the Master Server treats the server like an overloaded server. Ordinary repairs and rebalances skip the server. Only chunks that a client is waiting for and chunks with no redundancy left may still be replicated onto it. The page on replication priorities describes that rule. New chunks avoid the server entirely. The chunks on the server are still read, so the server still serves as a replication source. The two states raise no alert. A rebalance between Chunkservers has no state at all, because it is the Master Server's own work and not a state of any server.

Between Chunkservers, space usage decides at one point

For each chunk that the Master Server examines, the Master Server compares the space usage of the servers that hold a copy with the space usage of every server that could accept a copy. Those candidates are ordered from the emptiest upward. The Master Server looks for the widest gap. ACCEPTABLE_PERCENTAGE_DIFFERENCE does not stop a rebalance. It sets the gap, in percentage points of space usage, below which the rebalance slows down to the pace described further on.

A destination has to be a server that does not already hold a copy of that chunk and that satisfies the chunk's Storage Class. The server also must not be excluded by any rule that keeps copies apart. A rebalance therefore never reduces redundancy to improve evenness. The comparison also stays inside one location PRO (PRO Edition only). Adding a server in one location therefore does not move data out of another location.

Rebalance between Chunkservers is the last work considered

Of all the work that the Master Server can order for a chunk, a rebalance is the last thing it considers. A rebalance is skipped entirely while any priority queue holds a chunk that needs replication. The extra examinations that the Master Server makes outside its regular loop never rebalance at all. A Storage Class is also skipped, for milliseconds at most, when its last five rebalance attempts in a location were all blocked by the replication limits. The exception to the queue rule is a cluster where some server has one percent or less of its space free. In such a cluster, the rebalance runs regardless of what else is pending, because at that point moving data is the more urgent job.

The rebalances that remain are throttled in two stages. Rebalance copies get their own concurrency limits inside CHUNKS_WRITE_REP_LIMIT and CHUNKS_READ_REP_LIMIT. The limits form two sets. One set applies to a pair of servers whose usage differs by more than one and a half times the acceptable difference. The other set applies to a pair that has come closer than that. A new Chunkserver falls in the first set for as long as the new Chunkserver is clearly emptier than the rest. The number of transfers that the new Chunkserver can accept at once therefore limits how fast it fills.

Once the difference has narrowed to the acceptable difference or below, a second throttle takes over, cluster-wide rather than per server. The Master Server starts at most one more rebalance copy per interval. The interval grows longer when the gap gets smaller. At one percentage point apart, the rate is one copy per second. At a tenth of a point, the rate is one copy every ten seconds. The last stage is therefore slow by design. It is slow in proportion to how little is left to fix.

None of these mechanisms sets a completion time. The time that a new disk or a new Chunkserver needs to reach the usage of its peers depends on how much data has to move. The time also depends on how many chunk transfers the limits allow at once. Finally, the time depends on how much of the worker pool the client load leaves free, and on how often the priority queues hold the rebalance off. Raising the limits shortens that time, but takes throughput away from clients while the rebalance runs. The defaults are set to avoid that trade-off.

Five settings pace a rebalance

Setting	Default	What it decides here
HDD_REBALANCE_UTILIZATION	20	The percentage of its time that a Chunkserver's rebalance thread spends copying. 0 switches the slow rebalance off. A value above 100 is read as 100.
HDD_HIGH_SPEED_REBALANCE_LIMIT	0	Simultaneous moves per disk in the fast mode, and the switch that turns that mode on. The value caps at 10.
ACCEPTABLE_PERCENTAGE_DIFFERENCE	1.0	The gap in space usage between two Chunkservers, in percentage points, below which a rebalance slows down. A rebalance acts on any gap, however small. A value outside 0.1 to 10.0 is pulled back into that range.
CHUNKS_WRITE_REP_LIMIT	2,1,1,4,4	Transfers at once onto one Chunkserver. The third and the fourth number apply to rebalancing.
CHUNKS_READ_REP_LIMIT	10,5,2,5,10	Transfers at once out of one Chunkserver, in the same five groups.

A Chunkserver balances its own disks against fixed thresholds that have no setting: one percentage point from the average to start a move, and half a point to stop. A disk needs a hundred chunks before it counts toward the average. The Chunkserver waits a minute before the next round.