Replication priorities
A cluster that has lost a disk or a Chunkserver has more chunks to repair than it can repair at once. For that backlog, the Master Server decides two things: which chunk to repair first, and how much repair traffic to allow while the remaining chunks wait. This page describes both.
Queues hold the repair work, top first
The Master Server compares what the Chunkservers report holding with what the chunk's Storage Class requires. As soon as the reported copies no longer match the requirement, the Master Server files the chunk in a queue. The queue that the chunk goes into depends on what is at stake.
| Order | What puts a chunk in this queue |
|---|---|
| 1 | A client is waiting for the chunk: a read needs a missing part rebuilt, or a write needs the parts assembled into a whole copy. |
| 2 | endangered, or recoverable only from leftover parts, on a class asking for Redundancy Level 2 or higher. |
| 3 | The same, on a class asking for Redundancy Level 1. |
| 4 | At most one copy on a regular disk, with anything beyond it only on disks marked for removal. |
| 5 | Below the class on regular disks, with the difference sitting on disks marked for removal. |
| 6 | A conversion left half done: whole copies and Erasure Coding parts of one chunk at the same time, or parts in both widths. |
| 7 | undergoal for any other reason, stored in a format the class does not ask for, or missing a data part while parts share a Chunkserver. |
| 8 | More copies or parts than the class asks for, one part stored twice, or parts sharing a Chunkserver with the data still complete. |
| 9 | Copies on Chunkservers that do not match the class labels, where a better placement is available. |
- Queues 6 and 8 hold back the queues below them only after they themselves grow past a thousand chunks, so a trickle does not stop the rest.
Queue 1 keeps running even while Chunkserver reports are settling, when every other repair pass is skipped.
Combined capacity: PRIORITY_QUEUES_LENGTH. A full set displaces from less urgent queues. The chunk loop finds stragglers anyway.
The Master Server works through the order from the top. The Master Server processes the queues many times a second, on a timer much faster than the chunk loop. Each of those rounds goes through the queues in order and spends its allowance on the higher queues first. The queues always have priority over the loop that finds new work. The loop can reach a chunk that belongs in one of the lower queues while a higher queue is not empty. In that case, the loop files the chunk and does nothing else with it. A chunk one failure away from being lost is therefore never held up behind a chunk that only has a copy too many.
Two of the queues are exempt from that blocking until they grow. The queue for half-finished conversions and the queue for surplus copies hold back the queues below them only once the conversion queue and the surplus queue hold a thousand chunks each. A small, steady flow of chunks into either queue therefore does not stop the rest of the work.
No queue takes the whole allowance either. Within one round, every queue after the client-waiting queue is allowed at most three quarters of what is left of the round's allowance when its turn comes. That limit lets the lower queues progress during a long repair.
The queues have a combined capacity, PRIORITY_QUEUES_LENGTH. Once the total is reached, a queue that already holds more than a hundredth of that capacity accepts a new chunk only by displacing a chunk from a less urgent queue. The less urgent queue must also be over that hundredth. When there is nothing to displace, the chunk is not filed and waits for the loop to reach it again. A queue under that hundredth keeps accepting chunks regardless. That rule leaves room for the urgent chunks. Filing a chunk in a queue speeds up the work. The queues are not a record of what needs doing: a chunk that never reaches a queue is examined anyway when the chunk loop reaches it.
The chunk loop's budget ignores the backlog
The loop that finds new work is the chunk loop, the pass in which the Master Server reviews every registered chunk. Two options set the pace of the chunk loop. CHUNKS_LOOP_MIN_TIME is the shortest time that a full pass may take, 300 seconds by default. CHUNKS_LOOP_MAX_CPS is the most chunks that the Master Server may examine per second, 100000 by default. The pass takes whichever of the two results in the longer time. On a small cluster, the minimum time decides the length. On a large cluster, the rate decides it.
The queue rounds use that same per-second rate, split evenly across the many timer rounds of each second. The rate applies twice: once to the queues and once to the chunk loop's own pass through the chunk table. Within the queues' share, the queue for chunks that a client is waiting for is emptied before any of the other queues.
Both options are bounded: the pass time to between 60 and 7200 seconds, and the rate to between 10000 and 10000000 chunks per second.
A replication is a pull, capped both ends
The Master Server moves no data. It names a source to the destination Chunkserver. The destination fetches the chunk from that source with the ordinary read command, on the port that clients read from. The load of a repair is therefore read traffic on the source.
Two options cap how many of these transfers may run at the same time. CHUNKS_WRITE_REP_LIMIT caps how many one Chunkserver may be receiving. CHUNKS_READ_REP_LIMIT caps how many that Chunkserver may be serving. Each option accepts five values. The value that applies depends on why the transfer was ordered.
| Position | What it caps |
|---|---|
| First | Chunks with no redundancy left. |
| Second | Every other repair the queues carry. |
| Third | Rebalancing between two Chunkservers whose space usage differs by no more than one and a half times the acceptable difference. |
| Fourth | Rebalancing between two Chunkservers whose space usage differs by more than that. |
| Fifth | Repairs that a client is waiting for. |
The defaults are 2,1,1,4,4 for the write cap and 10,5,2,5,10 for the read cap. Two shorter forms are accepted. Both are reported as outdated. A single number sets all five positions to that value. Four numbers fill the first four positions and make the fifth the largest of them. If the Master Server cannot parse a value at all, the result depends on when the value is read. On a reload of the configuration, the caps that were already in force stay. At startup, the value stops the server from starting.
The caps count redundancy rather than transfers. A whole copy counts as one against both ends, and an Erasure Coding part as half of one. A receiving Chunkserver with a cap of 1 can therefore receive two parts at once, but only one whole copy. The caps are read as decimal numbers. A fraction is not rounded: the Master Server compares the count with a per-Chunkserver offset between 0 and 1 that changes every second. A cap set between two whole numbers therefore takes effect as a share of the seconds rather than as the number below it.
An endangered chunk reaches Chunkservers that the rest cannot
The Master Server decides per chunk which Chunkserver receives a copy. The set of candidates is not the same for every queue. Ordinary repairs leave out a Chunkserver that reports itself overloaded. Ordinary repairs also leave out a Chunkserver that the Master Server has put in a grace period on its own initiative, and a Chunkserver that is running its own high-speed internal rebalance. A Chunkserver in maintenance mode is left out too, for as long as it stays connected. For chunks that a client is waiting for and for chunks with no redundancy left, those Chunkservers are put back in, behind the unrestricted candidates. The excluded Chunkservers are therefore used only when nothing else is available. Space and the cap still apply, whatever the chunk's priority. A Chunkserver is never chosen as a destination when it has one percent or less of its space free. A Chunkserver that is already at its write cap is not chosen either. The page on allocation and space selection describes the choice between the candidates that remain.
The source is chosen the same way, at random from the copies whose Chunkserver is under the read cap for that position. A copy on a regular disk is preferred over a copy on a disk marked for removal. With REPLICATIONS_RESPECT_TOPOLOGY turned on, the closest source is preferred instead of a random one.
A stalled replication is canceled, not waited out
The destination Chunkserver allows one transfer 150 seconds. The destination does not wait the full time to find out that the transfer is too slow. Every 30 seconds, the destination estimates the finishing time from the rate so far. The destination stops the transfer as soon as the estimate exceeds the limit. In both cases, the destination reports a timeout. The chunk is left as it was.
The Master Server keeps its own deadline, 120 seconds from the moment it issued the order. When the loop reaches a chunk whose incoming copy is still unfinished after that deadline has passed, the Master Server cancels the transfer. The incoming copy is marked invalid rather than deleted, because an interrupted transfer leaves a file whose version cannot be assumed. The chunk is unlocked so that the next pass can order the replication again. Chunk versions and states covers what an invalid copy is and what happens to it.
The loop skips a chunk entirely while the chunk is locked, being written by a client, or already under any other operation. Repair does not compete with client I/O for the same chunk. Repair waits for the client I/O instead.
Repairs wait for a complete set of reports
Nothing is repaired for the first REPLICATIONS_DELAY_INIT seconds, 60 by default. In the PRO Edition PRO (PRO Edition only), the count starts at this server's last change of role. A newly elected Leader therefore waits for the same period. Even with a single Master Server, becoming Leader is such a change. In the Community Edition, the count starts at the moment the Master Server finished starting up.
The delay dates from a time when the Chunkserver registration protocol had no explicit end. The delay is still useful in a cluster whose Chunkservers differ widely in speed: the fastest Chunkservers can finish registering before the slowest have started. The delay gives the slow Chunkservers time to send their registration packet. You can also raise the value to delay repairs on purpose. Some large failures have to be followed by a change of configuration, for example to a Storage Class definition. After such a failure, a raised delay keeps the cluster from moving data for as long as the delay lasts.
The same protection covers a Chunkserver that reconnects later. That protection does not use a timer. The whole repair pass is skipped while any Chunkserver is registering or still sending chunk lists because a disk is being scanned or removed. The pass is also skipped while the Master Server is still processing the copies of a Chunkserver that has disconnected. Only the queue for chunks that a client is waiting for keeps running. That queue also runs during the initial wait. Repairs resume automatically once the reports are complete, without any reconfiguration.
A disconnected Chunkserver in maintenance mode suspends the pass in the same way, for as long as every disconnected Chunkserver is in that mode.
A value outside its accepted range does not stop a reload. The Master Server changes the value to the nearer end of the range and continues with the changed value. For the length and the rate of the pass, the Master Server also records the value that it was given and the value that it used instead.
Deleting surplus copies has its own limit
Surplus copies are removed in the same pass, but under a separate cap. Reclaiming space therefore never uses a replication slot. CHUNKS_SOFT_DEL_LIMIT sets how many deletions may run at the same time on one Chunkserver, 10 by default.
That cap adapts to the backlog. Every sixteenth pass, the Master Server compares the deletions that it had to refuse with the deletions that it issued. The Master Server also compares the size of the backlog with the size at the previous check. When the refusals outnumber the deletions done and the backlog has grown, the Master Server multiplies the cap by 1.5, up to CHUNKS_HARD_DEL_LIMIT. When the backlog shrinks, the Master Server divides the cap by the same factor, one step per check, until the soft value is reached again. If you set only the soft limit, the hard limit is three times the soft limit.
Deleting a surplus copy is not allowed to come before a repair either. A chunk with a copy too many is filed below every chunk that is short of one. The surplus is therefore removed after more urgent work, from the share of each round that the higher queues leave free.