Allocation and space selection
A chunk is placed once, at the moment it is created. The Chunkservers that will hold the chunk are chosen before the client sends any data. This page describes that choice: what the Master Server counts as free space, which Chunkservers the Master Server will not use, and how the Master Server picks among the rest. The page also describes what a client is told when the Master Server can pick none.
The Master Server picks, the client obeys
A client that needs a chunk that does not exist yet asks the Master Server for one. The Master Server reads the CREATE definition of the file's Storage Class. Where the class defines none, the Master Server reads the KEEP definition of the class instead. The definition says how many copies the new chunk is to have and which Chunkservers may hold them. The Master Server first reduces the connected Chunkservers to those that can accept a chunk at this moment. Then the Master Server matches that set against the definition and gives each copy a server of its own. Finally, the Master Server orders every chosen server to create the chunk. The client sends the data itself only once, to the first server on the list. The data then passes from server to server down the chain, in a chain-like write. The traffic between the client and the cluster therefore does not grow with the number of copies. The client receives the chunk and its list of servers only after those servers have answered. When every one of those servers refuses, the client gets an error for that write instead of a chunk.
Because the candidates are servers rather than disks, the result has two properties. Every copy is created on a different Chunkserver, because each connected server is in the pool exactly once. A Chunkserver is a process, not a machine. The Master Server tells servers apart by their connections and cannot know that two servers share hardware. Where several Chunkserver processes run on one physical machine (possible, and occasionally used), their copies count separately. Redundancy across physical machines is therefore the operator's responsibility. The selection also decides a number of machines. A new chunk is stored in copies, whatever redundancy format its class requires in the other storage states, so nothing is split into parts at this point.
In an instance divided into locations PRO (PRO Edition only), the selection runs once per location, against the part of the definition that applies to that location. Each location gets its own copies. A location that is switched off is skipped. So is a location for which the definition requires no copies. One location that can be served is enough for the write to go ahead. A location that could not be served is brought up to its definition afterwards, by the same replication that repairs any other shortfall.
Chunkservers report the free space
For each of its disks, a Chunkserver calculates how much space is still available and how much space is used. The Chunkserver sends the sums for the whole server to the Master Server. For a disk listed in mfshdd.cfg without a space option, the available space is the free space that the file system reports, less a reserve. The reserve is HDD_LEAVE_SPACE_DEFAULT in mfschunkserver.cfg, 256 MiB unless you set it otherwise. A disk with a space option is measured by that option instead. The exception is the shared indicator = on its own, which keeps the reserve as well. Disks marked for removal are summed separately and are not part of the figure that placement uses. Damaged disks and disks still being scanned are in neither sum.
How much space is used depends on how the disk is listed. Without a space option, the Chunkserver uses the totals that the file system reports. On a disk shared with something other than MooseFS, the count therefore also includes data that MooseFS did not write. With a space option, the Chunkserver sums the occupancy of the chunk files instead, measured in disk blocks rather than chunk lengths. Other data on the disk is not counted. Either way, a chunk written with holes in it counts for less than a whole chunk. Small files explains the difference between the length of a chunk and its occupancy.
Neither figure is updated continuously. A Chunkserver recomputes the figures for a disk when the chunks on that disk change, and in any case once a minute. The Chunkserver reports to the Master Server at most once a second when its totals have changed, and about once a minute in any case. Placement uses figures that lag behind the disks. This delay is one reason why a Chunkserver can still refuse a chunk that the Master Server has already assigned to that Chunkserver.
A candidate needs room beyond one chunk
A Chunkserver enters the pool for new chunks only when the space that it reports as free is greater than one chunk. The Master Server raises that requirement by ten chunks for every chunk copy that clients currently hold open for writing on that server. Each of those copies can still grow to a full 64 MiB, but none of that space is used yet. The margin is several times wider than that arithmetic alone requires, so the space is there well before the next chunk is added. A server close to full stops receiving new chunks even though it still has room. A server that serves many writes stops sooner than an idle server with the same free space.
The Chunkserver makes the last decision. A Chunkserver that is told to create a chunk picks the disk itself. The Chunkserver weights the choice so that larger disks receive proportionally more chunks. A disk is eligible when it is not damaged and not marked for removal or being removed. The disk must also be done with its startup scan, must not be forced by configuration to be a rebalance source, and must not be more than 99.9% used. When every disk that meets the remaining conditions is past 99.9%, only that last limit is waived. The remaining conditions always apply. If no disk qualifies, the Chunkserver refuses.
An overloaded Chunkserver drops out, a busy one waits
The Master Server keeps a load state for every connected Chunkserver. The state is based partly on what the server reports about itself and partly on what the Master Server observes. Each state either drops the server from the pool for new chunks or only holds the server back.
| Chunkserver state | Where it comes from | Effect on a new chunk |
|---|---|---|
| Overloaded | The Chunkserver keeps two pools of I/O workers, each sized by WORKERS_MAX. The Chunkserver reports the state once the workers that are busy at the same time in either pool pass three quarters of the limit, and withdraws the state when both pools fall below half | Dropped from the pool |
| Fast internal rebalance | The Chunkserver reports this state while moving chunks between its own disks at full speed | Dropped from the pool |
| Slow internal rebalance | The Chunkserver reports this state while moving chunks between its own disks at the background rate | Held back |
| Maintenance mode | An operator sets this state, or the Chunkserver announces a controlled stop | Held back |
The worker limit is set in mfschunkserver.cfg. The states that a Chunkserver reports about itself describe the machine's own capacity to answer. Whether neighboring Chunkservers are doing better plays no part.
Being held back is not the same as being dropped. The Master Server sets the servers that are held back aside. The Master Server then compares their number with the candidates that passed every earlier filter. If the servers set aside are more than a fifth of the candidates that passed, the Master Server puts all of the servers set aside back into the pool. Taking a fifth of a cluster out of service because those servers are busy would push every new chunk onto the remaining servers. That would make a busy cluster busier. For the same reason, the Master Server stops avoiding a Chunkserver in maintenance mode when avoiding it would mean avoiding too many servers.
Capacity, not free space, sets the order
The Master Server does not order the remaining candidates by how full they are. For each Chunkserver, the Master Server counts the chunk copies created anywhere in the Chunkserver's location PRO (PRO Edition only) since that server was last chosen. In an instance without locations, the count covers copies created anywhere in the cluster. The Master Server compares that count with the interval that applies to that server: the total space of the location divided by the total space of that server. The server that is furthest past its own interval goes first. Over many chunk creations, each Chunkserver therefore receives chunks in proportion to its size. A server twice as large receives about twice as many chunks, whatever its current usage. A different mechanism corrects usage that is already uneven: rebalancing moves chunks that already exist.
Two things move a server down the order without removing it. Chunk copies open for writing on that server count against it, and so do replications that it is currently receiving. Both weigh heavily enough to move a server that serves traffic to the end of the list. A Chunkserver that has not been chosen since it connected goes near the front of the order. A server added to a running cluster therefore starts receiving chunks at once instead of waiting its turn.
CREATIONS_RESPECT_TOPOLOGY adds one pass on top of that order. Setting the option to a number moves every candidate whose topology distance from the writing client is smaller than that number ahead of the rest. Both groups keep the order that they already had. Distant servers are only moved back, not removed. A write whose nearby servers cannot accept the chunk therefore still succeeds on servers further away.
Labels assign the copies, a shortfall writes nothing
The class definition is applied only at this step. The Master Server goes through the ordered candidates and assigns them to what the definition requires, one distinct server per copy. A definition that requires copies on different kinds of hardware is therefore met by an assignment, not by taking the first servers on the list. What the definition can express, and how strictly it binds, belongs to the labels that the Chunkservers have and to the class itself.
A shortfall is a refusal. When a location has fewer usable servers than the definition requires copies, or under strict label matching fewer matching servers, the Master Server creates no copy of the new chunk in that location. When no location can be given all of its copies, the Master Server creates no chunk and answers the client (a write with no server waits or fails). Up to 5.1.0, the Master Server created the chunk on the servers that could be assigned and made the missing copies afterwards. A copy that ended up on a server that the definition would not have chosen is still corrected afterwards. That correction is the lowest-priority work that the Master Server schedules, behind every chunk that is short of copies or parts. Replication priorities describes the order.
A shortfall that the filter itself produced changes only the answer. If the candidates that were dropped as overloaded or fast-rebalancing would have been enough to meet the definition, the Master Server tells the client to come back later instead of reporting a lack of space.
A write with no server waits or fails
When the selection falls short of the definition in every location, the answer depends on what the Master Server was able to rule out.
| What the Master Server found | What it answers | What the mount does with the answer |
|---|---|---|
| Candidates were dropped as overloaded or fast-rebalancing and would have sufficed, or a Chunkserver is missing while the cluster is still settling after a startup, a reconnection, or a server registering its chunks | Come back later | Retries and keeps retrying: three times after 1 ms, four times after 100 ms, then twice a second for as long as the condition lasts |
| A Chunkserver that the cluster knows about is disconnected, and nothing else is in progress | No Chunkservers | Retries with 300 ms more of delay before each attempt than before the previous attempt, until the attempts reach mfsioretries, then fails the write with ENOSPC |
| Every Chunkserver that the cluster knows about is connected and the definition still cannot be met: fewer servers have the room than the definition requires copies, or under strict label matching fewer of them fit the expression | No space | The same, unless the mount was given mfserroronnospace, which fails the write at once |
The answer that tells the client to come back works differently from the two answers that do not. That answer is temporary by design. The Master Server has found servers that would be enough and is waiting for them to become usable. The mount never fails a write because of that answer, so an application sees a pause instead of an error. With either of the two other answers, the Master Server reports that it has checked everything that it can see. The Master Server cannot tell whether the missing Chunkserver will reconnect, so the retry budget decides instead. With mfsioretries at its default of 30, the mount retries for roughly two minutes before the write fails. The command line tools reference lists both mount options with the rest.
Creating a chunk can also fail after the servers have been chosen. The Master Server's figures for free space lag behind the disks. A Chunkserver may therefore accept the assignment and then find no disk that can accept the chunk. If at least one of the chosen servers created the chunk, the write goes ahead with fewer copies. The shortfall is repaired later. If all of the chosen servers refused for lack of space, the client is told that there is no space. The client treats that answer exactly like the same answer from the selection itself. A refusal for any other reason is answered as an operation that was not done.