Skip to main content

Limits and scale

A MooseFS cluster grows along two axes. The two axes do not grow the same way. Data spreads: capacity and throughput are added machine by machine. Metadata concentrates: however many machines the cluster has, one machine answers every metadata operation and holds every object in RAM. This page describes both axes. It also describes the background work that lengthens as the cluster grows and the workload shape that changes what each axis costs.

Data scales out with the Chunkservers​

The capacity of a cluster and the bandwidth that it can serve are the sum of what its Chunkservers contribute. A client asks the Master Server where a chunk is. The client then exchanges the data with the Chunkserver directly, so no single machine carries all of the file data. Every added Chunkserver adds its disks and its network interface to the resources that the traffic spreads over.

An added Chunkserver takes effect in two steps. It receives its share of new chunks as soon as it registers, because new chunks are allocated in proportion to capacity. The data that already exists reaches the new server only by rebalancing. Rebalancing is throttled by design and has a lower priority than every repair. An added server therefore receives new writes at once. Its usage reaches the usage of its peers only as fast as the rebalance limits allow.

A cluster has a fixed ceiling of 60 000 Chunkservers connected at one time (limits). Deployments stay far below this ceiling.

A Chunkserver holds one entry in RAM for every chunk on its disks. Each Chunkserver therefore adds memory of its own to the sum. That memory cost grows with the chunk count, not with the terabytes (Chunkserver memory).

A network interface does not give every workload the same throughput. Large blocks reach the speed of the card and of the links between the machines. Small blocks stop below that speed, on the card, on the bus, or on the CPU that drives them. MooseFS modules hold their connections open and keep them alive instead of opening a connection for each transfer. The interface therefore fills before the socket table does. Which part runs out first depends on the instance. Two readings from the machines show which part runs out first: how much of the link is in use, and whether the interfaces are counting errors. MooseFS measures neither.

The throughput of a single mount does not grow with the cluster. Aggregate throughput rises with the number of Chunkservers. The ceiling of one mount is in the FUSE layer of its own machine, not in the cluster (per-mount throughput).

Metadata scales up with one machine​

One Master Server answers every metadata operation, such as a lookup, a create, or a rename. In the PRO Edition this Master Server is the Leader PRO (PRO Edition only). Followers replicate the Leader's metadata and are ready to take the role over. Followers serve no clients, so adding Followers adds availability, not metadata throughput.

Memory is the other half of the same axis. Every object, every name, and every chunk in the instance is a record in the Master Server's RAM. The memory of that one machine therefore decides how many objects, names, and chunks a cluster can hold (Master Server memory). What metadata holds lists which records multiply with which use. Clients add records too: every mount is a session. The Master Server keeps this record after the client disconnects. MooseFS sets no ceiling on how many mounts connect at once. The machine that runs the Master Server sets that ceiling instead (limits).

Background passes lengthen with the chunk count​

Some costs of size appear as time. The Master Server reviews every chunk in the chunk loop. The loop runs on a fixed budget: a shortest pass time and a cap on chunks examined per second. A pass takes whichever of the two works out longer. The budget does not grow with the backlog. When the rate cap is the limit, the time of a pass is therefore proportional to the number of chunks. The delay between setting a policy and seeing its effect grows with that number. The pacing options and their defaults are described in replication priorities.

Repair depends on the amount of data, not on the size of the cluster. When a disk or a Chunkserver is lost, its chunks are re-replicated under the fixed concurrency caps described in replication priorities. The time to restore redundancy therefore grows with the amount of data that was on the failed disk or Chunkserver. The caps count transfers per Chunkserver. This is the only reason why a bigger cluster repairs faster: many servers pull the missing copies at once, each within its own cap.

A Chunkserver has a cost of its own at startup. A disk that left no valid contents index behind is scanned chunk file by chunk file. The chunk lists are sent to the Master Server in batches. The cluster needs time to register everything that the Chunkserver holds. This time therefore grows with the number of chunks on the Chunkserver's disks (Chunkserver housekeeping loops).

Small files grow objects faster than bytes​

The metadata axis is counted in objects, names, and chunks, never in bytes. A tree of small files costs the Master Server as many nodes and names as the same tree of large files. The small tree holds a fraction of the data that the large tree holds. Every file that holds any written data costs at least one chunk. Each copy of that chunk has a fixed overhead on the Chunkservers' disks (small files). A workload of many small files therefore fills the Master Server's RAM and lengthens the chunk loop long before the workload fills any disk.

Some ceilings are fixed in the product​

Identifier widths and counts written into the code do not change with configuration or hardware. Limits lists them. Where MooseFS sets no maximum of its own, the same page names what decides the ceiling instead. The capacity of the cluster is one such case. Nothing in MooseFS caps the sum of what the Chunkservers report. The width of the counter that holds this figure limits the sum instead (max cluster capacity).

A chunk identifier is 56 bits wide. This width caps how many chunks one instance holds. At 100 000 new chunks every second, a cluster would take about 23 000 years to reach the cap. That rate is far above any real workload. The Master Server's RAM ends the growth long before that, the same way it does for objects and names (Master Server memory).