Mixing hardware
A MooseFS cluster does not require identical machines. Chunkservers of different sizes, disks of different capacities, and machines bought years apart work in one cluster. This is possible because the Master Server assigns work in proportion to what each server has rather than in equal shares. The cluster evens out space usage on its own, but not speed. This page covers that difference.
Capacity differences even out on their own
The Master Server assigns new chunks to Chunkservers in proportion to their capacity, so a server twice as large receives about twice as many chunks. The Master Server excludes only servers that are out of room, busy, or in maintenance mode. Allocation and space selection describes that choice. Usage that is already uneven is corrected by rebalancing. Rebalancing moves existing chunks at a deliberately throttled pace, inside each Chunkserver and across the cluster. Neither mechanism needs the disks or the servers to match. A cluster of unequal machines reaches even space usage, not equal chunk counts per machine.
Speed differences stay, so the slowest machine sets the limit
Placement ignores speed. The choice of servers for a new chunk uses free space, load, topology, and labels. Disk speed is not among these factors. A write to a chunk covers every copy. The write goes from server to server down a chain, so the write finishes at the speed of the slowest machine that holds a copy. A slow machine in a fast cluster therefore does not lower the total of the cluster. Instead, that machine slows every operation that happens to involve it. Plan with that in mind. The fast machines around slow hardware do not make the slow hardware any faster.
New hardware fills by rebalancing, so extend at once
A Chunkserver added to a running cluster starts receiving new chunks immediately, in proportion to its capacity. The same applies to a new disk inside a server. The data that already exists reaches the new hardware only by rebalancing. Rebalancing is throttled by design. How long rebalancing takes depends on how much data has to move and on the replication limits. That page describes the calculation.
When an expansion is meant for several Chunkservers, extend all of them at once rather than one machine at a time. A server extended first receives the replications that the remaining servers would have shared. That server gives part of that data back once the remaining servers are extended too. A staggered order therefore fills disks only to empty them again.
Labels separate hardware that you do not want mixed
Different capacities mix on their own. Different tiers do not. Consider a cluster that is meant to hold two kinds of hardware with different jobs, such as SSD next to HDD. Decide at design time which data belongs on which kind. Write the decision down as labels on the Chunkservers and as Storage Classes that name those labels. Without labels, both tiers form one pool. A chunk is then placed on either tier. Label expressions and matching covers how an expression on a Storage Class selects labeled servers.