Chunkservers

The design of a Chunkserver machine starts from its disks. Above all, give those disks to MooseFS whole: separate drives, not a RAID array. Like every MooseFS component, the Chunkserver runs on a POSIX-compliant operating system. Chunkservers needed for an Erasure Coding scheme calculates how many of these machines an Erasure Coding scheme needs. Mixing hardware covers what happens when the machines or their disks are not all alike.

The cluster is redundant, so RAID adds no more safety

MooseFS keeps data redundant itself, in copies or in Erasure Coding parts spread across machines. A RAID array under the chunk disks adds a second redundancy layer beneath the first. That layer adds complexity but no more safety. The layer can also interfere with the replication and fault tolerance of MooseFS.

The second argument against RAID matters more than the first. A RAID controller reports the array, not the drives inside it. Until the array itself degrades, the controller can hide a drive that returns errors, remaps sectors, or fails its SMART thresholds. Because the controller completes each operation, the kernel sees no error. Operations that slow down or hang reach the cluster instead of errors that could be counted. The Chunkserver takes a failing disk out of service by counting the errors that the operating system returns. Monitoring reads the same records, so the Chunkserver and the monitoring both see only what the controller lets through. Attach each disk directly, as JBOD or through an HBA in IT mode. The state of every drive then stays visible.

Each chunk disk holds one partition and a simple file system

Give the operating system a small disk of its own and the chunks the rest. Each chunk disk works as a single partition and is mounted individually. Each partition has a simple, stable file system with no redundancy of its own. The recommended file systems are XFS and ext4 on Linux, and ZFS on FreeBSD. On FreeBSD, use one plain pool per disk: no RAID-Z, no mirror, no extra copies, because MooseFS keeps the redundancy. ZFS is recommended on FreeBSD because FreeBSD supports ZFS more fully than any other file system, including FreeBSD's own UFS. UFS is also good for chunks if soft updates are off. mfshdd.cfg lists the mount points for MooseFS. The Chunkserver also keeps its data directory under /var/lib/mfs. That directory holds the identity of the Chunkserver, its chart history, and its lock, a few megabytes in all. The records of chunk IDs stay on the chunk disks themselves.

Several partitions on one disk also work. The Chunkserver accepts each partition as it accepts any other mount point. The arrangement is one to avoid, not one that is forbidden. The arrangement costs three things.

The first is a false picture of redundancy. Two partitions of one drive appear in the cluster as two disks. The drive under both partitions fails as one.

The second is a divided mechanical disk. The Chunkserver reads a chunk from beginning to end. The Chunkserver also tells the operating system to expect that, so the read-ahead of the kernel is useful. Both work best when nothing else moves the head.

The third is a split view of the health of the drive. The Chunkserver counts errors and withdraws a disk per entry in mfshdd.cfg, so errors on one partition never count toward the tolerance of its neighbors. Those neighboring partitions keep serving chunks from a drive that has already reported failures. Disk errors and disk removal covers that counting.

A disk reports less space than it holds

MooseFS sets no minimum size for a disk, but a small disk provides less space than it holds. The Chunkserver keeps a reserve on every drive listed without a space option in mfshdd.cfg (HDD_LEAVE_SPACE_DEFAULT in mfschunkserver.cfg, 256 MiB unless changed). The Chunkserver subtracts that reserve from the free space that it reports to the Master Server. The capacity that the Chunkserver reports is the size that the file system reports, less the blocks that the file system keeps for the superuser. When the free inodes would run out before the free space, the Chunkserver lowers both the capacity and the free space to what those inodes can hold at the disk's average chunk size. A Chunkserver keeps receiving new chunks as long as it reports more free space than one chunk, 64 MiB. That threshold rises by ten chunks for every chunk copy that clients hold open for writing on that Chunkserver. Allocation and space selection covers which disk a new chunk is then placed on. Rebalancing covers how unevenly filled disks even out over time.

Physical cores matter more than clock speed

The Chunkserver is a multi-threaded process. It serves clients and background jobs from two pools of worker threads. Each pool grows to 250 threads by default, so concurrent work spreads across every core that the machine has. Under heavy I/O, or with many clients reaching the machine at once, a high count of physical cores gives more throughput than a higher clock.

The processor demand follows the traffic

A Chunkserver with no traffic on it uses little of its processor. That use rises with the work that reaches the Chunkserver. How steeply the use rises depends on the size of the cluster rather than on the machine. The same rate of operations spread over six Chunkservers costs each Chunkserver far more than the same rate spread over twenty. Labels are the exception. When labels keep the data that generates the traffic on a subset of the servers, the work concentrates on that subset. Repair adds to all of it, because rebuilding the chunks of a failed machine runs replications on top of the regular work. Size the machine for the cluster that is repairing itself, not for the quiet cluster.

Compared with the Master Server, the demand is modest. Every operation that a Chunkserver performs corresponds to an operation of the Master Server. One Master Server answers all of those operations from a single thread, so the Master Server machine reaches its processor limit long before the Chunkservers do. There is no figure for which of the operations of a Chunkserver cost the most.

The default worker count is a ceiling rather than a target. On most instances, the number of workers in use never rises above the default minimum. Clusters that have needed WORKERS_MAX raised are rare. Watch the demand when a Chunkserver shares its machine with other software. Computation on storage nodes covers how to tell whether a Chunkserver is short of resources.

RAM follows the chunk count

Plan about 250 MiB of RAM for every million chunks that the machine will hold. That is a planning figure. On a running Chunkserver, the measured use is about 212 bytes per chunk, below the planning figure. The memory that the process uses changes with activity and configuration, so the planning figure is a baseline for sizing, not a ceiling.

A Chunkserver with no disks only votes

A Chunkserver with no chunk space defined in mfshdd.cfg registers with the cluster and works only as a voter PRO (PRO Edition only). It holds no data. Its vote counts when the Master Servers elect a Leader. A cluster design can therefore include a machine without data disks on purpose. That machine breaks the tie among an even number of Chunkservers that hold data. Quorum across sites covers when to do that.

A Chunkserver is sized by these figures

What you are sizing	Figure
Space a disk reports	The file system's size, less its root reserve. The reserve that HDD_LEAVE_SPACE_DEFAULT holds back, 256MiB by default, comes off the free space. A disk short of inodes reports less, in proportion to the chunks those inodes can still hold.
Free space worth keeping	One chunk, 64 MiB, plus ten chunks per chunk copy open for writing. Below it a Chunkserver takes no new chunks.
RAM	About 250 MiB per million chunks as a planning figure. The measured use on a running server is about 212 bytes per chunk, below that figure.
Worker threads	Two pools, each grown to WORKERS_MAX threads, 250 by default. The value is a ceiling. Most instances never leave the minimum.
Processor	Physical cores over clock speed. The demand per machine is higher in a smaller cluster.