Skip to main content

Chunkserver

A Chunkserver is the storage node of MooseFS, the component that keeps the contents of your files on its disks.

What it is​

A Chunkserver holds file data in the form of chunks, reports its space to the Master Server, and answers read and write requests from clients. A cluster runs as many Chunkservers as it needs capacity for. Every one of them keeps a connection to the Master Server.

A Chunkserver stores chunks, not files. Each chunk file has a chunk ID and a chunk version in its name, and nothing more. Names, directories, permissions, and the list of chunks that a file is made of are metadata and stay on the Master Server. A Chunkserver therefore has no view of the file system that it takes part in.

Why it matters​

The capacity of a MooseFS cluster and the bandwidth that the cluster can serve are the sum of what its Chunkservers contribute. A cluster therefore grows in two ways. A disk added to a Chunkserver adds capacity. A Chunkserver added to the cluster adds capacity and bandwidth.

The number of Chunkservers also decides how much the cluster can survive. MooseFS keeps the copies or erasure-coded parts of a chunk on different Chunkservers, so a Chunkserver is the unit whose loss the cluster is designed to withstand. That number also limits the redundancy that you can actually get. A Storage Class that requires more copies or parts than there are servers to put them on cannot be satisfied. Adding Chunkservers therefore increases reliability as well as size and throughput.

Data never passes through the Master Server. Clients ask the Master Server where a chunk is. The clients then exchange the data with the Chunkserver directly. For this reason, the Master Server does not become the bottleneck of a busy cluster.

How MooseFS implements it​

mfschunkserver is a user-space daemon that stores chunks as ordinary files on ordinary local file systems. That is deliberate. MooseFS defines no on-disk format of its own and uses only plain POSIX file operations. The same code therefore runs on different systems and hardware. You choose the local file system under each disk, including encrypted or compressed file systems. Ordinary disk tools remain usable for rescue and repair. The directories that the Chunkserver may use are listed in mfshdd.cfg, one per line. Each entry is normally the mount point of a single disk. The Chunkserver creates its own subdirectory tree there.

A Chunkserver is designed to get its disks individually, as JBOD. Each disk has its own file system and its own line in mfshdd.cfg. MooseFS keeps data redundant itself, across Chunkservers. MooseFS also detects a failing disk itself: the Chunkserver counts the errors that each disk returns and takes a disk out of use when there are too many. RAID under a Chunkserver therefore adds a second layer of redundancy that the cluster does not need. A RAID controller can also hide the errors of a failing disk from the Chunkserver. For these reasons, RAID is not recommended for Chunkserver disks. Hardware requirements for Chunkservers explains the choice of disks in full.

The Chunkserver opens the connection to the Master Server, not the other way round. On the first connection, the Chunkserver receives an identity that it stores locally and reuses afterwards, so a restarted Chunkserver is recognized as the same server. To register, the Chunkserver reports the address and port that it listens on, its used and total space, and the full list of chunks that it holds. From then on, the Chunkserver reports space and chunk count whenever they change. The Master Server sends back chunk operations (create, delete, duplicate, truncate, change version, and replicate), but never file data.

That list of chunks makes a returning Chunkserver useful. Every chunk has an identifier that is unique across the whole instance. The Master Server therefore recognizes each reported chunk as the chunk that was missing. The Master Server counts that chunk again toward the redundancy of its file. A Chunkserver that returns after a repair therefore brings its chunks back with it. The copies that the cluster had begun to recreate are no longer needed. A Chunkserver does not have to return whole. A Chunkserver that returns without one of its disks registers only the chunks that it still has. The chunks that the Chunkserver no longer reports are recreated from the copies or erasure-coded parts held elsewhere.

Clients connect to the Chunkserver's own listening port for data. A read is served from the local disk. A write goes to one Chunkserver. That Chunkserver stores the data and forwards the data to the next Chunkserver that is to hold the chunk.