Skip to main content

The Chunkserver

The Chunkserver is the storage role of a MooseFS cluster: the mfschunkserver process keeps chunks on local disks and serves them over the network. The Chunkserver concept covers what the role is for, and why a cluster grows by adding more Chunkservers. This page describes how the process lays out its store, which peers it communicates with, and which of its states the rest of the cluster reacts to.

The store is a fixed tree​

The Chunkserver writes to the local file systems listed in mfshdd.cfg. Under each listed path the Chunkserver keeps 256 subdirectories, named 00 through FF, and spreads chunk files across them. A chunk file is named after what it holds: chunk_<id>_<version>.mfs, with the chunk ID as 16 hexadecimal digits and the chunk version as 8. For example, the third version of chunk 0xA4F is on disk as chunk_0000000000000A4F_00000003.mfs. Before the data, the file holds 8 KiB of the Chunkserver's own metadata: a 4 KiB header and a 4 KiB table of block checksums. Small files covers what that fixed overhead costs when files are small.

Nothing in this store names a file. The ID and the version are all that a Chunkserver stores about a chunk. Directories, file names, and which chunks make up which file are metadata and stay on the Master Server.

The data directory is a different place, the directory that DATA_PATH points at (/var/lib/mfs in the packages). It holds no chunks. The Chunkserver keeps its chart history there, in csstats.mfs. Charts and statistics covers what that file holds and what a restart does with it.

The Master Server commands, but file data flows elsewhere​

The Chunkserver opens one connection to the Master Server and keeps it open. Over that connection the Chunkserver registers, reports its space and the chunks that it holds, and receives chunk operations in return. The Chunkserver never sends file data over that connection. Chunkserver housekeeping loops covers the background threads that scan disks, test chunks, and supply those reports.

Everything else arrives on the Chunkserver's own listening port. Clients read and write chunk data there. Other Chunkservers use the same port in both directions. A chained write is forwarded to the next Chunkserver in the chain. A Chunkserver that is ordered to replicate a chunk fetches the chunk from the data port of the Chunkserver that has it. Network ports covers which port numbers these are and how a Chunkserver's address reaches clients and peers.

In PRO the Chunkserver also reads the state of the Master Servers​

The PRO Edition PRO (PRO Edition only) gives the Chunkserver a part in choosing which Master Server leads. Each Chunkserver runs the supervisor pass on a schedule that the Leader assigns. In that pass the Chunkserver connects to the control port of every Master Server to read its state. Leader election describes the pass and the election that it serves. The control port is listed in network ports.

A Chunkserver whose mfshdd.cfg defines no space still registers and works as a voter only. It stores nothing, but it counts toward the majority in the election.

Some states change what the Master Server sends​

  • Overloaded, reported by the Chunkserver itself: allocation and space selection covers what the Master Server stops sending to an overloaded Chunkserver.
  • Internal rebalance, in either of its two speeds: rebalancing covers both and how the Master Server treats each.
  • Maintenance mode, a flag kept on the Master Server rather than reported by the Chunkserver.