Chunkserver housekeeping loops
A Chunkserver does more than execute orders. The Chunkserver replicates chunks, deletes chunks, and changes their versions when the Master Server orders it. However, a set of background threads runs on the Chunkserver without any order, over its own disks. This page covers that work: what each loop does, how often, and which part of the work is visible to the Master Server. The Master Server's own background passes are a separate mechanism, covered in the chunk loop and the file loop.
Each loop has its own thread and pace
| Loop | Pace | What it does |
|---|---|---|
| Disk watcher | One pass per second | Refreshes each disk's space figures, counts its recent I/O errors, starts scans, completes disk removals, deletes expired duplicate chunk files. |
| Chunk scrubbing | HDD_TEST_SPEED MiB/s per disk | Reads chunks that nobody asked for and checks every block against its checksum. |
| Internal rebalance | Continuous, driven by usage spread | Moves chunks between the server's own disks to even out their usage. |
| High speed rebalance | Only when enabled | Replaces the background rebalance with a faster one for as long as it is on. |
| Deferred closing | Ten passes per second | Closes chunk files left open after their last operation and drops cached checksum tables. |
| Usage estimate | Continuous, a full pass takes about a minute | Re-sums, in memory, the disk space that the chunks on each disk take. |
Each disk that is being scanned adds a temporary thread of its own, so several disks are scanned in parallel.
Scrubbing and both rebalance forms have pages of their own, linked above. The rest of this page covers the loops that do not have their own page. The page allocation and space selection describes what the Master Server does with a server that reports the high speed rebalance or an overload.
The Chunkserver scans a new disk and reports its chunks in batches
At startup, and again on every configuration reload, the Chunkserver reads mfshdd.cfg. A disk that the Chunkserver has no chunk list for is scanned. The scan goes through the 256 directories that chunks are stored in and records every chunk file by name. The scan does not read the contents of the files. A disk that left a valid index of its contents behind is loaded from that index instead of being scanned. Disk errors and disk removal covers when the index is written and when it is rejected. Progress is written to the log, at most one line a second, as the percentage increases.
When the Chunkserver connects to the Master Server, the Chunkserver registers with its space totals. The Chunkserver then sends the list of every chunk that it holds, CHUNKS_PER_REGISTER_PACKET chunks per packet, 1000 by default. The option is set in mfschunkserver.cfg, along with every other option that this page names. Registration does not wait for scans to finish. Once the Chunkserver is registered, every chunk that a scan finds is queued and reported as a new chunk. The reports are sent in batches of up to 25000 per packet, in the next iteration of the Chunkserver's event loop. The Master Server delays ordinary repairs while those lists are still arriving. This delay is covered in replication priorities.
A reload uses the same reporting in reverse. The chunks of a disk removed from mfshdd.cfg are reported as lost, in the same batches. A disk whose mark for removal changed has its whole list sent again, because the flag of every entry changed with the mark.
The disk watcher measures and reports to the Master Server
Once a second, the disk watcher visits every disk. It asks the file system for the disk's free space at most once a minute per disk, and sooner when a chunk was created or deleted there. The same reading also serves as a health check of the mount point. It catches a size change or a change to read-only that takes the disk out of use without a single I/O error. Disk errors and disk removal covers those conditions. What the space figures count as available, including the reserve that HDD_LEAVE_SPACE_DEFAULT keeps back, is described in allocation and space selection. Whenever the totals change, a space report is sent to the Master Server at the next check. The information that the Master Server has about the space of a Chunkserver is therefore at most about a minute behind the disks.
The same pass counts each disk's recent errors against the tolerance that takes a disk out of use. The counting and its consequences are described in disk errors and disk removal. Each error is also reported to the Master Server when it is found. Chunks that fail a check are reported as damaged, in the next iteration of the event loop and not on a schedule. Independently of the watcher, the Chunkserver reports its load and status to the Master Server once a second.
The usage estimate loop supports the watcher from memory. The loop goes through the chunk records of the Chunkserver and sums again, disk by disk, the space that the chunk files take. One full pass takes about a minute. The loop reads no disk. The sums replace the used-space figure of the file system on a disk whose mfshdd.cfg line has a space option. The sums also give the average chunk size. That size is used to lower the reported space of a disk when free inodes will run out before free bytes.
A chunk file stays open for a moment
Chunk files are not closed at the moment an operation ends. Ten times a second, the deferred closing loop checks recently used chunks. A file descriptor that has been idle for half a second is closed. The checksum table read into memory for the chunk is kept for 100 seconds before it is dropped. With HDD_FSYNC_BEFORE_CLOSE = 1, a chunk that was written is fsynced before its descriptor is closed. The default is 0, which leaves the flush to the operating system. After the loop closes a written chunk, the loop measures the file once more, so the space accounting shows what the write really used on disk.
A duplicate chunk file waits a week
A scan can find a chunk that the server already has: the same chunk on two disks, or an older version next to a newer one. The Chunkserver keeps the best copy, the newest version not currently in use, and sets every other such file aside rather than deleting it at once. The set-aside files are kept for HDD_KEEP_DUPLICATES_HOURS, a week by default. This time is counted from when their disk came into service. Then the disk watcher unlinks them, up to 256 files per one-second pass. A duplicate of a chunk is removed sooner when that chunk is written to or deleted.
As long as a disk still has duplicates, the Chunkserver logs a notice with the count and the time left. The disk is excluded as an internal rebalance destination. The contents index that would let the disk avoid a full scan is also not written. The duplicates themselves are never reported to the Master Server. The Master Server has one registered copy on this server, the copy that the Chunkserver keeps.
Deleting on the Master Server's order is immediate
Deletion is the only job that looks like background work but is not started by the Chunkserver itself. When the Master Server orders the deletion of a chunk, the Chunkserver unlinks the file right away, in its worker pool. Nothing waits in a local queue. The pacing that makes deletions look gradual comes from the Master Server. This pacing is a cap on how many deletions may be in progress per server, described in replication priorities.
Some loops show as disk activity without clients
Three loops read or write chunk data on their own. Scrubbing reads every chunk in turn at its configured rate. The rebalance reads whole chunks from one disk and writes them to another disk while it is in progress. The scan at startup or after a reload goes through the chunk directories of its disk, unless the disk is loaded from its index. Expired duplicates cost the unlinks that remove them. The disk watcher, the deferred closing loop, and the usage estimate loop measure data rather than move it.
A Chunkserver's disks can therefore be busy while no client is reading. The Master Server receives states and results, never the transfers. It receives the load and status once a second, the changed space sums, error events, and chunks found damaged. The work itself is visible only as disk activity on the Chunkserver. When disks that should be idle are busy, check the internal rebalance and the scrubbing rate first.