How It Works

This chapter describes one mechanism at a time, in depth: what the mechanism does, what states it passes through, what it costs, and where its limits are.

The life of a file opens the chapter, because a reader comes here for everything that happens to a file between its creation and its last byte leaving the disks. The metadata engine covers what the Master Server holds, how it survives a restart, and how the role passes to another machine. The chunk lifecycle describes what happens to a chunk from placement until the cluster takes the disk under the chunk out of use. Placement and policies covers the rules that decide which Chunkservers store a chunk, and how much redundancy the chunk is kept at.

Maintenance loops covers the background passes that bring the cluster back to what its policies require, and what decides how fast those passes run. The client covers what a mount costs, how much data it can transfer, and what happens when there is more than one mount. Limits and scale lists what runs out first as a cluster grows: the memory of one machine, the number of chunks that every background pass processes, and the ceilings fixed in the product.

This chapter answers "in this situation the system does X". Architecture covers what the parts are called and how they are wired together. Concepts covers what a term means. Operations covers how to carry a task out.