Two data rooms

A cluster divided into locations keeps redundancy separately in each site. This page describes one such cluster, from the requirement that justifies the division to the failures that the cluster is built for. It also lists what each choice costs. Planning covers the decisions themselves. When locations help answers whether to divide at all. The WAN cost of writes covers what the distance between sites does to a write. Quorum across sites describes who may lead after a split. Divide a new cluster into locations describes how the division is built. Every later change to the division is a location task.

Either room can be lost without a restore

An engineering department runs its applications in one data room and keeps a second room a few kilometers away. A dedicated 10 Gbit/s link joins the second room to the first. A rack in a third building holds one small server. The data is 400 TB of project files, which the applications write and read all day. The data also includes a scratch tree of intermediate results that the applications can recompute from the project files.

The design has to deliver the following, in this order: losing either room loses no project file and needs no restore. The applications are available again in the surviving room without anyone reconfiguring the cluster. A room cut off from the other room never accepts writes of its own.

Site	Location	Addresses	What stands there
Main room	west, the default location	192.0.2.0/24	2 Master Servers, 12 Chunkservers, the application hosts
Second room	east	198.51.100.0/24	1 Master Server, 12 Chunkservers, standby application hosts
Third building	witness	203.0.113.0/24	one small server that runs a Chunkserver with no disk space

Three locations, because two cannot break a tie

The division follows the rooms. The third site exists for the vote. With two locations, a cut between the sites divides the vote one and one. The default location breaks the tie, so the other site elects no Leader of its own even when the default site is permanently lost (quorum across sites). A third location makes two votes out of three a majority. That majority turns the loss of a whole room from an operator's decision into an election that the cluster runs by itself.

The third site stores nothing. The site costs one server and two network paths. The vote comes from a Chunkserver: the machine runs a Chunkserver with no disk space, so that Chunkserver registers and votes without holding a single chunk. Each path must reach its room without passing through the other room. A third site that is reachable only through the main room goes down with the main room, and its vote goes down with it.

The default location stays with the applications

west is the default location, so it receives any module that nobody mapped. The default location's vote decides a count that divides evenly. In a cluster of three sites, that even count returns whenever one of the three locations is switched off. A location that is OFF is no longer counted in the election (location tasks). A cut between the two locations that remain then divides the vote one and one. With the default location in the room where the applications work, the applications keep writing during the cut instead of waiting.

Master Servers stand in the rooms only

Two Master Servers run in west and one in east. The third site runs none. Every Master Server holds the whole metadata set and needs the same class of machine as the Leader. A one-server rack is not meant for that. The majority is counted from Chunkservers anyway (leader election and quorum), so a Master Server there would add nothing to the count.

The second machine in west keeps the role inside the room when the first machine fails. With a Leader in east, every metadata operation that the applications issue would cross the link, including the grant that starts each write (the WAN cost of writes). The Master Server in east is a synchronized Follower. That Follower lets the room take over the role when it has to.

The Storage Classes decide what crosses the link

Each tree gets a Storage Class with a definition per location. The trees meet the requirement in different ways.

Tree	In west	In east	What the choice buys and what it costs
projects	two copies, created and kept	two copies, created and kept	A room can be lost in the second between a write and its confirmation without the file being lost. Every write carries the round trip of the link.
ingest	two copies, created and kept	two copies, kept	The nightly load of raw captures is confirmed at local speed and reaches east by replication afterwards. Until that replication finishes, the new data is in one room only.
scratch	two copies, created and kept	nothing	Intermediate results are recomputed instead of stored twice, and neither the link nor the second room's disks carry them. They go down with the room that holds them.

A class is met inside one location rather than across the cluster. Each room therefore needs Chunkservers of its own to hold the copies that the class requires there. Capacity follows the same rule: each room is sized for the whole data set, because redundancy is maintained once per location. No class names witness, so the third site stores nothing.

A cut leaves one room writing and the other room reading

west keeps two votes of three: its own, and the third site's. The Chunkserver in the third site holds a single connection to a Master Server. That connection goes to the Leader that is already running. The Leader stays where it is. The applications keep writing. The copies in east fall out of date until the link returns and the cluster replaces them (self-healing).

east holds one vote of three and elects nobody. This is how the split-brain protection works. A Master Server that is neither Leader nor Deputy opens no client session, so the room's mounts stop instead of accepting writes of their own. Set DEPUTY_ELECTION_TIME in mfsmaster.cfg on that machine. After the configured seconds, the candidate becomes the read-only Deputy (leader election, quorum, and the role of DNS). The isolated room can then read what it holds. The projects class gives more than durability: the tree has copies in east and stays readable there. scratch has no copies in east and does not stay readable there. To turn an outage that you know will last into a state that the cluster is configured for, use a location task.

Losing a room, and losing the third site

• The main room is lost. east and the third site hold two votes of three, so the Follower in east is elected without anyone deciding it. The project tree is complete there, in that room's own copies. The scratch tree is lost with the room. Its class defines exactly that outcome.
• The second room is lost. The Leader does not move. The applications are not affected. The project tree is short of its east copies, counted against that location's own definition (Location), until the room is available again and the cluster replaces them.
• The third site is lost. Two votes of three remain, so nothing moves. The next failure has a different effect: while the site's location is still ON with its Chunkserver offline, that location votes for nobody. A cut between the rooms then leaves neither room with a majority. Switch the location OFF for as long as the site is unavailable. The count then falls back to the two rooms, where the default location breaks the tie.