Quorum across sites

A cluster divided into locations elects its Leader by locations. Each location votes as a unit. The default location breaks a tie. The majority is counted location by location covers that count and the weight of each location. Before you build the cluster, decide where the votes are: how many sites take part, which site is the default, and what decides an otherwise even count. When locations help covers whether the cluster should span sites at all. The WAN cost of writes covers what the distance between the sites costs a write. Quorum decides who may lead, not what leading costs.

The count is by site, not by Chunkserver

An undivided cluster counts its Chunkservers. A candidate becomes the Leader once more than half of the registered Chunkservers reach the candidate. That was the whole rule until locations were introduced. That rule still applies wherever a cluster is not divided. Two ordinary site layouts defeat that rule as soon as a cluster is divided.

A layout of unequal sites defeats the rule by weight. One main site with twelve Chunkservers and two backup sites with five each add up to twenty-two. Losing the main site leaves ten, which is less than half. No Leader can be elected, although two whole sites are running and hold a complete copy of the data between them.

A layout of two equal sites defeats the rule by symmetry. Losing one site leaves exactly half of the Chunkservers. Exactly half is not more than half, so again no Leader is elected on either side.

Counting by location solves both problems. A site's own Chunkservers decide only that site's vote. Their weight therefore stays inside the site and is never added to a cluster-wide sum. The votes themselves are counted over sites. With an odd number of sites, that count has no tie. The rule has one cost: an even number of sites brings the tie back. Something has to break that tie. The default location breaks it.

The default location breaks a two-site tie

With two locations, a network split between the sites divides the votes one and one. The tie goes to the default location. The site that holds the default location keeps or elects the Leader. The other site cannot, however many Chunkservers it has. Choose the default location for that reason. The default location is the site that stays writable through a split. Make it the site that the clients work in, or the site whose power and network you trust more.

The same tie sets the cost of a two-site design. When the default site itself is lost, the surviving site does not elect a Leader on its own. The surviving site also cannot be made to elect one from inside the cluster. The state and the default of a location are metadata. Changing either of them goes through a Leader. The tools work through a mount. A Master Server that is neither the Leader nor a DEPUTY opens no session at all. A DEPUTY opens the session and then answers every location command as read-only. No tool closes that gap either: a supervisor forces a state and cannot create a majority. Design the two-site cluster so that it does not need that recovery.

A third site makes a majority

Three locations give two votes to whichever side of a split holds two sites. Two out of three is a majority with no tie to break. The third location does not have to hold data. In a split between the two main sites, the Leader is on whichever of them the third site can still reach. That vote can come from a machine that stores nothing. The third site then costs one small server and a network path to both main sites, not a third room of Chunkservers.

The arbitrating site does need a Chunkserver of its own that is registered with the cluster. A location's vote follows the Chunkservers registered in that location. A location with no registered Chunkserver therefore has nothing to follow. Such a location is not the arbiter that the design requires. One registered Chunkserver meets the whole requirement. A Chunkserver with no disks satisfies that requirement.

warning

Do not count on a location in ON or IO with no Chunkserver registered in that location. Such a location takes no part in the count: it casts no vote and is not counted among the locations. A three-site cluster whose third site is empty therefore elects its Leader like a two-site cluster, and the default location decides a split between the two sites that hold data. The cluster does not have the arbiter that it appears to have. The screen gives no warning of that, because a location with nothing registered has nothing disconnected either, so its health reads ALL CHUNKSERVERS. Register a Chunkserver in the site before you rely on its vote. Switch the location OFF while the site is empty. Up to 5.1.0, such a location voted in favor unconditionally, on both sides of a split, so a split could leave both sites over the threshold at once.

The count follows the sites that are in the cluster, not the sites that exist. A location switched OFF leaves the cluster and is no longer counted. A three-site cluster with one site OFF then elects a Leader like a two-site cluster. The tie that the default location breaks returns. When a maintenance window takes a whole site out of a three-site cluster, the cluster therefore returns to the two-site design and its tie. This state lasts for as long as the window lasts. Plan for this before the window rather than discovering it during the window.

A Chunkserver without disks adds only a vote

A Chunkserver started with no disk space defined in mfshdd.cfg registers like any other Chunkserver and works as a voter only. That Chunkserver adds no capacity and holds no chunks, so the Chunkserver adds nothing to the redundancy of the data. The copies still need the Chunkservers that hold data. The voter changes the count inside its own location. A location's vote follows the majority of its own Chunkservers. Where the number of data-holding Chunkservers is even, one voter makes the count odd. After a split inside the site, the half that keeps the voter keeps the location's vote. Place the voter in the part of the installation that should win that split. Any small server can act as a voter. Hardware for Chunkservers covers what such a machine needs.