When locations help and when they do not

A cluster that spans more than one site can run undivided, with the sites as entries on a network map. Such a cluster can also be divided into locations, with each site keeping redundancy of its own. The literature calls the divided design multi-site or geo-replication. This page covers the choice between the two designs.

Locations divide the cluster, but topology only measures it

Both mechanisms are set up by mapping IP addresses. They answer different questions. Topology reduces the network to distances. Topology decides ordering: which copy a client reads first and which servers a repair prefers. Where a class or a cluster-wide setting requires it, topology also keeps copies of a chunk out of a single rack. Topology never divides the cluster. Locations divide the cluster into segments. Redundancy is maintained in each segment separately. The leader-election majority is counted location by location. A design that only requires reads to be served from nearby needs topology. A design in which a site must work on its own needs locations. One cluster can use both.

A module's IP address decides which location the module belongs to. You map ranges of addresses. A module that is not mapped belongs to the default location. The Location page covers the mapping, the states that a location can be in, and the limit on the number of locations in a cluster.

Division is for sites that must survive each other

A Storage Class defines placement per location. A class that requires a full set of copies in each site therefore keeps both sites complete on their own. One site's definition is never satisfied with the other site's space. Losing a whole data center then costs no data. The sections below answer a separate question: whether the surviving site can also keep a Leader.

Reads follow the division. The Master Server answers a client with the copies in the client's own location ahead of the rest. A site that holds a copy therefore serves its own reads (the order a client is handed covers the order in full). A backup site also does not have to store everything, because placement is defined per location and per class. A class for essential data mirrors that data to the disaster-recovery site. A class for scratch data stores none of that data in that site.

With redundancy per location, capacity and the number of servers are worked out once per location, not once for the cluster.

The division does not shorten the link

Locations change where data is required to be, not what moving it there costs. Every copy meant for the remote site is sent over the link between the sites, so the link's latency is added to writes. The cost of writes across sites covers what a write costs and when that cost rules out a stretched cluster.

Dividing does not create an arbiter either. The majority is counted over locations. An even number of sites can split the vote in half. A tie resolves in the default location's favor, so the sites are not equal. Quorum across sites describes what two, three, and more locations result in, before you commit to a number.

Locations are also not available everywhere as a fallback. The Community Edition runs with its single default location and cannot add a second location. A design that needs the division is a design for MooseFS PRO.