The WAN cost of writes

In a cluster divided into locations, some writes are sent over the wide area network (WAN) link between the sites. The client waits for those writes. The kind of operation decides which writes those are, not any setting. A new chunk goes where its Storage Class places new chunks. A modification goes to every copy that the chunk already has, wherever those copies are stored. This page describes what that means for the link that you have to provide. When locations help covers whether to divide the cluster at all.

Creating and modifying pay differently

A class has a separate placement definition for each storage state. A new chunk is created on the copies that its CREATE definition names, and only on those. A write into a chunk that already exists reaches every copy of that chunk. The KEEP definition maintains that set of copies. The application's operations therefore decide what crosses the link. Writing new files, or extending a file by a further chunk, creates chunks in the places that CREATE names. Changing the content of an existing chunk is a modification. Appending inside a file's last chunk is also a modification. A log file that grows all day modifies one chunk most of the time rather than creating chunks.

A modification waits for its farthest copy

The write path covers the mechanism. The client sends the data once, to the first Chunkserver of a chain. The Chunkservers forward the data along the chain to every copy. A block is acknowledged to the client only when every server in the chain has accepted that block. There is no primary copy that updates the remaining copies later. The copies in the writer's own location are first in the chain (topology mapping and distance describes the order). The client's own uplink therefore stays inside the client's site. But when the chunk has a copy in the far location, the chain runs through the link. Each block is then confirmed only after the far site's acknowledgment returns over the link. The link's round-trip time is part of every such write, block by block. Bandwidth does not make up for that time.

New data can stay local, at a price

A class can name only the writing site's Chunkservers in CREATE and both sites in KEEP. New chunks are then created and confirmed without using the link. The Master Server reviews each location against its own definition (location) and finds that the far location is short of copies. The Master Server replicates the chunks there afterwards, Chunkserver to Chunkserver across the link.

Between the client's confirmation and that replication, the new data exists in one site only. If a site is lost inside that window, every copy of the data written in that site is lost with the site. This pattern therefore does not give zero data loss across sites. The window has no fixed length. The window closes at the pace of replication. Replication shares the link with everything else on the link. Data that must survive the loss of the writing site from the moment that data is confirmed needs CREATE copies on both sites. Every write of new data then costs the link's round trip, like a modification does.

What the link must carry

• The modification stream. Every write into chunks that have copies on both sites crosses the link at the moment of the write.
• Replication. Each copy that the far site is to hold arrives as a separate transfer across the link. Such transfers carry the new chunks created locally under the pattern above and every copy rebuilt there after a failure.
• Metadata operations of the far site. There is one Leader in the cluster. A Follower opens no client session (Leader-Follower synchronization). Clients in the site without the Leader therefore cross the link for every metadata operation. These operations include the grant that starts each write and the report that closes the write. Quorum across sites covers where the Master Servers themselves should be placed.
• The changelog stream. Followers and Metaloggers in the far site receive every metadata change from the Leader over the same link.

The first two items size the link's bandwidth against the write and repair load. The last two items add the link's round-trip time to every operation of the far site. MooseFS sets no minimum bandwidth and no maximum round-trip time for the link. The link's needs follow from the workload's write rate and from how much of the data has copies on both sides. Timeouts are used instead of a threshold. A transfer that stops sending counts as a failed attempt and falls back to the client's retry budget (client background work). A Master Server drops any connection that stays idle past the Master Server's own timeout. This includes the connection to a Follower in the far site (network problems). One replication transfer has a fixed time to finish. A link that is too slow to move one chunk inside that time therefore cannot complete the copy at all (replication priorities). Budgets inside one site are covered in bandwidth and latency.