Leader-Follower synchronization

A MooseFS PRO instance runs several Master Servers, one Leader and the rest Followers. Every Follower keeps a live copy of the Leader's metadata. This page describes the path that keeps those copies current: what the Leader sends to a Follower, what the Follower does with that data, and what the copy is for. Leader election and quorum defines the roles themselves.

A Follower gets the Metaloggers' changelog stream

A Follower connects to the Leader's metadata port, the same port that the Metaloggers use, and registers there as a Master Server. Network ports lists the port. From then on the Follower receives the changelog stream. For every metadata operation, the Leader formats one changelog line. The Leader sends that line in a single broadcast to every synchronized Metalogger and Follower alike. There is no separate stream for Followers. That line has the same metadata version, and the changelog on disk is made of the same lines.

The two kinds of receiver differ in what they do with the stream. A Metalogger only stores the lines on disk and holds no metadata to apply them to. The Metalogger stream describes that path.

The Follower applies every line in RAM

A Follower is the same mfsmaster program as the Leader and holds the whole metadata in RAM. The roles are states of one process, not separate installations, as the Master Server page describes. The Follower applies each line from the stream to that in-memory copy as soon as the line arrives. The copy is therefore a running replica, not an archive that waits to be replayed after a failure.

The Follower checks each line while it applies the line. A changelog line includes the identifier that the operation produced on the Leader: an inode number or a chunk ID. The Follower compares that identifier with the identifier that its own replay produced. The changelog page explains why the format includes the identifier. A Follower that cannot apply a line, or whose replay does not match the line, stops relying on its copy and desynchronizes. The Follower then takes the refill path described below.

The Follower also keeps its own recovery record on disk. The Follower appends every applied line to its own changelog files, rotates those files when the Leader announces a rotation, and writes a metadata dump when the Leader orders one. The schedule of those orders is described in the save cycle. The save states covers what each server reports about its saves.

A Follower never creates a change. The Follower accepts no metadata operation of its own, adds no line of its own to the stream, and opens no client session. A client that reaches a Follower is given the Leader's address instead, as the quorum page describes.

A Follower far behind is refilled, not patched

At registration the Follower reports the metadata version that it holds. The Leader compares that version with a window of recent changes that the Leader keeps in memory. A gap within the window is closed on the stream itself: the Leader resends the missing lines first and treats the Follower as delayed until it has caught up. A gap beyond the window desynchronizes the Follower: the Follower discards the copy that it holds and drops the connection. The Follower then reconnects, reports version zero, receives a complete copy of the Leader's metadata over that second connection, and reconnects once more to join the stream. A Master Server holds no metadata at all when it starts, so a new Follower takes the same path, starting from the second step. The labels that a Follower has (INIT, DESYNC, DELAYED, SYNCED) and the transitions between them are covered in leader election.

The same connection also carries an audit in the other direction: a Follower reports the checksum of its metadata. The Leader compares that checksum with its own on a schedule, and orders a Follower whose copy has diverged to desynchronize and refill. Metadata ID, version, delay, and checksum describes that comparison and what it detects.

The copy lets the instance survive its Leader

A synchronized Follower is a candidate for the Leader role. When the Leader fails, the election gives the role to one of the synchronized Followers, under the majority rule that leader election and quorum defines. The instance then continues with the copy that this path has kept current. Changelogs, dumps, and the recovery point covers the recovery point of the whole arrangement: how much work a sudden failure can cost.

Once the copy is in place, the path also carries traffic in the opposite direction. On its regular schedule, the Leader normally leaves the dump to its Followers. The Leader orders every synchronized Follower to store one and writes none of its own. The Leader writes dumps itself in two cases. The Leader writes a dump of its own on the schedule that compares checksums. The Leader also falls back to saving as a single Master Server would whenever no Follower is synchronized, and it reports this in its log. The Leader can also download a stored dump back from one Follower over the same connection instead of producing its own. This behavior is off by default. The save cycle covers that arrangement. The save states list what each side reports afterwards.