Metadata ID, version, delay, and checksum

A Master Server describes the state of its metadata with four numbers. This page describes what each number is, where it comes from, what a comparison with a different copy decides, and which reading should worry you.

The metadata ID is drawn once and kept

The ID is a 64-bit number, shown as sixteen hexadecimal digits. A Master Server draws the ID the first time it holds metadata that has no ID. The ID is drawn, not configured. Nothing afterwards changes it.

The draw is recorded as a metadata change like any other. Anything that replays changes therefore ends up with the same number. A Metalogger and a Follower PRO (PRO Edition only) both do. Every metadata dump has the ID in its header. So does the checksum file that is written beside the dump.

A client receives the ID when it registers. The client presents that ID whenever it names its session again. A client that asks to resume an earlier session under the wrong ID gets a new session instead of the session that it named. A client that reconnects to a session that it still holds is refused outright when the ID is wrong.

A Follower that holds no metadata of its own adopts the ID of the Leader that it connects to. After the Follower holds metadata, the number is fixed. If a Leader answers with a different ID, the Follower logs both values and drops the connection instead of synchronizing. A replayed change that sets a different ID is rejected. The rejection puts the Follower into DESYNC.

A version marks a point in the history that an ID names

A version that is behind the version of a different copy is the ordinary state of anything that follows a stream of changes. The changes that such a copy is missing exist and are numbered. Applying them closes the gap. Nothing is lost and nothing has to be decided.

A different ID means that the two sets were drawn separately and never shared a history. No sequence of changes turns one set into the other set. Comparing their versions means nothing, because the same number counts unrelated things in each set. To bring the sets together, you must choose which set to keep and which set to discard. The cluster never makes that choice on its own. A metadata dump that has a foreign ID is therefore refused at startup rather than merged.

The version counts and numbers applied changes

The metadata version is a 64-bit counter. Every change raises it by one. The change is written to the changelog under the number that it was given. That number orders a dump and a stream of changelog lines in one sequence (see Changelogs, dumps, and the recovery point). The counter is stored in the header of each metadata dump and in the header of the checksum file. Every Master Server reports its current value.

The counter also changes when nothing is happening. After a quiet period, the Master Server logs an IDLE() entry. This entry raises the version and stamps it with the current second. The quiet period is half a second on a Leader PRO (PRO Edition only), and thirty seconds in the Community Edition. An idle cluster therefore keeps producing numbered, timestamped changes. A copy that has stopped receiving them is visible immediately rather than at the next write.

A Follower becomes synchronized again by comparing versions. The Follower sends its own version when it registers. If the Leader still holds the changes from that version onward, it answers with them. Otherwise, it refuses and lets the Follower download a whole dump. From then on, each change has its number. The Follower accepts a change only if the number matches its current version. After the Follower applies the change, the version must have increased by exactly one. A number out of place, an unparsable line, or an operation that fails is not skipped. Each of them puts the Follower into DESYNC.

The delay says how far behind the Leader

Only a Follower PRO (PRO Edition only) has this reading. A Leader cannot be behind anything, so it shows no value. The reading has two forms: one kept by the Master Server itself and one computed by the tools.

The coarse form is a value that the Master Server keeps next to its state. The tools show that value as a suffix on the state name. Leader election, quorum, and the role of DNS explains what the labels mean for the role. SYNCED and DELAYED are decided for each change. The timestamp in the change is compared with the Follower's own clock. A difference of one second or less is SYNCED. More than that is DELAYED. A synchronized Follower has no suffix at all. A rejected change or an order from the Leader puts the Follower in DESYNC. Applying more changes does not take the Follower out of DESYNC. The Follower discards its metadata, reconnects, and fetches a new dump.

The precise form appears in seconds in the delay column of the Current metadata group in the GUI, and in the metadata delay column of mfscli -SIM. The value is the Leader's clock minus the timestamp of the last change that the Follower applied. The tools also subtract one second of transfer time and show no value below zero. Both forms compare the clock of one machine with a timestamp from a different machine. Clocks that disagree therefore appear here as delay. The cluster raises its own alerts on the clock difference as well.

A per-section checksum names the part that diverged

Every metadata save writes a second, small file next to the dump. That file starts with the version and the ID that the dump was written under. It then holds one entry per section. Each entry has the four-character name of the section and a 32-bit checksum of the bytes of that section and nothing else. The file has an entry for every kind of record the metadata holds, and one entry each for the header and the end marker of the dump. A single number for the whole dump would show that two copies disagree. A table shows where they disagree. The checksum column of the Last metadata save group in the GUI and last save checksum in mfscli -SIM show one value calculated from that whole table.

The polynomial of these checksums is the standard 32-bit one. The values still cannot be reproduced from the bytes of the dump. The entry of a section is the XOR of one checksum per write that built the section, so it depends on where the writer split its writes. A value here is therefore meant for comparison with a different Master Server rather than for reproduction by an outside tool.

The Leader orders a mismatching Follower to resynchronize

The comparison runs between Master Servers PRO (PRO Edition only). On the schedule set by METADATA_SAVE_FREQ, the Leader has every synchronized Follower save its metadata. On the METADATA_CRCCHECK_FREQ schedule, the Leader also saves its own metadata at the same time. The Leader waits until every table has arrived before it compares anything. Setting both METADATA_CRCCHECK_FREQ and METADATA_DOWNLOAD_FREQ to zero does not turn off the check. The Master Server then falls back to running the check as often as it saves.

Two tables are only comparable when they were written under the same version and the same ID. When they were not, the Leader reports this in its log. Comparable tables that differ show a divergence. The Leader names every section whose checksum does not match and orders that Follower to desync. The Follower then discards its metadata and fetches a new dump.

When the Leader's own table is older than the table of a Follower, the Leader leaves out its own table and compares the Followers with each other. That comparison orders no Follower to desync. If two Followers differ, the Leader schedules a new comparison with its own table rather than acting on the older data.

Two copies of metadata cannot drift apart unnoticed

Three checks stop a divergence from going unnoticed. Each of them catches something different.

The numbering catches a lost, duplicated, or reordered change, because anything that replays the stream applies only the change that follows its current version.

The data the change itself carries catches a divergence that is correctly numbered but wrongly applied. The replaying side gets the same identifier on its own. It allocates the identifier for a new object or looks it up for an existing one. Where the two identifiers differ, the replay stops instead of applying every later change to the wrong object.

The checksum catches what neither of those checks detects: a value that both sides accepted and stored differently. Examples are a misread attribute, or a field calculated by code that has a bug on one side. The change stream does not reveal such a difference, so the whole metadata is compared on a schedule PRO (PRO Edition only).

Alerts cover the ID, state, delay, and checksum

The cluster evaluates each of these readings against a threshold PRO (PRO Edition only). The alerts reference lists the message and the suggested fix for each reading.

A Follower whose ID differs from the Leader's is an error. It is the most serious reading of the four. A Follower in DESYNC is a warning. This state is expected for a short time after a restart or an election. Investigate it when it persists. A delay of more than 1 second is a warning. More than 6 seconds is an error. A last-saved checksum that differs from that of a different Master Server is an error when both servers saved the same version. A last-saved version that is behind that of a different Master Server is only information. It is the normal state of a Leader, because a Leader saves less often than its Followers.

The version by itself is not one of these thresholds. The tools query each Master Server separately. Two versions read a moment apart therefore differ by whatever the cluster did in between and by whatever the network added. Use the delay, not the difference between two counters, to see how far behind a Follower is.

The settings behind the comparison

Setting PRO (PRO Edition only)	Default	What it decides here
METADATA_CRCCHECK_FREQ	24	Hours between the Leader's own saves, which are the rounds whose comparison includes the Leader's table.
METADATA_DOWNLOAD_FREQ	0	Hours between the Leader downloading a dump from a Follower instead of saving one of its own. 0 means that the Leader never downloads.
METADATA_SAVE_FREQ	1	Hours between saves, which is the schedule the Leader orders its Followers on, and the schedule the comparison falls back to when the other two are both zero.

All three are set on the Master Server, in mfsmaster.cfg.