The Metalogger stream

One connection carries everything that a Metalogger receives: a registration, a live feed of changelog lines, and a periodic download of the metadata dump. The component page describes the Metalogger process and the files that it keeps. This page describes what is sent over the connection and what happens when the connection breaks.

Registration opens the stream

The Metalogger connects to the Master Server's port for metadata receivers, listed in the network ports table, and registers with its version. When the Metalogger still holds its position in the stream, after a reconnection or after a restart next to its own files, the registration also includes a changelog version. The Metalogger asks the Master Server to continue the feed from that version. A Master Server that requires an authentication code answers with a 32-byte random challenge instead of accepting the registration. The Metalogger then repeats the registration with a SHA-256 digest of the challenge, keyed with the Metalogger's own AUTH_CODE. The Master Server compares digests, so the code itself is never sent over the network. Authentication between modules covers the exchange for every module pair.

A Follower Master Server PRO (PRO Edition only) registers on the same port. The Master Server tells a Metalogger and a Follower apart by the registration type, not by the port. The changelog feed below is one broadcast that both receive. Leader-Follower synchronization describes what a Follower does with that feed and what else its connection carries.

Every changelog line is broadcast as written

Each metadata operation gets the next metadata version number and one changelog line. The routine that records the line in the Master Server's own changelog also sends a copy, with its version number, to every synchronized Metalogger. The stream is therefore the same record that the Master Server keeps for itself. Changelogs, dumps, and the recovery point covers what that record is and when it rotates.

The Metalogger appends each line to changelog_ml.0.mfs through a buffered writer and flushes the file once per second. The Metalogger does not force the file to disk, so the copy is behind the feed by up to a second of buffering. Rotation is signaled, not inferred. The Master Server sends a rotation marker over the stream when it rotates its own changelog. The Metalogger then renames its files to match.

The dump downloads on schedule and at start

Every META_DOWNLOAD_FREQ hours the Metalogger downloads the newest metadata dump that the Master Server has stored on its own disk. The Master Server does not write a new dump for this download. A frequency set higher than half of BACK_LOGS is lowered to that half, so a dump is always younger than the oldest changelog file kept with it. A Metalogger that starts without a position in the stream registers, receives the feed from the moment of registration, and downloads a dump immediately. A start has no position when the data directory holds no metadata_ml.mfs.back, or when changelog_ml.0.mfs holds no complete line. Since 5.2.0, a Metalogger that starts next to both files reads its position from the last line of changelog_ml.0.mfs and registers like a reconnecting Metalogger. It downloads a dump only if the feed then shows a gap. Up to 5.1.0, every start downloaded a dump.

The Metalogger pulls the dump in blocks. The Metalogger requests each block by offset, up to 1000000 bytes at a time. For each block, the Metalogger writes the block, verifies the checksum that comes with the block, and syncs the block to disk. The Metalogger retries a failing block up to five times. The dump is written to a temporary file. That file replaces the previous dump only after its header and end marker are verified, so an interrupted download never damages the dump that the Metalogger already holds. After each dump download, the Metalogger pulls the Master Server's two newest changelog files over the same block protocol. The component page lists the files that all of this is written to.

A break is repaired from memory or a fresh dump

A break is detected by silence. Each side sends a keepalive after a second without traffic. The Metalogger drops a connection that has no traffic for MASTER_TIMEOUT seconds. The Metalogger retries a lost connection every MASTER_RECONNECTION_DELAY seconds. Lines produced during the break are not lost. The Master Server keeps its recent changelog lines in memory, as many seconds of them as CHANGELOG_PRESERVE_SECONDS sets, within the memory that CHANGELOG_PRESERVE_MB allows. A reconnecting Metalogger requests the stream starting from the line after the last line that it holds. When the memory still holds lines from that point, the Master Server replays the missed lines and switches to the live feed. The break then costs nothing.

When the memory no longer holds lines from that point, the feed restarts from the current point. The Metalogger detects the gap in the version numbers at the first line, discards all of its changelog_ml.*.mfs files, and reconnects like a Metalogger that starts without a position: a new dump, then the feed. The Metalogger's record then restarts at that dump instead of keeping a gap in the middle.

A change of Leader PRO (PRO Edition only) breaks the stream the same way. The component page describes how the Metalogger finds the new Leader when it registers.

The files exist for a lost Master Server

Everything that the stream maintains is kept for the case in which the Master Server is lost together with its metadata. A replacement Master Server rebuilds the metadata from the dump and the changelog files on the Metalogger's disk, as the Metalogger component page describes.

The settings below decide what this path does

Setting	Set on	Default	What it decides here
AUTH_CODE	the Metalogger	none	The key the Metalogger answers the Master Server's challenge with. When no key is set, the Metalogger cannot register with a Master Server that asks for one.
META_DOWNLOAD_FREQ	the Metalogger	24	Hours between dump downloads. A value above half of BACK_LOGS is lowered to that half.
BACK_LOGS	the Metalogger	50	Changelog files kept beside the dump, and the ceiling the download frequency is measured against.
MASTER_TIMEOUT	the Metalogger	10	Seconds of silence after which the Metalogger drops the connection.
MASTER_RECONNECTION_DELAY	the Metalogger	5	Seconds between attempts to reconnect.
CHANGELOG_PRESERVE_SECONDS	the Master Server	5000	Seconds of changelog lines the Master Server keeps in memory for a receiver that reconnects. A value above 100000 is lowered to 100000.
CHANGELOG_PRESERVE_MB	the Master Server	500	Memory those lines may occupy. The Master Server keeps whichever of the two figures comes out smaller.