Alerting

A MooseFS cluster evaluates its own state and shows the result on screen. The screen covers the hours when somebody is watching. This page covers the rest: rules that run outside the cluster, on the metrics it exports, and reach a person who is not watching.

The cluster's own alerts stop at the GUI

The GUI evaluates a fixed set of conditions each time it renders a page. The GUI then marks the object that each condition concerns: a Chunkserver row, a disk, a Master Server, the chunk matrix summary. Each alert has a severity of Error, Warning, or Info. Each alert also comes with its message, its explanation, and the fix that the cluster suggests. All alerts, with their severities, are in Alerts.

The evaluation is part of rendering, so an alert exists while the page is on somebody's screen. The GUI computes the alert again from scratch on the next request. That makes the alert a diagnosis rather than a notification. The rest of this page is therefore about a rule engine that runs outside the cluster.

The rules read exported metrics and inherit their shape

These rules run on the Prometheus side, on the metrics that the exporter returns. Prometheus metrics covers how to scrape them. Metrics covers what each name reports. The shape of that data decides how you write a rule:

Every exported value is a gauge, the counters included, so rate() and increase() have nothing to work on.
When the cluster has no Leader, most of the tables that the exporter reads are not produced at all. Their series then stop being printed instead of falling to zero. Write a condition that has to work in that case with absent(), not with == 0.
Every metric that these rules read is in the set that the exporter returns with no scope parameter. The scrape job therefore does not need to be widened for any of them.
On a cluster with more than one location PRO (PRO Edition only) the mfs_info_chunks_ names, the matrix summary included, have an extra segment, all_locations. The chunk rules below then stop firing. Prometheus metrics covers the names and the prefix list that keeps such a scrape valid.

The numbers in these rules do not all have the same source. A comparison threshold called the cluster's own is the value that the cluster itself uses when it raises the matching alert. Changing that threshold makes the rule disagree with the GUI. Every threshold called an example is a decision you make: no value for it exists in the product. The same applies to every for: window except one, which is named where it occurs. The right value depends on the size of the cluster and on how long a restart takes there. The value also depends on how much repair work you are willing to let run before somebody hears about it. The severity label is a routing key for the receiver, not something the cluster reports.

Every block on this page is a complete rule file.

Missing and endangered chunks cost data

groups:
  - name: moosefs-data
    rules:
      - alert: MooseFSChunksMissing
        expr: mfs_info_chunks_summary_allchunks_missing > 0
        for: 5m
        labels:
          severity: critical
      - alert: MooseFSChunksEndangered
        expr: mfs_info_chunks_summary_allchunks_endangered > 0
        for: 15m
        labels:
          severity: critical
      - alert: MooseFSChunksUndergoal
        expr: mfs_info_chunks_summary_allchunks_undergoal > 0
        for: 1h
        labels:
          severity: warning

MooseFSChunksMissing. Chunks that the cluster cannot read from anywhere. Any value above zero is the cluster's own error condition, so there is no threshold to choose. Start at cluster health problems. The window is an example. The window exists because the matrix counters are not valid while Chunkservers are connecting or disconnecting, which happens right after a restart.
MooseFSChunksEndangered. Chunks with no redundancy left, one failure away from being unreadable. Any value above zero is the cluster's own error condition. Self-healing covers what the state counts and what the cluster does about it on its own. The example window gives that repair time to finish. Base the window on how long repair takes on your cluster.
MooseFSChunksUndergoal. Redundancy below what the Storage Class requires, with more than one copy or part still readable. The cluster treats any value above zero as a warning. A count that keeps rising, or a count that never falls, is the reading that matters, so the example window is long. The reasons why a count stops falling are in self-healing.

Missing, forgotten, and overloaded Chunkservers need different windows

groups:
  - name: moosefs-chunkservers
    rules:
      - alert: MooseFSChunkserverDown
        expr: >-
          mfs_chunkservers_connected == 0
          unless on (ip, port) mfs_chunkservers_misc{maintenance_mode=~"on.*"}
        for: 30m
        labels:
          severity: critical
      - alert: MooseFSChunkserverQuorumAtRisk
        expr: count(mfs_chunkservers_connected == 0) * 2 >= count(mfs_chunkservers_connected)
        for: 1m
        labels:
          severity: critical
      - alert: MooseFSMaintenanceModeLeftOn
        expr: mfs_chunkservers_misc{maintenance_mode=~"on.*"}
        for: 4h
        labels:
          severity: warning
      - alert: MooseFSChunkserverOverloaded
        expr: mfs_chunkservers_queue_state == 1
        for: 15m
        labels:
          severity: warning

MooseFSChunkserverDown. A Chunkserver that the Master Server has registered and cannot reach. The unless clause drops the servers that a planned stop has left in maintenance mode. MooseFSMaintenanceModeLeftOn watches those servers instead. The window is a decision: 30 minutes matches CS_TEMP_MAINTENANCE_MODE_TIMEOUT in mfsmaster.cfg. At that point the cluster itself stops treating a controlled stop as planned. A cluster whose servers reboot slower than that needs a longer window. The fix is in Chunkserver and disk recovery.
MooseFSChunkserverQuorumAtRisk PRO (PRO Edition only). A Leader keeps its role only while it can reach more than half of the Chunkservers. The Leader loses the vote once the disconnected Chunkservers reach half of the total. On an even count, that is exactly half, not half plus one. The threshold is the cluster's own. On a cluster divided into locations, the count is made per location. The mfs_chunkservers_connected series does not have the location as a label, so this form of the rule is valid for a single-location cluster. Go to cluster health problems.
MooseFSMaintenanceModeLeftOn. Maintenance mode set by hand does not clear when the server reconnects. Unless somebody set an expiry, the mode does not clear on its own either. The window is an example. This rule detects the server that nobody came back to.
MooseFSChunkserverOverloaded. The value 1 is the state that the cluster reports when a Chunkserver's request queue is heavily loaded. The cluster raises a warning on that state. The window is an example: an overload that lasts through a burst of writes is ordinary. An overload that continues after the burst is not.

Disks report damage, the cluster reports fullness

groups:
  - name: moosefs-disks
    rules:
      - alert: MooseFSDiskDamaged
        expr: mfs_disks_misc{status_str=~".*damaged.*"}
        labels:
          severity: critical
      - alert: MooseFSDiskFull
        expr: mfs_disks_used_percent >= 99
        for: 15m
        labels:
          severity: critical
      - alert: MooseFSDiskNearlyFull
        expr: mfs_disks_used_percent >= 90
        for: 15m
        labels:
          severity: warning
      - alert: MooseFSChunkserverFull
        expr: mfs_chunkservers_hdd_regular_used_percent >= 99
        for: 15m
        labels:
          severity: critical

MooseFSDiskDamaged. A Chunkserver has taken a disk out of service after too many failed reads and writes. The counting that produces that result is in disk error handling and disk removal. That counting already includes a tolerance period of its own, so the rule needs no window. The fix is in Chunkserver and disk recovery.
MooseFSDiskFull and MooseFSDiskNearlyFull. Both thresholds are the cluster's own: 99 percent used is the error that the cluster raises on a disk, and 90 percent the warning. The windows are examples. They keep a temporary peak from firing the rule. Space management covers what to do about the space.
MooseFSChunkserverFull. The same 99 percent, read across the regular space of a whole server. At that point the reading is no longer a disk problem. It becomes a placement decision that the Master Server makes about the whole cluster. Cluster health parameters describes that decision.

Metadata rules watch the Leader and the lag

groups:
  - name: moosefs-metadata
    rules:
      - alert: MooseFSNoLeader
        expr: absent(mfs_info_masters_metadata_version{state=~"LEADER|MASTER"})
        for: 1m
        labels:
          severity: critical
      - alert: MooseFSMetadataOutOfSync
        expr: mfs_info_masters_metadata_delay > 6
        for: 5m
        labels:
          severity: critical
      - alert: MooseFSMetadataDelayed
        expr: mfs_info_masters_metadata_delay > 1
        for: 5m
        labels:
          severity: warning
      - alert: MooseFSMetadataSaveTooLong
        expr: mfs_info_masters_last_metadata_save_duration > 2700
        labels:
          severity: warning

MooseFSNoLeader. No Master Server reports itself as LEADER, nor as MASTER, which is the state that a lone Community Edition server reports. One minute is the figure that the cluster's own alert text advises. Investigate an election that has not finished by then instead of waiting for it. Go to Master Server problems.
MooseFSMetadataOutOfSync and MooseFSMetadataDelayed PRO (PRO Edition only). How far a Follower's metadata is behind the Leader's, in seconds. Six seconds and one second are the cluster's own thresholds, the first an error and the second a warning. The windows are examples. Both rules point at the same places to look: the clocks and the load on the Follower. The fix is in Master Server problems.
MooseFSMetadataSaveTooLong. How long the last completed metadata dump took, in seconds. The cluster's own threshold is 45 minutes, which is the 2700 in the expression. The rule needs no window: the value describes a save that has already finished, so the value does not flap.

Some things worth alerting on have no metric

Whether the components are running. The exporter describes the cluster that the Master Server has registered. A component that never registered is therefore not a series that fell to zero. The up series that Prometheus writes for every job covers the scrape target itself. A host exporter covers the processes on each machine.
Free space where the Master Server writes metadata. A Master Server that cannot write its metadata dump into DATA_PATH writes it to a fallback location and exits. No MooseFS metric includes the free space of that file system. A host exporter that watches the Master Server machines covers that free space.

The receiver handles a firing alert

Prometheus loads these files through its rule_files setting. Prometheus passes a firing alert to the receiver that it is configured with, which is usually Alertmanager. Grouping, silencing, repeat intervals, and the notification channels themselves are settings of that receiver. The documentation of the receiver describes them. The MooseFS side of the arrangement ends with the metric names and the thresholds in these rules.

The cluster's own alerts stop at the GUI​

The rules read exported metrics and inherit their shape​

Missing and endangered chunks cost data​

Missing, forgotten, and overloaded Chunkservers need different windows​

Disks report damage, the cluster reports fullness​

Metadata rules watch the Leader and the lag​

Some things worth alerting on have no metric​

The receiver handles a firing alert​