Prometheus metrics
MooseFS reports the state of a cluster to Prometheus through an exporter that belongs to the GUI server rather than to a daemon of its own. The GUI server answers one more HTTP path. Every request to that path runs mfscli in its JSON mode and flattens the answer into metrics. The exporter keeps nothing between requests. Each scrape queries the cluster again and returns the figures that mfscli prints at that moment.
The exporter keeps no cache, so every scrape costs the cluster the full collection. Two settings decide that cost: how often Prometheus scrapes, and how much each scrape requests. Both are set in the scrape job. On a large cluster, a frequent scrape of everything is monitoring traffic that competes with the work that the cluster exists to do. For this reason, narrow what a job collects, and scrape it no more often than the readings are used.
The exporter answers on the GUI port
The address is the GUI address followed by /metrics. For example, a cluster whose GUI is at http://192.0.2.10:9425/ exports its metrics at http://192.0.2.10:9425/metrics. The port is the GUI port: 9425 in the packages, and GUISERV_LISTEN_PORT in mfsgui.cfg once the port has been changed (network ports).
The exporter has no switch of its own. The path is listed in the request table that the GUI server ships with. That table is a file that is not meant to be edited, so a running GUI server answers on /metrics. Two components have to be installed on that host: the GUI server, and mfscli, which the GUI server runs to collect the data.
The collection runs on the host of the GUI server. A scrape therefore connects to the cluster exactly as the GUI does: to every Master Server, and to every connected Chunkserver for disk figures and Chunkserver charts. A host that runs the GUI server needs to reach the ports of both the Master Servers and the Chunkservers. The GUI covers those ports together with where that server may run.
The path is matched exactly, so /metrics/ and /Metrics do not reach the exporter. The exporter answers GET requests and returns text/plain. A HEAD request returns 405 Method Not Allowed.
Metrics are served through the GUI port. The GUI server has no login of its own. Whatever protects that port therefore also protects the metrics: the GUI is guarded by where it listens.
A scrape job names the endpoint and its parameters
scrape_configs:
- job_name: moosefs
scrape_interval: 60s
scrape_timeout: 30s
metrics_path: /metrics
static_configs:
- targets: ['192.0.2.10:9425']
params:
masterhost: ['mfsmaster']
masterport: ['9421']
scope: ['SIG,SIC,SCS,SHD']
mastercharts: ['none']
cscharts: ['none']
The params block is optional. Without it, the exporter reads the cluster through the host name mfsmaster on port 9421 and exports its default selection. The entries in this job set the address explicitly and narrow the selection.
The interval and the timeout in this job are an example. The cluster fixes neither figure. The cluster does fix the resolution of the data that the chart metrics are built from: charts are counted in one-minute buckets. The exporter reads the newest completed bucket. A scrape that runs more often than once a minute reads that same value again until the next bucket closes. The HTTP connection has its own time limit: GUISERV_TIMEOUT in mfsgui.cfg, which is 300 seconds by default.
The parameters decide what one response carries
| Parameter | Default | What it does |
|---|---|---|
masterhost | mfsmaster | Names the host the exporter reads the cluster through. |
masterport | 9421 | Names the Master Server port it connects to. |
scope | default | Selects the data sets, by their mfscli letters. |
mastercharts | all | Selects Master Server charts, by name. |
cscharts | all | Selects Chunkserver charts, by name. |
prefix_whitelist | empty, nothing is narrowed | Keeps only the metrics whose name starts with one of these prefixes. |
prefix_blacklist | the chunk matrix prefixes, always applied | Drops the metrics whose name starts with one of these prefixes. |
Each data set holds one part of what the cluster reports. Each data set also has its own family of metric names:
scope | What the data set holds | Metric names |
|---|---|---|
SIG | General cluster summary | mfs_info_general_* |
SIM | Master Server states | mfs_info_masters_* |
SLI | License information | mfs_info_licence_* |
SIC | Matrices of chunk Redundancy Levels | mfs_info_chunks_* |
SIL | Self-check loops | mfs_info_fs_loop_*, mfs_info_chunk_loop_* |
SMU | Master Server memory usage | mfs_info_memory_* |
SCS | Connected Chunkservers | mfs_chunkservers_* |
SML | Connected Metaloggers | mfs_metaloggers_* |
SHD | Disk data | mfs_disks_* |
SSC | Storage Classes | mfs_storage_classes_* |
SMO | Operation counters | mfs_operations_* |
SQU | Quota information | mfs_quotas_* |
default is the value that the exporter uses when scope is absent. This value means SIM, SLI, SIG, SIC, SIL, SCS, SML, and SHD. To widen or narrow that selection, name the data sets that you want, separated by commas. default counts only when it is the only value, so scope: ['default,SMO'] selects SMO and nothing else. A letter combination that the exporter does not recognize is dropped without a message. none switches every data set off.
mastercharts and cscharts take chart names the same way, one list for the Leader Master Server's charts and one for the charts of every connected Chunkserver. all selects every chart except cpu, which you have to name explicitly. none leaves the charts out of the response.
The values of scope, of both chart parameters, and of both prefix lists are read as letters, _, and commas. Every other character is dropped before the value is used. A chart or a prefix whose name contains a digit cannot be selected this way.
prefix_whitelist is applied first, and prefix_blacklist is then applied to what remains. The exporter has its own blacklist, which applies to every response. A prefix_blacklist that you supply is added to that list instead of replacing it. On a cluster without locations PRO (PRO Edition only), that built-in list excludes the matrix of chunk Redundancy Levels, which does not fit metric names well. The summary of the matrix is still exported.
Narrowing the selection keeps a scrape cheap. The default set queries every connected Chunkserver separately for its disks. cscharts: ['all'] reads the charts of every connected Chunkserver. The exporter caches neither result, so every scrape collects both again. Each response reports the cost of its own scrape: mfs_cli_execution_time gives the collection time. Its command label gives the exact query that produced it.
The parameters are query parameters of the same request that Prometheus makes. You can therefore fetch a narrowed selection by hand before you put it into a job:
# the readings that answer whether the cluster is healthy, and nothing else
http://192.0.2.10:9425/metrics?scope=SIG,SIM,SIC,SIL&mastercharts=none&cscharts=none
# the expensive part on its own, for a job that scrapes it less often
http://192.0.2.10:9425/metrics?scope=SCS,SHD&mastercharts=none&cscharts=none
# the default set plus operation counters, spelled out because `default` counts only when it stands alone
http://192.0.2.10:9425/metrics?scope=SIM,SLI,SIG,SIC,SIL,SCS,SML,SHD,SMO&mastercharts=none&cscharts=none
A metric name is the path, behind mfs_
Every name starts with mfs_. The rest is the position of the value in the mfscli answer, with its keys lowercased and joined with _. For example, the total space of the cluster is held under info and then general, so it is exported as mfs_info_general_total_space. Charts follow their own naming rule: a chart becomes mfs_mastercharts_ or mfs_cscharts_ followed by the name of the chart.
Every metric is a gauge. A true or false field is exported as 1 or 0. Labels show which object a value belongs to. The labels come from a fixed set of keys: ip, port, hostname, name, id, csid, path, hostname_path, sclassname, and state. A chart metric has one label instead: master or chunkserver. On a Chunkserver chart, the label names the server as <host>:<port>. The host is the name that reverse DNS returns, or the IP address when no name resolves. On a Master Server chart, the exporter asks for the charts of the Leader. The label then reads leader.
Text fields do not become metrics of their own. An object that has them gets one extra metric with the suffix _misc and the value 1. That metric has every text field of the object as a label. Formatted counterparts of numbers, the fields whose names end in _human, are not exported.
mfs_info_general_total_space 123456789
mfs_info_masters_metadata_delay{ip="192.0.2.12",state="FOLLOWER"} 0.0
mfs_info_masters_misc{ip="192.0.2.11",state="LEADER",version="5.1.0"} 1
mfs_cscharts_ucpu{chunkserver="192.0.2.21:9422"} 0.5
In the response, each of these is preceded by its # TYPE line. Where the metric has a description, the metric is also preceded by a # HELP line with that description. The unit is in brackets where the description names one. A metric that has no description is exported without that line. The descriptions themselves are on Metrics, sorted by the part of the cluster that a metric reports on. One more metric is at the top of every response: mfs_cgi_info, which has the version of the GUI server as a label.
Several locations add a segment to chunk metric names
A cluster divided into more than one location PRO (PRO Edition only) reports its chunk information per location. The key of the location becomes a segment of the metric name: all_locations for the cluster as a whole, as in mfs_info_chunks_all_locations_summary_allchunks_missing. Since 5.2.0 these names are valid Prometheus metric names. Up to 5.1.0 the key contained a space, so no name under mfs_info_chunks_ was valid. The definitions of Storage Classes (SSC) get the same kind of segment after storage_modes, one for every location, as in mfs_storage_classes_storage_modes_location_0_create_full_copies. Up to 5.1.0 that key contained a space as well.
The built-in blacklist does not cover the chunk names with the all_locations segment, so on such a cluster the exporter also returns the full matrix of chunk Redundancy Levels. Some lines of that matrix have a label list that starts with a comma, as in {,target=…}, and the Prometheus text format does not accept that. To keep those lines out of a scrape, put mfs_info_chunks_all_locations_allchunks,mfs_info_chunks_all_locations_regularchunks on prefix_blacklist. The summary of the matrix is still exported. Leaving SIC out of scope drops all chunk information instead. The other data sets are unaffected.
Every response is 200 OK, failures in the body
The body of the response says what went wrong. Its status is the same in every case. To see what a scrape receives, fetch the path yourself with the parameters of the job appended to the address:
curl 'http://192.0.2.10:9425/metrics?scope=SHD&mastercharts=none&cscharts=none'
An HTML page instead of metrics means that the GUI server did not reach the Master Server at masterhost. The page says so in its heading. A body with a Failed to parse JSON line below the mfs_cli_execution_time metric means that the collection command returned no data. The most common cause is that mastercharts or cscharts named a chart that does not exist. A chart name that contains a digit is one way to cause this. A response with mfs_cgi_info alone means that scope, mastercharts, and cscharts are all at none, so nothing was collected. A Chunkserver that the exporter could not reach is missing from the response instead of being reported in it.
The metrics become charts and alerting rules elsewhere
Grafana covers drawing these metrics. Alerting covers setting conditions on them.