Skip to main content

Grafana

Grafana never connects to MooseFS. Grafana queries Prometheus instead. Prometheus scrapes the MooseFS metrics exporter. A panel is therefore only as good as the scrape that feeds it. The job, its parameters, and what one scrape costs the cluster are in Prometheus metrics.

Grafana needs one thing from you before any of that data is visible: a data source of type Prometheus, which points at the Prometheus server that stores the scraped metrics. You add a data source through a form in Grafana's own interface. Grafana's documentation covers that form. Everything below is a query that you write against that data source.

MooseFS ships no dashboards​

There is no dashboard file to import. The dashboard is yours to build. You build it from the exporter's output. Metrics lists the names with what each name reports and its unit. Prometheus metrics describes how a name is put together, so you can read a name that appears on a live target but not in that list.

If the cluster has more than one location, resolve the chunk metrics before you build panels on them. Prometheus metrics covers what happens to their names, and how to keep them out of a scrape.

Which readings earn a panel​

What to watch covers which readings matter, and what a bad value means. The sections below cover which of those readings a graph answers better than a table does, and how each reading behaves as a series once it reaches Grafana. Every metric named here is in the exporter's default parameter set. A scrape job that passes no parameters therefore already collects all of them.

A capacity panel shows the fill rate​

mfs_info_general_total_space and mfs_info_general_avail_space are cluster-wide single series. Two lines on one time-series panel answer the question that a capacity panel exists for. The question is how fast the cluster is filling and how much time that leaves, not how full the cluster is today. Any table shows the current fill faster. Raw space and user space covers what those two figures include.

mfs_disks_used_percent gives the same reading per disk, labeled with that disk's path, hostname, ip, and port. A second panel on those series shows whether the free space is spread across the cluster or concentrated on a few servers.

Chunk state takes two panels​

mfs_info_chunks_summary_allchunks_missing and mfs_info_chunks_summary_allchunks_endangered are at zero in a healthy cluster. You need to see the moment either of them is not. Use a stat panel for them, not a line at the bottom of a graph.

mfs_info_chunks_summary_allchunks_undergoal is the opposite case and needs the graph. Its shape over an hour, read beside mfs_info_chunks_summary_allchunks_stable, shows whether the cluster is repairing or stuck. A single reading shows neither.

Chunkserver panels name the server that dropped​

mfs_chunkservers_connected reports 1 or 0 for every Chunkserver that the Master Server knows about, a disconnected Chunkserver included. One panel therefore shows both how many Chunkservers should be there and how many are. The labels hostname, ip, port, and csid name the server that has disconnected.

mfs_chunkservers_errors and mfs_chunkservers_queue belong on the same row. They show a server that is failing or falling behind while it is still connected. No count of connections shows that.

Master Server operations show what the workload asks​

mfs_mastercharts_lookup, mfs_mastercharts_getattr, mfs_mastercharts_open, mfs_mastercharts_createobj, mfs_mastercharts_unlink, and mfs_mastercharts_rename make up the panel to check when someone reports that the file system got slow and nothing on the cluster changed. Those metrics show the shape of the load. The shape changes when an application starts behaving differently.

These values come from the Master Server's own charts. Each sample in those charts covers one minute instead of accumulating since startup. For this reason, graph the values as they arrive, not as a rate. Prometheus metrics covers how the exporter reads a chart.

Disk bandwidth shows where the data moves​

mfs_disks_stats_min_read_bytes and mfs_disks_stats_min_write_bytes report the bytes that each disk read and wrote over the last minute. One panel therefore shows both the figure for the whole cluster and the disks that produce it. mfs_disks_stats_min_rbw and mfs_disks_stats_min_wbw are not that traffic. Each is the speed of the disk while it worked: the bytes divided by the time that the disk spent reading, or writing and syncing. A disk that is slower than its peers stands out there. mfs_disks_stats_min_read_ops and mfs_disks_stats_min_write_ops beside the bytes separate a few large transfers from many small ones.

All three sets have the same disk labels. An uneven spread is therefore visible while it is still a performance question and not yet a full disk.

MooseFS draws its own charts as well​

The Master Server and every Chunkserver keep charts of their own. The GUI draws those charts for the cluster that it is connected to. Charts and statistics covers what those charts show, and the ranges that they hold. For one cluster and a question about the last few hours, the GUI is the shorter way. Reading the charts there needs neither Prometheus nor Grafana.

Use Grafana when the history has to outlast the ranges that a MooseFS chart keeps, or when several clusters belong on one screen. Also use Grafana when MooseFS has to appear on the same time axis as the rest of the infrastructure. Someone still has to look at a panel. Turning the same metrics into a condition that notifies you on its own is covered in alerting.