The CLI
mfscli prints the state of a cluster as text. The readings that the GUI displays are available here as well, except for its cluster map and its drawn charts. Chart data comes as numbers. The output comes in three shapes: framed tables for a terminal, a flat format for grep and awk, and a single JSON object for anything that parses. This page covers which section answers which question, how to read the output, and how to use it in a script. The switches themselves are listed in mfscli(1).
mfscli reads over the Master Server's client port
mfscli -SIG
With no other option, the tool looks for a Master Server at the host name mfsmaster on port 9421, the same port a mount uses. -H names another host and -P another port:
mfscli -H 192.0.2.10 -P 9421 -SIG
When you call mfscli with neither a section (-S) nor a command (-C), the tool prints a line that names both options and stops. There is no default view. mfscli also stops, with a non-zero exit code, when no Master Server answers and when the Master Server that answers is older than 3.0.0.
Two sections go beyond the Master Server: -SHD and -SCC open their own connections to every connected Chunkserver, each on the port that Chunkserver listens on (9422 by default). The machine that you run them from has to reach the Chunkservers as well. If that machine cannot reach them, those sections report a timeout, but the rest of the output is still printed.
The tables include no alert list. The GUI turns readings into named alerts. It uses the same numbers and shows each alert beside the object that the alert concerns. The alerts themselves are collected in the alerts reference.
The question names the section to ask for
A section is requested with -S and its two-letter label. Several labels fit in one call. -S may be repeated as well, so mfscli -SIG -SCS and mfscli -SIGCS ask for the same two sections. The value is matched as a substring, which matters when labels are joined: -SCSCC asks for the Chunkserver list (CS) and the Chunkserver charts (CC). Through the letters SC inside it, it also asks for the Storage Classes. Sections are printed in their own fixed order, not in the order that you asked for them.
| What you want to know | Section |
|---|---|
| Space, trash, sustained files, object counts, and the last metadata save | -SIG |
| The state of each Master Server, its metadata delay, and its last save | -SIM |
| How many chunks are missing, endangered, or below what their class asks for | -SIC |
| Which files lost a chunk | -SMF |
| How far the file loop and the chunk loop have got | -SIL |
| What the Master Server spends its memory on | -SMU |
| Which Chunkservers are connected, loaded, or in maintenance mode | -SCS |
| Which disks report errors, are being scanned, or are marked for removal | -SHD |
| Which clients have the file system mounted, and with which rights | -SMS |
| How many operations each mount sends | -SMO |
| Who holds a file open, and who holds a lock on it | -SOF, -SAL |
| Which export rules the Master Server serves | -SEX |
| Which Storage Classes and override patterns exist | -SSC, -SPA |
| Which quotas are set and how much of each is used | -SQU |
| Which Metaloggers are connected | -SML |
| The state of each location | -SLO PRO (PRO Edition only) |
| What the license covers and how much of it is used | -SLI PRO (PRO Edition only) |
| Chart data as numbers | -SMC, -SCC |
Three labels represent groups. -SIN asks for the cluster summary, the Master Servers, the Metaloggers, the locations PRO (PRO Edition only) and the license, the chunk matrix, the loops, the missing files, and the memory usage in one call. -SMD asks for the four mount sections (-SMS, -SMO, -SOF, -SAL), and -SRP for the two redundancy sections (-SSC, -SPA). In an installation with several locations, -l narrows the output to one of them. Reading a multi-site cluster covers what changes.
A few columns decide, and sizes are binary
The chunk matrix (-SIC) ends with the summary line that most checks rely on. The line counts the view that -m selects. The default view counts every chunk in the cluster, disks marked for removal included. The matrix starts with a warning row whenever those counts are not reliable. The JSON output states the same in progress_str. Cluster health parameters covers how to read both. Chunk versions and states covers what the grades mean.
In the Chunkserver list (-SCS), queue is the number of tasks waiting on that server. queue state labels that number: Normal, Overloaded, Rebalance, Fast rebalance, or G(<seconds>) while a grace period runs out. The GUI covers how each state affects the server, and why G(<seconds>) applies to Chunkservers older than any that are still built. maintenance reads off, on (permanent), or on (<n> seconds left) where CS_MAINTENANCE_MODE_TIMEOUT gives manual maintenance an expiry, or on (temp ; <n> seconds left). The last form is the mode that a controlled stop sets by itself. With no Leader, the column reads not available. Disconnected servers are listed in a separate table.
In the disk list (-SHD), status reads ok, ok (errors), damaged, invalid, scanning, removal ready, removal not ready with the reason in brackets, or unknown with the reason in brackets when the Chunkserver is unreachable or too old to report. More than one reading can apply at once. The readings are then separated by commas. last error gives the time of the newest failure that the disk recorded. The chunk on which that failure happened is shown only with -j, as last_error_chunkid. Wait for removal ready before you pull a disk. That reading means that what the disk held is now safely elsewhere. This section shows one time window at a time, the last minute by default. -m picks the hour or the day, and the average instead of the maximum.
The Master Server list (-SIM) shows state per server, the metadata delay of each Follower PRO (PRO Edition only) in seconds, and the outcome of the last metadata save. Metadata save states covers what a save state means.
Sizes in the tables are printed with binary prefixes, KiB, MiB, and GiB, where each step is 1024 of the step below. The percentages beside those sizes are computed from the same pair of numbers. The machine-readable formats print the byte counts instead.
Two output formats are meant for scripts
The framed tables are for a terminal. -p replaces them with a flat format. -s sets the separator and turns the flat format on with that same switch. The flat format is also used automatically whenever standard output is not a terminal, so mfscli -SCS | cat needs no switch. A flat row is the table title, a colon, the separator, and then the cells. A filter matches on that row:
mfscli -SIC -p -s ',' | grep 'chunkclass missing'
There is no header row to skip and no column names in the output to match on. The flat format is not the framed table with the frames taken off. The flat format prints byte counts and microsecond figures as plain numbers rather than in binary units. It omits the columns that the table computes, including percentages of used space. In several sections, the flat format orders and names its columns differently from the terminal view. A column position read off the screen is not the position in a flat row.
-j turns the whole run into one JSON object, printed once at the end. When no Master Server answers, the object is still printed, with an empty dataset and the reason in its errors array. The run then ends with a non-zero exit code. Sections appear under dataset. Failures appear in the errors array inside the same object, not on standard error. A section that fails therefore leaves the object valid and reports in place what went wrong.
JSON also includes more than the tables do. For this reason, prefer JSON over parsing flat rows. -SIC returns every variant of the matrix at once, each with its own summary. -SHD returns all three time windows with both the maximum and the average. -SMO returns the current hour and the previous hour. Byte counts and timestamps appear next to their human-readable forms. Where a table needs -m to choose a variant, the object contains all variants.
The object has no schema version. Its keys use the CLI's own column names. Instead, a script can pin the version that produced the object. That version is in the version field of every object. mfscli -v prints the same version. The Prometheus metrics are built from this same output.
-C changes the cluster instead of reporting
-C sends an order rather than asking a question. The argument follows the letters after a slash, with no space. An address is written as four numbers with the port after them:
mfscli -H mfsmaster -CM1/192.0.2.21/9422
| Command | What it orders |
|---|---|
-CM1/<ip>/<port> | Switch the Chunkserver into maintenance mode |
-CM0/<ip>/<port> | Switch it back to standard mode |
-CBW/<ip>/<port> | End a Chunkserver's grace period and return it to normal work |
-CRC/<ip>/<port> | Remove a Chunkserver from the list of active servers |
-CTR/<ip>/<port> PRO (PRO Edition only) | Remove it temporarily, accepted by a Master Server that is an elect or a Deputy |
-CRS/<session id> | Remove a session |
Orders need a Leader. -CTR is the exception: it is written for the case where there is no Leader. The outcome is printed as a line of text and does not affect the exit code. A command refused because the Chunkserver is still active, or the session still in use, ends the run at zero, exactly as a successful command does. You can combine commands with each other and with -S in a single call.
mfscli fills these commands in for you where they apply. In the framed table, the row of a disconnected Chunkserver shows the exact command lines for that server. The lines use the host and port that you invoked the tool with. The maintenance switch matches the server's current state: -CM1 while maintenance is off, and -CM0 while it is on, including the temporary form. The removal order is -CRC where a Leader is running, and -CTR where the Master Server that answers is an elect or a Deputy. Where no Master Server answers as a Leader, and where the removal order has no form that the Master Server understands, the cell reads not available instead of a command. The flat and JSON formats have no command columns. The row of an expiring mount shows its -CRS line.
Deciding when to give such an order is a task rather than a reading. Maintenance mode, retiring disks, and taking a Chunkserver out of the cluster are steps described in disks and Chunkservers. A session is removed once the mount that it belongs to has disconnected permanently.
A health check reads numbers, not exit codes
The exit code answers one question: whether a Master Server answered. The code is non-zero when no Master Server answered, and when the Master Server that answered is too old to talk to. A cluster with missing chunks and half its Chunkservers offline still exits zero. A check therefore runs the sections that it needs and tests the values:
#!/bin/bash
# Exits non-zero when a chunk is missing or endangered, or a Chunkserver is disconnected.
set -eu
state=$(mfscli -H mfsmaster -SIC -SCS -j)
echo "$state" | jq -e '
.dataset.info.chunks.summary.allchunks
| .missing == 0 and .endangered == 0
' > /dev/null
echo "$state" | jq -e '
[.dataset.chunkservers[] | select(.connected == false)] | length == 0
' > /dev/null
A Master Server that does not answer ends the script on the first command. Each test that fails ends the script at that point. Where locations PRO (PRO Edition only) are in use, the chunk matrix has one more level of keys above the summary. Without -l, its only key is all_locations, and the path becomes .dataset.info.chunks["all_locations"].summary.allchunks.
What to watch covers which readings need a check of their own, and what to compare them against.