Skip to main content

Network problems

A network fault in MooseFS is rarely reported as one. It appears as a Chunkserver that dropped out, as replication that started for no reason, as a mount that stopped answering, or as a client session that the cluster still lists after the client has disconnected. This page maps those symptoms back to the connection that produced them, separates the symptoms that are not network faults at all, and gives an order to work in.

The cluster runs no test of the network. The cluster reports what happened to its own connections, so its messages name consequences rather than measurements. One reading is closer to a measurement than the rest: a mount times its own exchange with the Master Server once a minute.

Each side sees part of the same fault​

One break leaves a record on each machine it touches. No single record includes what the other machines recorded. Collect all the records before deciding what failed.

The Master Server sees silence, not its cause​

The Master Server watches its connections for silence rather than for loss. When a second passes with nothing else to send, the Master Server sends a keepalive down each connection. Supervisor connections are the exception. They get no keepalive, but the timeout still kills them. When nothing has arrived on a connection for the length of its timeout, the Master Server kills that connection. In MooseFS PRO PRO (PRO Edition only) the Master Server does not kill a client connection while the instance is changing state, that is, during an election. The timeouts are in mfsmaster.cfg: MATOCS_TIMEOUT for Chunkservers, MATOCL_TIMEOUT for clients, and MATOML_TIMEOUT for Metaloggers, supervisors, and Follower Master Servers PRO (PRO Edition only). Each is ten seconds by default and ten seconds at the lowest, because a smaller value in any of them is raised to ten. A module that could not answer for ten seconds therefore leaves the same trace as a module whose link was cut.

One line distinguishes a socket that the peer closed from a connection that the timeout killed. connection with CS(<address>:<port>) has been closed by peer says that the peer closed the socket, which is what a stopped process does. A connection killed on the timeout leaves only the disconnect line. That line, and the Chunkserver's own account of the same event, are in logs.

Replication starts on its own afterwards. Nothing else has to happen for it to start: the chunks on a disconnected server are below their goal. Being below goal schedules a replication. Maintenance mode is the only state that holds back those repairs.

The Chunkserver times out, then loops on reconnect​

A Chunkserver applies its own timeout to its connection to the Master Server. MASTER_TIMEOUT in mfschunkserver.cfg sets that timeout. At the default of 0 the option takes the value that the Master Server sends back at registration. After a failed attempt the Chunkserver waits MASTER_RECONNECTION_DELAY seconds, 5 by default, and tries again. The Chunkserver keeps the address that it last reached the Master Server at. After a connection that had registered breaks, the first attempt goes to that remembered address. An attempt that fails discards the address. The Chunkserver then resolves MASTER_HOST again before the next attempt. In MooseFS PRO PRO (PRO Edition only) a name that resolves to several addresses is read as a list. The Chunkserver then asks DNS again only after it has worked through the whole list. A server caught in that loop repeats those lookups, so a rise in DNS queries from one machine is a symptom of the loop rather than its cause.

Two failures never reach the Master Server at all, because no connection was made. master connection module: can't resolve master host/port (<host>:<port>) means that the name does not resolve. master connection module: localhost (<address>) can't be used for connecting with master (use ip address of network controller) means that the name resolves to a loopback address on that machine, so check what that machine resolves MASTER_HOST to before looking at the path. The Master Server refuses the same address from its own side once a Chunkserver has registered with a loopback address. The Master Server's line for that refusal is in logs.

Toward clients and toward the other Chunkservers in a write chain, the Chunkserver reports what it could not send. It logs send_and_free: 'send(read data)' disconnected when the other end had already disconnected, and send_and_free: 'send(read data)' timed out when the socket accepted no byte at all for five seconds. Those five seconds limit the stall, not the whole send. The send itself has thirty times that budget, 150 seconds, to finish. read status, write status, and write init replace read data for the other packets. Not every one of them goes to a client. write init goes to the next Chunkserver in the write chain. write status goes back to the previous link in that chain, which is a client only for the first Chunkserver of the chain. An idle client connection is dropped after five seconds of silence in any case. The Chunkserver logs csserv: connection timed out for it. That line includes neither an address nor a port, so it does not say which client it concerns. A client therefore sends a keepalive of its own on the connections that it is not using (client background work).

The client names the unreachable address​

A mount that cannot open a connection to a Chunkserver logs the address, the port, and the error that the attempt returned: readworker: can't connect to (<address>:<port>): <error> and writeworker: can't connect to (<address>:<port>): <error>. Read the address before the error. An address that belongs to no network that this client can reach is a problem of address rewriting rather than of the link.

When the connection opens and then stops carrying data, the client logs a different line:

file: <inode>, index: <n>, chunk: <id>, version: <n>, part: <n> - readworker: connection with (<address>:<port>) was timed out (lastrcvd:<t>,now:<t>,lrdiff:<t> received: <n>/<n>, try counter: <n>)

lrdiff is the time without data that exceeded the five-second limit. received is how much of the block had arrived by then. That pair distinguishes a transfer that never began from a transfer that stopped part of the way through.

A break in the connection to the Master Server is logged as master: connection lost (header) or master: tcp recv error: <error> (header). After that, the mount registers again every two seconds. The first of those attempts goes to the address that the mount already holds. The mount resolves the name again only after an attempt has failed.

When you read all of this, remember that the log starts late. A retried operation reports nothing until the attempt that mfslogretry names, so an empty log during a short outage does not prove that nothing failed. The default and how to lower it are in client and mount problems.

A blocked port shows on one side only​

Every MooseFS connection is opened by the same side every time. A client connects to the Master Server and the Chunkservers. A Chunkserver connects to the Master Server and other Chunkservers. A Metalogger connects to the Master Server. A Follower Master Server PRO (PRO Edition only) connects to the Leader. No Master Server connects to anything else. The Master Server sends everything back down the connection that the connecting module opened (the MooseFS protocol and its ports). The port of each component is listed in network ports.

Because of that asymmetry, a filter shows up in one place and not in others:

  • Block the Chunkserver data port toward the clients and the cluster looks healthy. The Chunkserver registers, the GUI lists it as connected, and the Master Server logs nothing. Only reads and writes fail. The client logs readworker: can't connect to with that one server's address.
  • Block the Master Server's client port and no mount comes up, but the Chunkservers stay connected and the cluster reports nothing wrong with itself.
  • Block the Master Server's Chunkserver port and the Chunkserver drops out as it does when a link is cut. The Chunkserver logs a timeout while its process keeps running.

Check from the machine that logged the failure, toward the address and port in the message, and in the direction the connection is opened. A check in the opposite direction proves nothing about the path that MooseFS uses. Read the error at the end of the client's message on its own. A refusal and an attempt that got no answer are different failures and do not have the same cause.

A stale name resolves to nothing there​

A module resolves the Master Server's name only when it needs an address. The module then keeps that address for as long as attempts to it succeed. A failed attempt discards the address. The next attempt then resolves the name first. A connected module does not consult DNS at all. A Chunkserver discards its cached address on a connection timeout and resolves before the next try. A mount whose connection broke re-registers every two seconds. The first of those attempts goes to the address that the mount already holds. The mount resolves the name again only after an attempt has failed. A mount that is starting retries up to ten times. It resolves the name again for each attempt and gives each attempt a two-second connection timeout. The mount then gives up with can't connect to mfsmaster ("<host>":"<port>"). A loss of DNS therefore changes nothing while everything stays connected. It stops anything that disconnects from reconnecting.

In MooseFS PRO PRO (PRO Edition only) the name provides the list of Master Servers. Nothing else provides that list (leader election, quorum, and the role of DNS). A client takes one entry per attempt and resolves the name again only after it has worked through the whole list. An address that answers nothing therefore fails one attempt out of as many as the record holds instead of failing all of them. A stale record then looks like a Master Server that fails intermittently: the same command works, then does not, then works again.

The cluster detects the disagreement itself PRO (PRO Edition only). The check runs on a schedule rather than in response to a fault. Every CONFIGURATION_TEST_PERIOD hours, 24 by default, the Leader or the Deputy asks each connected module for the list that the module resolved. Modules built before the 4.56.0 release are not asked at all. The Master Server compares each answer with its own list entry by entry. A module that differs is asked a second time. That second request forces the module to resolve the name afresh first. Only the answer to it produces the message, which names the module that differs. A stale name is therefore reported at the next run of the check, not at the moment of the fault:

CL(<address>) has incorrect list of master IP/PORT numbers
CS(<address>:<port>) has incorrect list of master IP/PORT numbers

Metaloggers and Follower Master Servers produce the same line with their own label in front of the address. That label shows the module's synchronization state as well as its role. A registered Metalogger appears there as METALOGGER-SYNC(<address>) or METALOGGER-DELAYED(<address>). A Follower Master Server appears there as MASTER(<address>), MASTER-SYNC(<address>), or MASTER-DELAYED(<address>), with no word for its role in the line. Search the log for has incorrect list of master IP/PORT numbers rather than for a label. This is the only message in the product that says that a machine holds a different list of Master Servers from the rest. A module that produces it is misconfigured whatever the other readings show.

The proof on the client side is .params in the root of the mount. In MooseFS PRO the file names every address that the mount got from DNS, one - resolved ip:port number[<n>]: <address>:<port> line each, next to the address that the mount chose. Read that list on two machines that disagree, and the stale entry becomes visible.

No MooseFS message names the MTU​

MooseFS sets no MTU, measures no packet size, and reports none. The MTU is a property of the interfaces and of every switch port and router between them. It is a layer below anything that the protocol handles. The cluster can therefore only report the timeouts that follow from a mismatch. Those are the ordinary timeout messages of the Master Server, the Chunkserver, and the client. None of them names a cause. Suspect the MTU when timeouts are spread across machines that are otherwise healthy and not busy. Check every interface and every switch port on the path rather than the two endpoints. MTU describes what to set and where the setting has to apply.

Address rewriting can hand out an unroutable address​

The Master Server repeats the address that a Chunkserver registered with. Two mechanisms rewrite that address for the asking machine before the Master Server sends it: mfsipmap.cfg maps addresses explicitly and takes precedence, and MULTILAN_BITS with MULTILAN_CLASSES rewrites by prefix. Both are described in the MooseFS protocol and its ports, and their options in mfsmaster.cfg and mfsipmap.cfg.

The prefix rewrite applies only when the asking machine's address and the Chunkserver's address each fall into one of the classes listed in MULTILAN_CLASSES. Leave one of the two networks off that list and the rewrite does nothing at all. The address goes out unchanged. A client in the network that was left off is given an address in the network that it cannot reach. Nothing logs it, because the Master Server answered the question it was asked. The symptom appears on the client: readworker: can't connect to (<address>:<port>) with an address outside the client's own networks, while other clients read the same file. Every address that a Chunkserver may register with has to be covered. Every client network has to be covered as well.

REMAP_* is a different mechanism and does not replace the prefix rewrite. It rewrites the address once, at registration, identically for every machine that asks, so it changes what the whole cluster uses rather than what one group of clients is told.

Replication orders include the registered address and are not rewritten for anyone, so the network that replication uses and the network that clients read over can be different networks. A fault on one network then produces a symptom that the other network never shows: repairs that make no progress while clients read at full speed, or the reverse.

Two more configuration faults look like flapping links. A Chunkserver whose socket address and working address map to different locations PRO (PRO Edition only) is killed at registration with chunkserver's socket peer ip (<address>) is mapped to different location (<n>) than working address (<address>:<port>) which is mapped to <n>. Divide a new cluster into locations describes how to map addresses to locations. A Chunkserver that registers with an address and port that a different connected server already holds is refused with csdb: found cs using ip:port (<address>:<port>,<id>), but server is still connected. When the same server reconnects before the Master Server has cleaned up its previous connection, it matches on the ID too. That case produces csdb: found cs using ip:port and csid (<address>:<port>,<id>), but server is still connected instead. Both of them end in can't accept chunkserver <address>:<port>. All of these faults loop: connect, refuse, retry.

Not every disconnect is the network's fault​

The Master Server measures the time without a packet. Anything that keeps a module from answering for the length of the timeout therefore ends the connection exactly as a broken link would. The causes below account for most of what is not the network:

  • A Chunkserver under load. The Queue state column and the per-disk operation times show whether the server was busy. Both are in the GUI and in what to watch. The disks themselves are covered in Chunkserver and disk recovery.
  • A disk answering slowly. The client's readworker: connection with (<address>:<port>) was timed out counts silence on a connection, not the state of a link. A disk that has stopped delivering blocks produces that silence. received: <n>/<n> in the same line shows how far the transfer got before it stopped.
  • A busy Master Server. high packet travel time between client and master (<seconds>s) - ignoring in time sync is logged above 100 ms. The message has two causes: a path that really is that slow, or a Master Server with no time to answer. Compare the message with the Master Server's own load before you treat it as a network fault.
  • A pause that is not a failure at all. An unreachable Master Server does not fail operations at once. The mount retries for about two minutes on its default budget before an application is given an I/O error. A hang of that length is therefore the budget running out rather than proof that the path failed. The budget and the options that change it are in client and mount problems.

A client that disconnected leaves a trace of its own. Its session is kept after the lost connection for SESSION_SUSTAIN_TIME, one day by default. The mount therefore stays listed in the Inactive mounts (parameters) table on the Mounts tab of the GUI under the mnt_inactive alert. That alert's text names packet loss, latency, misconfiguration, and over-restrictive firewalls as possible causes (Alerts). One such entry is a client that disconnected. If the same client appears there repeatedly, that points to the path between the client and the cluster.

Work from the cheapest check upward​

  1. Take the mount's own readings: master.reconnects and master.usec_ping in .stats, and the addresses in .params. Client and mount problems describes what each of them shows.
  2. Read the system log on every machine that the connection touched, and remember that a retried operation starts logging late. Logs describes what the lines look like and which of them to recognize.
  3. Read the cluster's own view: disconnected Chunkservers, inactive mounts, and the alerts active at that moment (what to watch).
  4. Compare what each machine resolves the Master Server's name to, and search the Master Server's log for has incorrect list of master IP/PORT numbers.
  5. Check the ports from the machine that logged the failure, toward the address in the message, in the direction the connection is opened.
  6. Check the MTU on every interface and every switch port along the path.
  7. Decode the traffic with mfsnetdump. It reads packets from an interface or from a saved capture file and prints them as MooseFS packets. Use it after the checks above narrow the fault to one connection.

When the readings show that the cluster is sound and one client is not, go to client and mount problems. When the readings point to the Master Server itself, go to Master Server problems.