Addresses and ports

This page describes how to move a running cluster to different addresses or ports: renumbering a subnet, giving a Chunkserver a new address, changing the set of addresses that the Master Server's name resolves to, and moving a service to another port. Every step here is a configuration change on a cluster that keeps serving. Every step is applied with a reload rather than a restart. The first setup of the addresses is part of prepare the operating system. For an address that stopped working although nobody changed it, see network problems.

A Chunkserver is known by identifier, not address

This identifier makes the whole operation safe. Understand it before the first step. The Master Server gives each Chunkserver a number when the Chunkserver first connects. The Chunkserver keeps that number in chunkserverid.mfs in its data directory. The Master Server uses the address and port to find the entry, but they do not identify the entry.

A Chunkserver that reconnects from a different address therefore keeps its entry. The Master Server recognizes the identifier and moves the existing entry to the new address. The Master Server records in its log that it found the server by its identifier. The log line also names the previous address of the server. There is no second entry. The count of registered servers does not change. The chunks are counted again as the server reports them.

The only way to lose the entry is to lose the file. A Chunkserver whose chunkserverid.mfs is missing asks for a new identifier. At an unchanged address and port, the Master Server finds the old entry by the address and answers with the identifier that the entry already holds. The Chunkserver writes that identifier back to the file. At a new address, the Master Server has nothing to match. The Chunkserver therefore gets a new identifier and registers as a server that the cluster has never seen. Moving disks to another machine causes both changes at once. The old entry then remains as disconnected until CS_DAYS_TO_REMOVE_UNUSED in mfsmaster.cfg expires it, seven days by default. Copy the data directory with the disks, and do not delete that file in the process.

Where the addresses are set and what applies them

Every option below is a runtime option. The process re-reads it on reload, without a restart. A listening option applies to what the module opens next. The connections that are already open stay open during the reload. An option that names where the module connects to behaves differently. A Chunkserver whose Master Server address or port changed disconnects from the Master Server on purpose. A Metalogger and a Follower Master Server PRO (PRO Edition only) drop their connection to the Master Server on every reload, even a reload that changes nothing.

Where	Options	What they name
mfsmaster.cfg	MATOCS_LISTEN_HOST, MATOCS_LISTEN_PORT, MATOCL_LISTEN_HOST, MATOCL_LISTEN_PORT, MATOML_LISTEN_HOST, MATOML_LISTEN_PORT, MASTER_HOST PRO (PRO Edition only), BIND_HOST PRO (PRO Edition only)	where the Master Server listens for Chunkservers, clients, and the Metaloggers and other Master Servers. Which Master Server a Follower reaches, and from which local address it reaches it
mfschunkserver.cfg	MASTER_HOST, MASTER_PORT, BIND_HOST, CSSERV_LISTEN_HOST, CSSERV_LISTEN_PORT	which Master Server to reach, which local address to reach it from, and where clients reach this Chunkserver
mfsmetalogger.cfg	MASTER_HOST, MASTER_PORT, BIND_HOST	which Master Server to reach and from which local address
mfsgui.cfg	GUISERV_LISTEN_HOST, GUISERV_LISTEN_PORT	where the GUI listens

Two things are not covered by this table. A mount gets its Master Server address at mount time and has no reload at all. To change a client that is configured with an address rather than a name, mount it again (client and mount tasks). In an installation with several Master Servers PRO (PRO Edition only), mfsmaster.cfg has no separate port option for the connection to the Leader. A Follower reaches the Leader on the port that the Follower itself listens on, so that port has to be the same on all Master Servers.

A name is resolved again at every reconnection

A Chunkserver and a client resolve the name that they are configured with every time they build a connection, rather than once at startup. A Chunkserver that loses its connection reuses its current address for one attempt. The Chunkserver resolves the name again on the next attempt. At the default MASTER_RECONNECTION_DELAY of five seconds, a stale address is used for one cycle at most. Instead, a Metalogger keeps the address that it resolved. The Metalogger reuses that address as long as the attempts succeed. A failed attempt discards the address, and so does a reload. The Metalogger then resolves the name again.

Because of that resolution, the Master Server's addresses can change without a change on any other machine. Make the name resolve to the new addresses, and the cluster follows on its own. In MooseFS PRO PRO (PRO Edition only), a name that resolves to several addresses is read as a list of candidates. The Community Edition takes one address at random from the addresses that the resolver returns. The Community Edition picks again at every resolution. An old record left in place therefore costs one failed attempt per pass through the list in MooseFS PRO. In the Community Edition, the same record is a candidate in every pick, so a module can pick it and fail to connect. Remove the old records as soon as nothing answers on them.

If you change the name that a module is configured with, rather than the addresses that the name resolves to, that is a configuration change. It needs a reload.

To verify in an installation with several Master Servers PRO (PRO Edition only), read the Leader's log. The Leader asks every module for the list of addresses that the name gives that module. The Leader then compares that list with its own. The Leader's log reports a module that disagrees by its address. A prefix in front of the address names the kind of module: CS for a Chunkserver, CL for a client, METALOGGER for a Metalogger, and MASTER for a Master Server. The address of a Chunkserver includes its port. The address of a client and the address of a Metalogger do not. The check runs on its own schedule, set by CONFIGURATION_TEST_PERIOD, every 24 hours by default. The check therefore confirms a finished change and does not show a change in progress. A value of 0 turns the check off.

The exports are on addresses, so they go first

Client access is granted by address range. This step goes wrong when the order is wrong. A client whose new address matches no entry in mfsexports.cfg is refused at registration. The client loses its session, along with the files that it held open and the locks that it took.

Add the new range before anything moves. Leave the old range in place until nothing uses it. The reload that installs the new range affects more than that range, as described under Changing an export password disconnects every mount. The Master Server keeps one checksum for the exports. The checksum covers the parsed entries, not the bytes of the file. Reordering lines, editing a comment, and changing whitespace do not change the checksum. A change to the value of any entry changes the checksum, so every mount is dropped at once. Each mount then registers again. A mount whose entry still admits it reconnects with its session. The order of the steps here is meant to produce that outcome.

Two more files contain addresses, and both are re-read on the same reload. mfsipmap.cfg rewrites the Chunkserver addresses that the Master Server gives to clients. A wrong entry there sends clients to an address that they cannot reach. Nothing visible on the Master Server breaks. mfstopology.cfg sorts addresses into a distance map. An address that is no longer in the map breaks nothing visible. Placement then stops respecting the topology that it was meant to respect.

In a cluster divided into locations PRO (PRO Edition only), the address ranges mapped to each location are metadata rather than a file. They are checked at registration. A Chunkserver has its connection closed when its address maps to a location other than the location that it registered from. The log names the mismatch. Map the new ranges before the servers move, as in Location tasks.

Changing a listening port keeps existing connections

When a module is told to listen somewhere else, it opens the new socket first. The module closes the old socket only after the new socket is listening. Connections that are already established are not affected. They were accepted on the old socket and keep working. The new address applies to what connects next. A reload that cannot open the new socket keeps the old socket and reports the failure. A mistyped port number therefore does not take a module off the network. This applies to a port that is already in use and to a privileged port, in every module. Since 5.2.0, it also applies to an address name that does not resolve: a Chunkserver keeps the old socket and logs socket address has changed, but can't be resolved, and a Chunkserver started with such a name refuses to start. Up to 5.1.0, a Chunkserver does not check whether the new listening address resolves. It closes the old socket, listens on any address on a random port, and registers with the Master Server with port zero, so no client is sent to it.

In one case, a module does more than that on purpose. A Chunkserver that changes where clients reach it re-registers with the Master Server immediately. The Master Server gives the address of the Chunkserver to clients and would otherwise keep giving out the old port. The disconnection is graceful, so the Master Server records it as temporary maintenance and starts no rebuild.

Clients are the exception again. A mount keeps the port that it was given at mount time, so a changed client port takes effect at the next mount.

The order that keeps the cluster serving

1. Put the new client and server ranges into mfsexports.cfg, into the location mappings PRO (PRO Edition only), into mfsipmap.cfg and into mfstopology.cfg where those are used. Then reload the Master Server. Verify that the Exports checksum on the Metadata Servers (Masters) subtab of the Info tab in the GUI has changed and that the mounts have reconnected.
2. Add the new addresses to the Master Server's name in DNS, and leave the old records in place for now.
3. If the Master Server's own listening options are changing, change them and reload the Master Server. Verify in the Master Server's log that the server listens where you configured it.
4. Move the Chunkservers one at a time: change the address on the machine, adjust MASTER_HOST and the listening options if either is changing, and reload the process. Since 5.2.0, the Chunkserver's log names the new listening socket, for example main server module: socket address has changed, now listen on *:80. Up to 5.1.0, that line names the address and the port that the process had before the reload, so do not read the new port from it. The Chunkservers tab of the GUI and mfscli show the port that the server is registered with. Before you start the next Chunkserver, verify there that the server is connected at its new address, that the count of registered servers has not grown, and that the chunk matrix is clean.
5. Move the Metaloggers, then the clients. A client configured with a name follows on its own. Remount a client that is configured with an address.
6. Remove the old records from DNS and the old ranges from the exports file after nothing answers on them.

The move is finished when every Chunkserver shows as connected at its new address, the registered count is the same as before the work started, and the chunk matrix shows nothing missing, endangered, or undergoal.