Skip to main content

Several processes on one machine

For a production cluster, the guideline is to install the server components separately, each on a machine of its own, sized for its role as described in Planning & Sizing. Sharing a machine is the exception. This page shows how to share a machine when there is a reason to. It covers the combinations that run with default configuration and how to start a second process of the same component next to the first. It also covers what two Chunkservers on one machine do to redundancy. Whether the shared machine is physical, a virtual machine, or a container is a separate decision. Its consequences for each role are covered in Bare metal, VMs, and containers.

Different components share one machine without conflict​

Every MooseFS component is a process, not a machine: mfsmaster, mfschunkserver, mfsmetalogger, and mfsgui each run as one. Two different components on one host compete for neither ports nor files of their own. Each daemon locks its working directory with a lock file that has the name of that daemon. The components do share that directory, because every component defaults to the same DATA_PATH. One pair of components notices this. A Master Server started beside a Metalogger reads the copy of the metadata that the Metalogger keeps. The Master Server refuses to start when that copy is newer than its own copy or has a different metadata identifier (a refused start names its reason). Where a daemon listens at all, it listens on a default port of its own. The Metalogger only connects out to the Master Server.

Combinations that occur in practice:

  • A Metalogger beside a Chunkserver. A Metalogger on its own asks little of a machine, so the Metalogger shares hardware that is already running. A Metalogger that is meant to supply the metadata for a replacement Master Server belongs on a machine sized for that job instead. The same page covers that sizing.
  • The GUI beside the Master Server, or on any other host. The GUI connects to the client port of the Master Server, like any other client of that port. The GUI page covers the GUI.
  • A Chunkserver beside the Master Server. This works, but the machine is sized as a Master Server machine first. The metadata must fit in RAM. The Chunkserver gets what is left.
  • Every component on one host: a single-node test cluster, described in the last section.

Two processes of the same component need extra work, because they conflict over everything that the first process owns by default.

For the Master Server, the useful case is Master Servers of different clusters. Each of them serves a namespace of its own. The reason is either performance, because the master processes of the clusters then work on CPU cores of their own, or keeping the metadata of different tenants in separate processes and files. Redundant Master Servers of one PRO cluster do not belong on one machine, because they would share the failure that they exist to survive. Two things stay shared and need attention. The first is the RAM of the machine: the metadata of all the clusters must fit into it together (Master Server hardware). The second is name resolution: each cluster needs a resolvable master name of its own.

A second process needs its own configuration, path, port​

Each additional process of the same component must own three things:

  • A configuration file of its own, given at start with the -c switch (mfschunkserver(8)). mfsmaster(8) and mfsmetalogger(8) take the same switch.
  • A DATA_PATH of its own. The working directory holds the lock file of the daemon. For a Chunkserver, the directory also holds chunkserverid.mfs, the identity that the Chunkserver presents to the Master Server. A second process pointed at the same directory finds the lock taken and does not start.
  • Listen ports of its own. Two processes cannot listen on the same address and port. The Master Server also finds the entries in its register of Chunkservers by that pair (a Chunkserver is known by identifier, not address). The listen-port options end in _LISTEN_PORT. Options that end in _PORT alone, MASTER_PORT among them, name the port that a process connects to. These options stay as they are. The configuration reference of each component lists them. Network ports lists the defaults.

A Chunkserver also needs a disk list of its own: HDD_CONF_FILENAME in mfschunkserver.cfg that points to a separate mfshdd.cfg, with no directory shared between the processes. Each listed directory is locked by the process that uses it. By default, a Chunkserver refuses to start because of invalid entries. So a copied configuration that still reads the disk list of the first process fails on the locks of those directories.

The packages include a systemd instance template for each component (moosefs-pro-chunkserver@.service and the matching templates of the other components). An instance named b reads its configuration from /etc/mfs-b/, so the steps below use that layout. The template expects the working directory to match the instance name. The PIDFile of the template names the lock file under /var/lib/mfs-b. The template does not pass that path to the daemon. If a configuration file names another directory, the daemon runs correctly, but the unit cannot find the daemon. The unit waits for the lock file that never appears. The unit gives up with Job for moosefs-pro-chunkserver@b.service failed because a timeout was exceeded, then restarts the same attempt. With the template, set DATA_PATH to /var/lib/mfs-b.

  1. Create the instance's configuration directory and working directory:

    mkdir /etc/mfs-b
    cp /etc/mfs/mfschunkserver.cfg /etc/mfs-b/mfschunkserver.cfg
    mkdir /var/lib/mfs-b
    chown mfs:mfs /var/lib/mfs-b

  2. Create /etc/mfs-b/mfshdd.cfg with one line per directory, as in mfshdd.cfg. Name only directories that the first process does not list, and make them owned by the user that the daemon runs as:

    /mnt/chunk03
    /mnt/chunk04
    chown mfs:mfs /mnt/chunk03 /mnt/chunk04

  3. In /etc/mfs-b/mfschunkserver.cfg, point the process away from the defaults of the first one. Side by side, the two files differ in exactly three lines. The first process runs on the defaults, which are commented out in a new configuration file:

    # /etc/mfs/mfschunkserver.cfg, the first process, on its defaults:
    # DATA_PATH = /var/lib/mfs
    # HDD_CONF_FILENAME = /etc/mfs/mfshdd.cfg
    # CSSERV_LISTEN_PORT = 9422

    # /etc/mfs-b/mfschunkserver.cfg, the second process, values of its own:
    DATA_PATH = /var/lib/mfs-b
    HDD_CONF_FILENAME = /etc/mfs-b/mfshdd.cfg
    CSSERV_LISTEN_PORT = 9522

    The default port 9422 belongs to the first process. Any free port works for the second process. Everything not listed here stays identical in the two files.

  4. Enable and start the instance:

    systemctl enable --now moosefs-pro-chunkserver@b.service

    The unit runs mfschunkserver -c /etc/mfs-b/mfschunkserver.cfg start. Where systemd does not manage the process, the same command with stop or reload controls the process directly. General start and stop procedures are in starting and stopping.

To verify, list the connected Chunkservers:

mfscli -H mfsmaster -SCS

Both processes appear as separate Chunkservers with the same IP address and different ports. mfscli takes the Master Server's address with -H. The Chunkservers tab of the GUI shows the same list.

warning

The cluster now counts two Chunkservers where there is one machine. Copies placed on different Chunkservers can share this one piece of hardware. The next section keeps them apart.

The Master Server cannot see shared hardware​

The Master Server registers each Chunkserver under the address and port that the Chunkserver announces. Nothing in the protocol names the machine that uses that pair. Allocation and space selection covers that mechanism. Placement under a plain goal treats every registered server as an independent target, so two copies of one chunk can be stored on the two processes of one machine. One hardware failure then loses both copies.

To keep full copies on different machines, distinguish copies by IP address in the Storage Class. Append /[IP] to the label expression, and give the class the STRICT labels mode with -m s. The servers that share an address then supply at most one copy (label expressions and matching describes the grouping and the modes). Create the class with mfsscadmin. Run it from inside a MooseFS mount, or point it at one with -M (data placement tasks):

mfsscadmin create -m s -K '2*/[IP]' twocopies_twomachines

The mode enforces the rule. A class without -m gets STANDARD. Under STANDARD, when no matching server is available for a copy, the copy is given a server that does not match, so that the chunk is written complete. On a machine whose two processes are the only candidates, both copies are stored there. Later, the cluster has nothing to correct. STRICT never uses a server that does not match. Where the processes of one machine are the only candidates, a new chunk of this class cannot get its two copies, so the write fails until a Chunkserver with a different address can take the second copy. Up to 5.1.0, such a chunk was written in one copy and got its second copy later.

Files in this class then get two copies, each on a Chunkserver with a different IP address. The IP test has three limits:

  • It compares registered addresses. Two processes that register different addresses of the same machine count as different IPs, so keep the processes of one machine under one address.
  • It applies to full copies only. For Erasure Coding parts the distinguish extension is ignored (mfsscadmin(1)).
  • CHUNKS_UNIQUE_MODE in mfsmaster.cfg applies the same test to every class at once. The option exists for compatibility with 3.x and overrides the distinguish settings of all Storage Classes. mfsmaster.cfg(5) says to leave it at 0 and put the rule in the classes instead.

A single-node test cluster is the regular installation​

For evaluation, every component runs on one machine. Follow installing the server components and first mount and verification, and perform every step on the same host. The default ports of the components do not conflict. Two points are specific to the single host:

  • The name of the Master Server must resolve to the network address of the machine, not to loopback. A Chunkserver pointed at a loopback address refuses to start at all. The Chunkserver logs localhost (127.0.0.1) can't be used for connecting with master (use ip address of network controller), then init: master connection module failed !!!, and exits. In /etc/hosts, map the name to the host's address:

    192.0.2.10 mfsmaster

  • Each copy of a chunk is stored on a different Chunkserver process. A Storage Class that requires two copies therefore needs a second Chunkserver process, set up as in A second process needs its own configuration, path, port. This tests real replication. This setup does not add redundancy, because the machine remains one failure domain. Losing the machine loses every copy at once.