Skip to main content

Prepare the operating system

This procedure prepares the operating system on every machine that will run a MooseFS component. The procedure brings the system up to date, adds the MooseFS package repository, and prepares the Chunkserver disks. The procedure also makes the Master Server name resolvable, sets up a time service for every machine's clock, and sets the memory and CPU policy on the Master Server host. Run the procedure after the pre-flight checklist passes and before installing the server components.

Start from a supported, updated operating system​

Install one of the supported operating systems on every machine. That page also decides what to do with a system without a package of its own. Then bring the system fully up to date with its package manager before you add anything of MooseFS. An updated system has the current kernel, libraries, and security patches. The packages you install next resolve their dependencies against current versions instead of old ones.

Add the MooseFS package repository​

MooseFS installs from packages served at repository.moosefs.com. Add the repository on every machine, servers and clients alike. Install the server components and Install clients describe how to install the packages themselves.

The paths below name the moosefs-5 tree, which serves the PRO releases that this documentation covers. The Community Edition is released on a line of its own, currently 4.59.x (an instance runs one edition throughout on the server side). The Community Edition packages are under moosefs-4 in the same places: replace that one segment of the path to install them.

On Debian and Ubuntu the signing-key step needs gnupg, which supplies gpg and which apt-key calls in turn. A minimal install of either does not include gnupg. Install it first with apt install gnupg. Without it the newer form fails with gpg: command not found and the older form with gnupg, gnupg2 and gnupg1 do not seem to be installed. Either way no key is added.

Ubuntu​

  1. Add the signing key. For Ubuntu 22.04 LTS and 24.04 LTS:

    curl https://repository.moosefs.com/moosefs.key | gpg -o /etc/apt/keyrings/moosefs.gpg --dearmor

    For Ubuntu 16.04 LTS, 18.04 LTS, and 20.04 LTS:

    wget -O - https://repository.moosefs.com/moosefs.key | apt-key add -

  2. Add the repository entry with the codename of your release. For Ubuntu 24.04 LTS (Noble) on amd64:

    echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/moosefs.gpg] http://repository.moosefs.com/moosefs-5/apt/ubuntu/noble noble main" > /etc/apt/sources.list.d/moosefs.list

    For another supported release, replace the codename in both places. The signed-by clause belongs to the newer signing-key step. Keep it for Jammy. Drop it for Focal, Bionic, and Xenial: apt-key puts the key in the global trusted.gpg and creates no file at that path. For Ubuntu 20.04 LTS (Focal) on amd64:

    echo "deb [arch=amd64] http://repository.moosefs.com/moosefs-5/apt/ubuntu/focal focal main" > /etc/apt/sources.list.d/moosefs.list

  3. Refresh the package index:

    apt update

Debian​

  1. Add the signing key. For Debian 11 (Bullseye) and newer:

    mkdir -p /etc/apt/keyrings
    curl https://repository.moosefs.com/moosefs.key | gpg -o /etc/apt/keyrings/moosefs.gpg --dearmor

    For Debian 9 (Stretch) and Debian 10 (Buster):

    wget -O - https://repository.moosefs.com/moosefs.key | apt-key add -

  2. Add the repository entry with the codename of your release. For Debian 12 (Bookworm), and with the codename replaced for another release:

    echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/moosefs.gpg] http://repository.moosefs.com/moosefs-5/apt/debian/bookworm bookworm main" > /etc/apt/sources.list.d/moosefs.list

    For Debian 9 (Stretch) and Debian 10 (Buster) drop the signed-by clause: apt-key puts the key in the global trusted.gpg and creates no file at that path.

  3. Refresh the package index:

    apt update

A Raspberry Pi uses the same steps. It runs Debian, so follow the steps above and replace arch=amd64 with arch=armhf or arch=arm64, whichever the model and the system installed on it use. MooseFS keeps no repository of its own for the Raspberry Pi.

RHEL, CentOS, Rocky Linux, AlmaLinux, and Fedora​

  1. Add the signing key:

    curl "https://repository.moosefs.com/RPM-GPG-KEY-MooseFS" > /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS

  2. Add the repository file that matches your release. For EL8:

    curl "https://repository.moosefs.com/MooseFS-5-el8.repo" > /etc/yum.repos.d/MooseFS.repo

    For another release, replace el8 in the file name. The repository serves el6 through el10. Fedora has a file of its own, MooseFS-5-f44.repo. The file names the release in its baseurl. Nothing substitutes that name, so a file left at el8 installs the EL8 build on any release. The first dnf install from the new repository asks once to import the signing key that you added in step 1. It refuses to import the key unattended.

The repository files above are for the EL releases and Fedora, not for openSUSE. The .rpm packages are still compatible with openSUSE Leap, as supported operating systems covers.

FreeBSD​

Create the repository configuration and refresh the package index. For FreeBSD 14 on 64-bit x86:

echo 'moosefs: { url: "http://repository.moosefs.com/moosefs-5/freebsd/14:x86:64", enabled: yes, mirror_type: NONE }' > /etc/pkg/moosefs.conf
pkg update

macOS​

macOS has no package repository to add. The client installs from a downloaded package. macFUSE has to be installed first, because the MooseFS package neither includes nor installs it. The steps are part of installing clients.

Verify the repository​

After the refresh, the package manager finds the MooseFS packages of the version that this repository serves. On FreeBSD the search also lists the MooseFS packages that FreeBSD ships in its own repository. Those packages have a version of their own, so read the version in the output, not the names:

apt-cache search moosefs   # Ubuntu and Debian
yum search moosefs         # RHEL family
pkg search moosefs         # FreeBSD

Each Chunkserver disk gets its own file system​

A Chunkserver stores chunks as regular files on local file systems, one file system per disk. Chunkserver hardware explains why the disks are given to MooseFS whole, as separate drives without RAID underneath and as one partition each. This section describes how to prepare and mount them.

On Linux, give every data disk a single partition and put the file system on that partition. On FreeBSD a ZFS pool uses the whole disk. The only requirement is a file system with basic POSIX behavior. Chunkserver hardware lists the suggested file systems and describes how a ZFS pool is laid out. Do not put chunk storage on network storage such as NFS or iSCSI. Network storage adds its own latency and failure modes between the Chunkserver and what the Chunkserver treats as a local disk.

When you format the disk, keep the default inode density of the file system. Each chunk is a file that holds at most 64 MiB. An instance with many small files holds many chunks far smaller than that. The Chunkserver watches the free inode count and lowers the space that it reports to the Master Server when inodes would run out before bytes. Formatting presets that reserve one inode per several MiB, such as mkfs.ext4 -T largefile4, can therefore cost usable space.

Mount each disk at its own mount point. Name the mount point so that the physical drive can be found from the name: /mnt/chunk01 for the drive in bay 1, or names that include the disk type (/mnt/ssd01, /mnt/hdd01) in mixed machines. When a Chunkserver reports errors on a mount point, that name identifies the drive to pull. In /etc/fstab, identify the disk by UUID or label, not by device name: /dev/sdb can point at a different drive after a reboot or a hardware change. The example uses parted to partition the disk. Any partitioning tool works. The example formats /dev/sdb1, the first partition of a SATA or SAS disk. On an NVMe drive the same partition is /dev/nvme0n1p1.

parted and mkfs.xfs come from the parted and xfsprogs packages. A minimal install includes neither. Without them the example fails with command not found, so the disk stays unformatted.

parted -s /dev/sdb mklabel gpt mkpart primary 0% 100%
mkfs.xfs /dev/sdb1
mkdir -p /mnt/chunk01
echo "UUID=$(blkid -s UUID -o value /dev/sdb1) /mnt/chunk01 xfs defaults 0 2" >> /etc/fstab
mount /mnt/chunk01

On FreeBSD the disk goes into a pool of its own. The mount point is set on the pool, so there is no /etc/fstab entry to write. The example uses /dev/da1, the second SCSI disk:

zpool create -m /mnt/chunk01 chunk01 /dev/da1

Use each mount point for MooseFS only. The Chunkserver reports the total and free space of the file system to the Master Server as chunk space. Anything else stored on the same file system is therefore counted as space for chunks that is not there. A disk that must be shared, with another process or another Chunkserver process, is marked as shared in its mfshdd.cfg entry. The entry forms and space limits are listed there. Listing the mount points in mfshdd.cfg is part of installing the server components.

To verify, reboot the machine and confirm that every data disk is mounted again at its intended mount point:

df -h /mnt/chunk01

Make the Master Server name resolve on every machine​

Every component finds the Master Server by resolving one host name, set per component (for a Chunkserver, MASTER_HOST in mfschunkserver.cfg). The packaged configuration uses mfsmaster unless you change it. Create the DNS records for that name before you install anything. Every machine of the cluster has to resolve the name: Chunkservers, Metaloggers, and clients, not only the machines that administrators work from.

With a single Master Server, one address record is enough:

mfsmaster    IN A    192.0.2.11

With several Master Servers PRO (PRO Edition only), the same name has one record per Master Server. The cluster elects its Leader from the resolved set. mfssupervisor finds the Master Servers the same way (mfssupervisor(8)). DNS carries the list of Master Servers covers the mechanism and what an incomplete or stale answer does to the cluster.

mfsmaster    IN A    192.0.2.11
mfsmaster    IN A    192.0.2.12
mfsmaster    IN A    192.0.2.13

Without a DNS server, entries in /etc/hosts on every machine give the same answers, at the cost of updating every machine by hand whenever an address changes.

To verify, resolve the name on each machine and confirm that the full set of addresses is returned:

getent hosts mfsmaster

On FreeBSD that command returns one address even when the name has several. On FreeBSD, read the set with host mfsmaster, which queries DNS. For a name kept in /etc/hosts, read the file itself, because host does not look there.

Every machine keeps its clock in step​

Run a time service on every machine of the cluster and leave it running. Which service to run depends on the system, not on MooseFS: systemd-timesyncd, chrony, and ntpd all keep a clock synchronized.

The cluster reads those clocks and reports what it finds. The Master Server stamps every file system operation with its own clock, so the timestamps in the tree come from that one machine (timestamps and what moves them). In MooseFS PRO PRO (PRO Edition only), the difference between the clock of a Follower and the clock of the Leader is a separate reading. The cluster raises a warning on a small difference and an error on a larger one. mfssupervisor names the pair of Master Servers whose clocks differ. The messages are listed in Alerts. Master Server problems describes what to do about them. Some of the work that elects a new Leader happens at agreed moments. Chunkserver clocks that drift from the clock of the Master Server can therefore delay the election that follows a Leader's failure.

The machines that only mount the file system have no reading of their own and no threshold. Their clocks still matter. A client stamps nothing: every timestamp in the tree comes from the Master Server. A client compares those timestamps against its own clock. A machine whose clock is off therefore reads file ages, find -mtime, and any tool that builds by modification time against the wrong reference.

On a system with systemd, read the state on each machine:

timedatectl

Confirm the line System clock synchronized: yes. On a system without systemd, read the state from the time service that the system runs.

Give the Master Server RAM for all metadata​

The Master Server keeps the entire metadata in RAM and answers every client from a single thread. Size the machine so that the whole metadata fits in physical memory (Master Server RAM from the object count). Swap does not make up the difference, because a Master Server that starts swapping answers at the speed of a disk on the one thread that every client waits on.

On Linux, let the kernel overcommit memory. The Master Server saves its metadata image from a forked copy of its own process. When the kernel refuses to overcommit, that fork fails once the process is large. The Master Server then saves in the foreground instead. The foreground save blocks all operations until it finishes. Set the policy now and make it persistent:

sysctl -w vm.overcommit_memory=1
# /etc/sysctl.conf
vm.overcommit_memory = 1

On systems that ship no /etc/sysctl.conf, Debian 13 among them, put the same line in a file of your own under /etc/sysctl.d/. To verify, run sysctl vm.overcommit_memory after a reboot and confirm that it prints 1.

Run the Master Server CPUs at full frequency​

The Master Server's operation rate depends on a single core, so frequency scaling shows up directly as metadata latency. Master Server hardware explains why the machine is configured for maximum performance with power saving off. Set the CPU governor to performance, either through the tuned service or directly.

With tuned:

apt install tuned            # Debian, Ubuntu
yum install tuned            # RHEL family
systemctl enable tuned
systemctl start tuned
tuned-adm profile latency-performance

tuned has no profile called performance. A request for that profile is refused with Requested profile 'performance' doesn't exist. Three of the profiles that tuned does ship set governor=performance. The steps above use latency-performance, because it also keeps the processor out of its deeper idle states. throughput-performance sets the same governor and tunes for bulk transfer. accelerator-performance sets it for machines with accelerator cards. tuned-adm list prints all profiles, and tuned-adm active names the active profile.

Without tuned, set the governor for every CPU:

for cpu in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
    echo performance > "$cpu"
done

A governor set this way lasts until the next reboot. tuned reapplies its profile at boot. A manual setting has to be repeated from the system's startup scripts.

To verify:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

prints performance.

When every item passes, the packages go on​

What has to hold on the machines, section by section:

  1. The system: every machine runs a supported system and is fully up to date.
  2. The repository: the package manager on every machine finds the MooseFS packages (verify the repository).
  3. The Chunkserver disks: each data disk has a file system of its own, is mounted again at its own mount point after a reboot, is identified in /etc/fstab by UUID or label, and holds nothing but chunks.
  4. The Master Server name: getent hosts mfsmaster returns the complete set of addresses on every machine of the cluster, Chunkservers, Metaloggers, and clients included.
  5. Memory on the Master Server host: the whole metadata fits in physical memory, and vm.overcommit_memory reads 1 after a reboot.
  6. CPU frequency on the Master Server host: the governor reads performance, and something sets it again at boot.

With the operating system prepared on every machine, continue with installing the server components.