Block device operations
This page covers working with MooseFS Block Device. The tasks are starting the mfsbdev daemon, mapping a file stored in MooseFS as a local block device, putting a file system on the device, resizing the device, unmapping it, and bringing mappings back after a restart. The client forms page describes the Block Device as a form of the client.
The tool needs Linux, nbd, and root
mfsbdev works on Linux only. It serves devices through the kernel's nbd (Network Block Device) module, so the module must be loaded before the daemon starts:
modprobe nbd
The daemon refuses to start when no /dev/nbdX device is present or accessible. The daemon reports which of the two cases applies. A missing device means that the module is not loaded. A permission error means that the daemon needs root privileges.
Start the daemon before any mapping
mfsbdev start connects to the Master Server and goes into the background. At this point, the daemon only listens on a Unix socket, by default /dev/mfs/nbdsock. Every other mfsbdev command communicates with the daemon through that socket.
mfsbdev start -H mfsmaster.example.com -x /etc/mfs/mastersecret
-H names the Master Server (default mfsmaster), and -P names its client port (default 9421). -x reads the master password from a file. -p takes the password on the command line. The two options exclude each other. Without either option, the daemon connects without a password. -S makes a subdirectory of the MooseFS tree the root for all mappings (default /). -o accepts mfsmount-style tuning options, such as write cache and read-ahead sizes. mfsbdev(8) lists all options. A daemon started with -l listens on a different socket. Every later command for that daemon needs the same -l. In the examples on this page, substitute your Master Server's name for mfsmaster.example.com. /etc/mfs/mastersecret represents a file that you created beforehand and that holds the master password. On a cluster without a password, omit -x.
To verify, run mfsbdev list. If the daemon is running and no mapping exists yet, the command prints nothing. If the daemon is not running, the command prints can't connect to socket '/dev/mfs/nbdsock' instead.
Map a file as a block device
mfsbdev map turns one file stored in MooseFS into a block device on the local machine:
mfsbdev map -f volumes/vm1.img -s 100Gi -n vm1
The daemon answers with the mapping it created:
started block device: (/dev/mfs/vm1->/dev/nbd0 : MFS:/volumes/vm1.img : 100.000GiB)
-f names the path inside MooseFS, relative to the root that the daemon was started with. With -s, the file is created when it does not exist yet. The directories on that path are not created. Create them on a mount first. Without -s, the file must exist. The device then gets the size of the file. -s accepts SI and IEC suffixes (100G, 100Gi). The device size is rounded down to a whole number of blocks. The block size defaults to 4096 bytes. -b sets a different block size.
The daemon picks the first free /dev/nbdX device unless -d names one. The device number depends on which devices were free at map time, so address the device through the symbolic link that the daemon creates in /dev/mfs. -n names the link. Without -n, the name is built from the master host, port, and file path. In the link name, characters other than letters, digits, ., and - are replaced with _.
-r maps the device read-only. The same file can be mapped read-only in several places at once.
A read/write mapping locks the file cluster-wide
When the daemon maps a file, it takes a BSD-style lock on that file: shared for a read-only mapping, exclusive for read/write. The Master Server holds file locks for the whole instance, so the lock applies to every client. A second read/write map of the same file, on this host or on any other host, fails with MFS file ... is locked (likely mapped elsewhere).
map -i proceeds even when the lock cannot be taken. A file that is reached through a read-only export can be mapped only with -r and -i together. The lock itself cannot be taken through such an export.
The exclusive lock keeps the mapped file to a single writer. A local file system such as ext4 or XFS assumes that it has its block device to itself. A file system written through two devices at once is corrupted. Use -i only for devices that nothing writes to.
Verify a mapping with list and lsblk
mfsbdev list shows every mapping that the daemon has:
mfsbdev list
file: volumes/vm1.img ; device: /dev/nbd0 ; link: vm1 ; size: 107374182400 (100.000GiB) ; blocksize: 4096 ; rwmode: rw ; ignore_locks:no
-t m and -t u print each mapping as a ready-to-run map or unmap command instead.
The device is also visible to the system, under its nbd name and through the link:
lsblk /dev/nbd0
ls -l /dev/mfs/vm1
Put a file system on the device
The mapped device is raw. Create a file system on the device, then mount the device like any other block device:
mkfs.xfs /dev/mfs/vm1
mkdir -p /mnt/vm1
mount /dev/mfs/vm1 /mnt/vm1
lsblk -f /dev/nbd0 then shows xfs as the device's file system and /mnt/vm1 as its mount point.
Resize a mapped device
mfsbdev resize changes the size of a device while the device stays mapped. The command needs Linux kernel 4.18 or higher. On an older kernel, the daemon answers can't resize block device - kernel 4.18+ is needed.
mfsbdev resize -n vm1 -s 200Gi
Select the device with -f, -d, or -n. Without -s, the device gets the current size of the mapped file. The new size is rounded down to a whole number of blocks. The resize changes the device alone. A file system on the device keeps its size until you grow that file system with its own tools.
mfsbdev list confirms the change. After the resize above, the line of the mapping reads size: 214748364800 (200.000GiB). On a file system mounted from the device, df -h /mnt/vm1 still reports the old capacity of the file system.
Unmap a device that is no longer used
Unmount anything mounted from the device, then remove the mapping. Select the mapping with -f, -d, or -n:
umount /mnt/vm1
mfsbdev unmap -n vm1
Unmapping disconnects the nbd device and releases the lock. The file stays in MooseFS with everything written to it and can be mapped again. mfsbdev list no longer shows the mapping.
Mappings end with the daemon
The daemon keeps its mappings in memory. mfsbdev stop, a TERM signal, or a reboot of the host disconnects every mapped device. A daemon that is started later has no mappings. Mappings therefore do not survive a restart on their own. The daemon recreates them at startup when start is given a mappings file with -i.
The file uses the mfsbdev.cfg format. Don't write the file from scratch. Create the mappings once with map, then have list -t i print them in that format and save the output.
mfsbdev list -t i > /etc/mfs/mfsbdev.cfg
After a restart, start the daemon with the saved file:
mfsbdev start -H mfsmaster.example.com -x /etc/mfs/mastersecret -i /etc/mfs/mfsbdev.cfg
The daemon creates every mapping listed in the file before it begins serving commands. If one of the mappings cannot be created, the daemon reports the reason and exits. The daemon does not run with only part of the set.