Skip to main content

The mfsio API

libmfsio is a C library that makes your own program a MooseFS client. The program calls mfs_open, mfs_read, mfs_write, and the related functions declared in the mfsio.h header. The program talks to the Master Server and the Chunkservers itself. Nothing is mounted, so the machine's namespace does not change. This page covers what the library needs, how a program connects through the library, and where the library falls short of a mount. Client forms compares the library with every other form of the client.

An application uses the library for two reasons. The first reason is throughput. Everything that one mount does goes through a single FUSE connection. An application that needs more than that from one machine reaches the per-mount throughput ceiling. The second reason is the mount point. A program that links the library needs no mount point on its host. What the FUSE layer costs describes both sides of that trade-off. One such program already exists: mfsbdev is built on the library. Block device operations covers running mfsbdev.

The client packages install the library and header​

The library is not a separate download. The MooseFS client packages install it: the shared object libmfsio.so.1 and the public header mfsio.h. Some packaging also installs a static libmfsio.a next to them. Some packaging splits these files into a runtime package that holds the shared object and a development package that holds the header and the link-time libmfsio.so. A build host then needs both packages, but a host that only runs the finished program needs the runtime package.

Where the compiler and the run-time loader already search the install directories, building against the library needs one flag:

cc myprog.c -o myprog -lmfsio

The install prefix can put the header or the library in a directory that the compiler does not search. In that case, add -I for the header directory and -L for the library directory. Add an rpath or LD_LIBRARY_PATH if the loader cannot find libmfsio.so.1 at run time. On FreeBSD, the header and the library are installed under /usr/local. That directory is never on the default search path:

cc myprog.c -o myprog -I/usr/local/include -L/usr/local/lib -lmfsio

A program opens and closes one session​

A configuration structure holds everything that the connection needs. mfs_make_cfg returns a structure filled with defaults. You replace the fields that you want to change. mfs_init opens the session. mfs_term closes it at shutdown. The string fields belong to the structure, so replacing a field means freeing the old value first. The structure itself can be released as soon as mfs_init returns.

#include <mfsio.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void)
{
    mfscfg *cfg = mfs_make_cfg();
    free(cfg->masterhost);
    cfg->masterhost = strdup("mfsmaster.example.com");
    if (mfs_init(cfg, 0) < 0) {
        fprintf(stderr, "cannot connect to the MooseFS instance\n");
        mfs_free_cfg(cfg);
        return 1;
    }
    mfs_free_cfg(cfg);

    int fd = mfs_open("hello.txt", O_CREAT | O_WRONLY, 0644);
    if (fd >= 0) {
        mfs_write(fd, "hi\n", 3);
        mfs_close(fd);
    }

    mfs_term();
    return 0;
}

The stage argument of mfs_init exists for programs that fork. Stage 1 connects before the fork. Stage 2 starts the worker threads after the fork. Stage 0, used in the example above, does both at once.

By default, the configuration points at host mfsmaster on the master client port listed in network ports, with / as the root. masterpath changes that root to a subtree. The program then reaches that subtree and nothing above it. A password goes in masterpassword as plain text, or in mastermd5pass as 32 hexadecimal digits. mfs_init uses the password and zeroes those bytes in place. masterhost also accepts several addresses separated by ; PRO (PRO Edition only). The client then follows the Leader redirect that it receives from whichever address answers.

Set mountpoint to a value that names your program. mountpoint is a label, not a path: the Master Server records it with the session. Session listings show this label in the place where a mount shows its mount point. An administrator tells your program apart from the mounts by this label. Programs that do not set mountpoint all register as [MFSIO].

Each process that links the library keeps buffers of its own inside its address space. The write cache is 128 MiB and is taken in full when the session starts. The read-ahead pool grows to 128 MiB when reads need it. Both figures are per-process defaults. A host that runs many such programs therefore uses this memory for each program.

Every call carries the caller's identity​

The library samples the caller's identity per call: the effective uid, the effective gid, the supplementary groups, and, for calls that create an object, the umask. The API offers no way to pass an identity of your own choosing. Data calls differ only in timing, not in source. mfs_read and mfs_write and their p-variants act with the uid captured when the descriptor was opened. mfs_readdir and mfs_readdirplus act with the snapshot taken at mfs_opendir.

This has two consequences for the design of the program. Identity is process-wide, so one process cannot act as several users. A service that handles requests for different identities gives each identity a process of its own. Also, at most 256 supplementary groups are sampled. A process that belongs to more groups falls back to its effective gid alone. Group-based permissions then deny what they would otherwise allow.

The Master Server decides the rest, in the same way that it decides for a mount. The session registers in exactly the way that mfsmount registers. The entry in mfsexports.cfg that covers the program's address therefore decides whether the session is read-only and whether a password is required. The same entry decides which subtree the session may reach and how root is mapped. Session and export covers what a session is admitted to do.

Code written for a mount needs changes here​

The library offers POSIX-shaped calls, not the POSIX calls themselves. A few things that a mount gets from the kernel are not available here. POSIX compliance covers how far MooseFS follows POSIX where a mount presents the file system. The differences below are those that a program meets when it works without a mount.

  • mfs_open returns a descriptor of the library's own, not a kernel file descriptor. Only mfs_* calls accept this descriptor. Code outside the library and child processes therefore cannot receive the open file.
  • There is no kernel between the program and its data, and therefore no page cache and no memory mapping. The library keeps its cache in the process.
  • FIFOs, sockets, and device nodes can be created and inspected, but not opened: mfs_open on such an object fails with EINVAL. Opening them is a kernel service, but there is no kernel here to provide it.
  • There is no working directory. Every path resolves against the configured root. That root is also a boundary: .. in that root fails with EPERM instead of staying in the root. An absolute symbolic-link target restarts at that root rather than at the real root of the instance.

Data-path failures block before they surface​

By default, a lost chunk or an instance with no space left does not fail a call at once. The data path retries, up to io_try_cnt times, 30 by default. ENXIO or ENOSPC reaches the caller only after the retries are used up. A single mfs_read or mfs_write can therefore stall for a long time first. This behavior suits a batch job but not a program that answers requests under a deadline. For such a program, set error_on_lost_chunk and error_on_no_space to 1 to fail immediately instead. Alternatively, lower io_try_cnt, or set io_timeout to the number of seconds after which retrying stops.

Writes are buffered as well. mfs_write copies into the process cache and returns. Worker threads send those blocks to the Chunkservers in the background. The program waits for that transfer at the flush, which is mfs_fsync or mfs_close. The flush also tells the program whether the transfer succeeded. A late ENOSPC, EDQUOT, or EIO is therefore returned there rather than at the write. Check what those two calls return. A program that only checks mfs_write can lose data that it treats as stored.

The installed header is the function reference​

Every mfs_* call is documented at its declaration in mfsio.h. The documentation lists the arguments of the call, its return value, and the errno values that it sets. The header that the client packages install is that reference. The sources include a Doxygen configuration. It renders those comments, a project overview, and a page of worked examples as browsable pages.