Skip to main content

Where MooseFS does not fit

MooseFS spreads file data over many machines and keeps the metadata of the whole instance on one machine. Some workloads suit that design and some work against it. This page names the workloads that work against that design, the rule that excludes each, and how to get the same result another way. Where MooseFS fits covers the workloads that this split of data and metadata suits.

Metadata work does not spread over more machines​

The rate at which a cluster answers metadata operations is the rate of one machine. A single Master Server serves all of its clients from one loop, whatever the rest of the cluster looks like. In MooseFS PRO the Followers replicate the Leader's metadata and refuse client sessions PRO (PRO Edition only), so the Followers add availability but no metadata throughput. Adding Chunkservers adds capacity and data bandwidth but leaves the operation rate unchanged (limits and scale).

A workload whose cost comes from the number of file system operations rather than from the bytes they move gets no faster as the cluster grows. The client's own caching keeps part of that work off the network. A repeated lookup or attribute read is answered on the client machine as long as its cached copy is still valid, and never leaves that machine (what the FUSE layer costs). Operations that change something are not answered on the client. Each of them reaches the Master Server, which records the operation.

Small files cost more than their bytes​

Every file, directory, and link is a record in the Master Server's memory, whatever that object holds. Every chunk that holds data also has a minimum size. The cluster accounts at least 72 KiB for such a chunk, once for every copy kept. Quotas count that size. On the Chunkserver's disk, a small chunk occupies much less, because the chunk file is written sparsely. A chunk that holds 15 bytes occupies 12 KiB on a file system with 4 KiB blocks: 8 KiB of chunk metadata and one block of data (small files). Both costs follow the number of objects and chunks and not the volume of data. Millions of small files therefore fill the memory of one machine and occupy more space than the data they store. The number of objects that one instance can hold also has a fixed ceiling (limits). A growing cluster reaches the memory limit of the Master Server before that ceiling (Master Server memory).

You size for this cost rather than avoid it. Count the objects and the chunks that the workload creates. Price the machines from those counts (start from the workload, Master Server RAM from the object count). Erasure Coding does not lower this cost for small files. Every erasure-coded part is a chunk file with its own header, so a small file kept in parts can occupy more disk space than the same file kept in copies (part headers give the saving back on small files). A Storage Class can keep every file below a set length in copies.

One mount moves only what one channel carries​

Every operation of a mount, and every byte that the mount reads or writes, crosses one channel between the kernel and the mfsmount process that serves the mount. The speed of that channel is the ceiling on what the mount moves. That speed is a property of the client machine. Hardware added to the cluster does not raise that speed (the per-mount throughput ceiling).

Split work that needs more from one machine across several mounts. Each mount has its own channel, session, and set of caches (multiple mounts). Such work can also bypass the kernel. An application that links libmfsio becomes the client itself. mfsbdev presents one stored file to the machine as a block device on top of the same library (clients). The library also has costs. The application has to be written for the calls of the library. Nothing appears in the machine's namespace for other tools to use.

Erasure Coding does not fit changing data​

A Storage Class says how data is kept in each storage state separately. An Erasure Coding definition is accepted only for ARCHIVE and TRASH. A class that requests erasure-coded parts in CREATE or KEEP is refused when you define it. Data in use is therefore held as whole copies and costs what copies cost (choosing between copies and Erasure Coding).

The savings come later. A class whose ARCHIVE definition names a scheme converts the chunks of a file once that file has stopped changing. The raw space is freed at that point. A workload that keeps rewriting everything never reaches that point: price it at the cost of copies.

A cluster divided into locations PRO (PRO Edition only) holds data on both sides. A write into a chunk that already has copies on both sides is confirmed to the application only after the far copy has accepted the write. The copies are written along one chain. No copy fills the remaining copies in afterwards. The round trip of the link is therefore inside every such write. Bandwidth does not remove that round trip. Clients in the site that does not hold the Leader also cross the link for every metadata operation that reaches the Master Server (the WAN cost of writes).

A class can keep new data off the link by creating copies in the writing site and keeping them in both sites. This creates a period of exposure. Until the far site has its copy, the new data exists in one site only. If a site is lost inside that period, the new data is lost with it.

Nothing between the modules is encrypted​

No pair of MooseFS modules encrypts its traffic. File data and metadata are sent in the clear. The monitoring interfaces are answered without any credential. The commands of those interfaces that put a Chunkserver into maintenance or drop a session are answered without a credential too. The only requirement is reaching the Master Server's port (authentication between modules).

The layer below MooseFS handles encryption. MooseFS traffic is carried over a VPN or a comparable tunnel between the machines. Both ends of a MooseFS connection work the same with or without the tunnel. An instance whose traffic leaves a trusted network is therefore protected by that lower layer rather than by an option inside MooseFS. MooseFS checks for itself who joins and who mounts. Server modules can be required to use a shared code, an export entry can require a password, and chunk I/O can be made to carry a stamp of its own.