Skip to main content

Kubernetes

A Kubernetes cluster reaches MooseFS the way every other client does: through a mount made by the MooseFS client. Kubernetes adds a question: who makes that mount, and when. Two arrangements answer this question:

  • The MooseFS CSI driver makes the mounts itself and presents them as PersistentVolume objects. Dynamic provisioning requires exactly that. This is the recommended arrangement.
  • A node mount handed to pods as hostPath is a mount kept on each node. It needs nothing beyond the client package. This is the simpler arrangement, for simpler tasks.

This page covers what the two arrangements share: what the MooseFS side must allow before either arrangement works, and how a pod mounts the claim.

The MooseFS servers themselves stay outside the integration. Today they run outside Kubernetes as well. The CSI driver requires one thing: a MooseFS instance that is running and reachable over the network. The driver neither installs a MooseFS instance nor manages one. MooseFS provides no manifest, operator, or Helm chart that brings up a MooseFS instance inside a Kubernetes cluster. Running both on the same nodes is possible. Keeping the Chunkservers off the nodes that mount is a matter of configuration. Still, prefer a dedicated instance that the driver and its workers only connect to. Pairing the two systems on one set of machines means reconciling the failure handling of both before any data is at stake. The page on bare metal, virtual machines, and containers covers where the Master Server and the Chunkservers run. That page also describes what changes about the needs of a server component in a container, and what does not.

What a Kubernetes cluster gets from MooseFS​

  • Shared read-write access. Many nodes mount the same tree at the same time. All of them see the same namespace. The Master Server holds locks taken through a mount for the whole instance, not inside one client (file locking). Two pods on two nodes therefore contend for a file the way two processes on one machine do.
  • Redundancy that survives a node. The number of copies, or the Erasure Coding scheme that protects the data, is a property of the directory tree that contains the volumes (Storage Class). This property does not depend on where a pod runs.
  • Capacity added by adding machines. Space grows with the Chunkservers attached to the instance, without a change on the Kubernetes side.
  • Standard servers rather than storage appliances. Hardware covers what each role needs from its machine.

The integration does not change the instance itself. The Master Server holds the metadata. The Chunkservers hold the data. A client process mounts the tree (Components). On the Kubernetes side, storage reaches a pod as a PersistentVolume bound to a PersistentVolumeClaim.

Admit the nodes before anything else​

A mount from a Kubernetes node is checked like every other mount, against the entries in mfsexports.cfg on the Master Server. An entry must cover the addresses that the nodes mount from, and the part of the namespace that the nodes are meant to reach. A client that no entry covers is refused at mount time (session and export). Two options on that entry decide what the integration can do:

  • alldirs lets a client mount a subdirectory of the exported path. The CSI driver mounts one directory: the directory that its configuration names as the root of every volume that the driver creates. The driver needs alldirs whenever that directory is below the exported path. Without the option, the driver works only when that directory is the exported path itself.
  • admin decides whether a client may run the operations that change the cluster. Setting a quota is one of these operations. That operation requires more than this option: the entry must also be read-write and leave root unmapped. The linked page describes these conditions. An entry that maps root refuses a quota even with admin on it.

The file's first entry, * / rw,alldirs,admin,maproot=0:0, includes both. In a narrowed file, either option can be missing. Access control covers what the options do and how to write the entries.

The nodes also need a route to the whole instance, not to the Master Server alone. A client talks to the Master Server for metadata and to every Chunkserver for data. Network ports lists the port that each connection uses.

POSIX ownership on a volume decides what a non-root pod may write​

MooseFS keeps the owner and the group of every object, and enforces the permission bits on them. A directory created by a client that runs as root belongs to root. A pod that sets runAsUser to anything else cannot write to that directory. The Master Server holds the ownership, so the ownership is the same for every node and every pod that reaches the object. The workarounds differ in how much separation each one gives up:

  • Change the ownership before the application starts. An init container that runs as root can chown the mounted path. Kubernetes runs that container to completion before the application container starts.
  • Map the identities at the export. maproot=UID[:GID] makes the Master Server treat uid 0 from that mount as the user that you name. mapall=UID[:GID] does the same for every other user on the mount (access control). The mapping covers the whole export entry, so the mapping applies to every pod on every node that the entry admits.
  • Take ownership out of the decision. The noowner extra attribute makes an object report the uid and gid of the process that accesses the object. Every process then counts as the owner and passes the owner check. Only uid 0 gets the real values. Set the attribute with mfsseteattr, documented in mfseattr(1). A directory that has the attribute passes it to every new object created in that directory:
mkdir -p /mnt/mfs/pv_data
mfsseteattr -r -f noowner /mnt/mfs/pv_data
warning

noowner stops ownership from separating anyone inside that tree: every user of every mount that reaches it passes the owner check on every object. Set it on the subtree the volumes live in, never on the root of the instance. Prefer mapped identities or an init container wherever one of them fits.

A mount's session outlives the pod​

Every mount, whether a node made it or the driver did, registers with the Master Server and gets a session. A lost network or a stopped node looks like a client that disconnects without unmounting. Such a client leaves its session sustained for the time that SESSION_SUSTAIN_TIME in mfsmaster.cfg sets. This time is one day unless the setting is changed. A client process that survives and reconnects registers with the same session identifier and finds its files still open. A node reboot produces a new mount. That mount registers as a new session. Nothing returns to the sustained session. Until that time runs out, the sustained mount stays listed among the instance's mounts. The files that it held open stay open. A cluster that reschedules pods often therefore lists more sessions than it has running mounts. The CLI and the GUI both have a command that removes a sustained session before its time runs out.

Mount the claim in a pod​

After the claim is bound, a pod mounts it by name. This works the same for both arrangements:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mfs-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mfs-app
  template:
    metadata:
      labels:
        app: mfs-app
    spec:
      containers:
      - name: app
        image: your-app-image
        volumeMounts:
        - mountPath: /data
          name: mfs-storage
      volumes:
      - name: mfs-storage
        persistentVolumeClaim:
          claimName: mfs-pvc

Both replicas write to the same tree at the same time, as two clients of one instance.

What to settle before production​

  • Redundancy. Give the tree that contains the volumes a Storage Class that matches the value of the data. Data placement tasks covers the classes and how to assign them.
  • Space. The two arrangements limit the space that a volume uses in different ways. Under hostPath you put a quota on the tree yourself. In contrast, the CSI driver sets a quota for each volume that it creates.
  • Network. The client exchanges data with the Chunkservers directly, so throughput and latency between the nodes and the cluster decide the performance that the pods get. Budget them with bandwidth and latency.
  • Traffic in the open. No transport between MooseFS modules is encrypted, with no option to turn encryption on (authentication between modules). The layer below protects traffic that leaves a trusted network: a VPN or a comparable tunnel between the machines. This layer is transparent to both sides of a MooseFS connection.
  • Backups. Redundancy inside the instance is not a copy kept somewhere else. Copies of the metadata leave the machine as a dump or through a Metalogger. The patch tools PRO (PRO Edition only) transfer the changes of a directory to another instance, onto a directory that is on a MooseFS mount.