Skip to main content

POSIX compliance

Through a mount, MooseFS behaves like an ordinary file system. This page says how far that goes.

What it is​

POSIX is a standard that describes the expected behavior of a file system and of the calls that a program makes into the file system. A program written against that standard runs on any file system that follows the standard, with no change made for that file system. From a mount, MooseFS is a POSIX file system. The file system is a directory hierarchy reached with the ordinary calls. Several names may lead to the same file. The hierarchy holds every object type that a Unix program expects: regular files, directories, symbolic links, FIFOs, sockets, and block and character devices. Objects have the attributes that go with them: owner, group, permission bits, and timestamps. MooseFS also supports extended attributes, POSIX ACLs, and both families of advisory lock. The Master Server coordinates these across the whole cluster rather than on one machine. See extended attributes, ACLs, and file locking.

MooseFS then adds behavior that POSIX does not describe. Each addition has a page of its own:

  • Snapshot: a file or a whole subtree copied without copying the data.
  • Quota: a limit on how much of the file system one directory subtree may consume.
  • Storage Class and Patterns: per-object rules for how the data is kept redundant and which servers hold it.
  • Extra attributes: per-object flags with no POSIX counterpart.
  • Trash: a deleted file stays recoverable for a retention time that you set on files and directories. A retention of zero keeps a subtree out of the trash. The trash also changes what unlink does to the last name of a file.

Why it matters​

The POSIX interface lets existing software store its data in MooseFS unchanged: nothing to link against, no protocol to speak, no format to convert. The POSIX interface also fixes the vocabulary, so the tools and the habits of a local file system carry over. The cost of a call does not carry over. A metadata operation that the client cannot answer from its own caches costs a round trip to the Master Server. A workload made of many small operations therefore waits for the network on each operation.

How MooseFS implements it​

POSIX was written for a single machine. MooseFS is a cluster, so every decision that the standard makes for a local file system has to be carried over to a network. Those decisions include what a client may hold in its own cache, how a change reaches the other mounts, and where a lock is kept. The Master Server decides all of that. The Master Server keeps the metadata and the locks for the whole instance. The Master Server also tells each client what that client may cache. Behavior that a local file system gets from its kernel is therefore a design decision here, taken one case at a time. MooseFS is tested with the tools available for checking a file system against POSIX. Those tests pass.

The client is a FUSE file system, so the kernel passes every call made inside the mount to the client. The client implements the operations that a POSIX program uses and forwards each one to the Master Server or to the Chunkservers. Those operations are lookup, reading and setting attributes, creating each kind of object, linking, renaming, opening, reading, writing, fsync, statfs, the extended-attribute calls, and the lock calls.

The Master Server makes the permission decisions. It tests the identity in the request against the object's permission bits, or against the object's ACL when an ACL is set. FUSE gives the client only the calling process's user ID and primary group. The client therefore reads that process's supplementary groups from the operating system and sends the whole list with the request. A mount whose group cache is switched off sends the primary group alone. The Master Server evaluates membership on that group alone.

The Windows Client PRO (PRO Edition only) follows POSIX semantics as far as Windows allows. Like any other client, the Windows Client connects to the instance and works on the same files and directories that POSIX mounts see. Windows and Unix-like systems do not represent permissions and file attributes the same way. That difference limits what the driver can reproduce. Permissions mapping says how the two permission models are translated.

The trash changes one POSIX rule: what unlink does to the last name of a file. The standard frees the space and puts the file out of reach once its link count is zero and no process holds the file open. MooseFS moves the file to the trash instead. The space stays counted. The file stays recoverable through a metadata mount for as long as its retention lasts, a day by default. Setting the retention of a subtree to zero, with mfssettrashretention, keeps that subtree out of the trash. By default, an empty file is left out too. MooseFS destroys the empty file at once, whatever its retention says. Trash retention says when an empty file does reach the trash.

Defaults and limits​

  • A client caches attributes and directory entries for a short time. A change made through one mount therefore becomes visible through another mount only after the cached copy expires. An extra attribute on the object removes that delay. With that attribute set, the Master Server tells the client not to cache the object.
  • Another mount is guaranteed to read data written through one mount only after the writing process calls fsync() or close(). Until then, what the reading mount gets is undefined. Multiple mounts and parallel access describes how a write reaches the other mounts.
  • The Master Server decides at each open whether a client may cache the contents of a file. After another session modifies the file, the Master Server withdraws that permission. The next open elsewhere returns without the permission.
  • Access time updates follow a cluster-wide setting, not a setting of each mount. The setting ranges from an update on every access to no update at all.
  • A name may be at most 255 bytes. The number of hard links to one object is capped cluster-wide. A few names beginning with a dot are reserved in the root of a mount and cannot be created there.
  • Locking is cluster-wide only when the mount asks for cluster-wide locking. A mount can be told to keep either family of lock local to its own machine.
  • An export can refuse chosen operations outright. A call that POSIX allows may therefore still be denied by policy.