What metadata holds and what it costs in RAM

The Master Server keeps several kinds of record in memory. Each kind has its own size, its own allocator, and its own reason to multiply. For what metadata is and why the memory of one machine limits the whole cluster, see Metadata and namespace and Master Server memory.

Each record kind gets a section in the dump

The dump that the Master Server writes to disk is a sequence of named sections. Each section is named after one kind of record, has its own format version and its own length, and is read back by name:

Section	What it carries
SESS	Sessions: one record per registered client, and the export rights it registered with.
SCLA	Storage Class definitions.
PATT	Patterns and what each of them forces.
NODE	The objects: one record per file, directory, symbolic link, FIFO, socket, or device.
EDGE	The names: one record per name in one directory.
FREE	Inode numbers released by deletion and not yet available again.
QUOT	Quotas: one record per directory that carries one.
XATR	Extended attributes.
PACL	POSIX ACLs.
OPEN	Which session holds which inode open.
FLCK	BSD locks taken with flock.
PLCK	POSIX record locks taken with lockf.
LOCA	Locations and the address ranges assigned to them.
CSDB	Every Chunkserver that the cluster has registered, connected or not.
CHNK	The chunks: identifier, version, and the classes of the files using them.

The write order is decided by the references between sections rather than by importance. SCLA and PATT come before NODE, because a node refers to its Storage Class by number. The definition must exist before a node can refer to it. XATR and PACL come after NODE, because they attach to inodes. FLCK and PLCK come after OPEN, and CSDB after LOCA, for the same reason. Changelogs, dumps, and the recovery point covers how the dump is written to disk and what happens between two dumps.

A node and an edge are separate records

A node has a common part with the fields that every object has: inode number, the three timestamps, owner and group, permission bits, type, Storage Class, extra attributes, and trash retention. After the common part comes a tail that depends on the type of the object. A directory node holds the running totals for its subtree and a pointer to its quota record. A file node holds the length and the chunk table. A symbolic link node points to its target text. The target text is allocated separately and sized by its length. A device node holds the device number. A FIFO or a socket holds only the common part and the link count. The Master Server allocates each node in one of five sizes according to its type, so a directory and a file do not cost the same.

An edge is one name under one parent. The size of an edge depends on the length of the name and grows in steps of eight bytes. A long name therefore costs more than a short one. The edge also holds the links that the Master Server follows from a parent to its children and from a child to all of its parents.

Hash tables index both kinds of record. Each table doubles in size and rehashes gradually as it fills. Every object and every name therefore also costs an index entry.

Neither allocator returns memory to the operating system. Records are allocated from large blocks. A freed record goes onto a free list for its own size class. Its memory is then reused for the next record of that size. Deleting a million files therefore lowers the memory that the Master Server reports as used, but not the memory that the Master Server holds.

The chunk registry grows with chunks, not files

A chunk costs a record in the registry, whatever the file that it belongs to looks like. The record holds the identifier, the version, the flags, the lock state, and the Storage Classes of the files that refer to the chunk. A file node points to a chunk table that lists the chunks in order, eight bytes per chunk. Like everything else, the table is allocated in rounded size classes, so this part of the cost depends on the total stored size rather than on the number of objects.

In addition, the Master Server keeps one list entry per stored part per server: one for each copy, and one for each Erasure Coding part. A chunk kept as eight data parts and two parity parts therefore has five times as many list entries as the same chunk kept in two copies. In both layouts, the list entries come on top of the one registry record of the chunk.

Deletion does not release metadata at once

A file that loses its last remaining name keeps its node, its chunk table, and its chunks. The file moves to the trash tree when trash retention sends it there. In the trash tree, the file gets a new edge. The name on that edge is the whole path that the file had, rather than its last component. A trashed file costs slightly more than the same file did in the namespace. A file that goes to the sustained area instead has the same records. It keeps them until the last session closes the file. Names: link, unlink, rename, and exchange covers when a file goes to the trash and when it goes to the sustained area.

The inode number is released last. It goes onto the list of deleted nodes. It becomes available again only after the delay set by INODE_REUSE_DELAY. This delay prevents a client from resolving a cached inode to a different object.

Some records track activity, not storage

A session record stays after its client disconnects. It is discarded only after SESSION_SUSTAIN_TIME, so the number of sessions is not the number of mounts currently connected. Each inode that a session holds open is a further small record, indexed from both sides. Each lock held is a record under the inode that it covers. The Master Server counts BSD locks and POSIX record locks separately, because it keeps each family in a table of its own. Each POSIX ACL is a record per object per ACL type. The record includes a table of the named users and groups.

Storage Classes and patterns are the exception. Both are kept in fixed tables that are sized once for the maximum the product allows. Defining one of them therefore costs as much as defining all of them.

Extended attributes grow metadata fastest

Extended attributes are the only part of metadata that a workload can grow without limit and without noticing. An application that sets attributes on every file it writes multiplies metadata far beyond what the file count suggests. All of this metadata is held in the RAM of the Master Server. Count extended attributes when you size that machine (see Requirements for Master Servers).

An object that has any extended attribute gets an entry in a separate table, keyed by inode. Each name–value pair on that object is a further record inside the entry. Names and values are interned. Identical strings are stored once and reference-counted, so the same attribute name on a million files costs one copy of the name. However, the record for each pair and its two references are still stored a million times.

Extra attributes do not work this way and are not part of this calculation. Extra attributes are a field inside the node, one byte wide whatever you set. This is the practical difference that Extended and extra attributes covers.

The Master Server measures itself structure by structure

The Master Server reports its metadata memory broken down by structure. For each structure, the Master Server gives the memory allocated and the memory actually in use. The Master Server also gives the utilization calculated from these two values and the share of the total that the structure uses. The structures reported are Chunk hash, Chunks, CS lists, Edge hash, Edges, Node hash, Nodes, Deleted nodes, Chunk tabs, Symlinks, and Quota.

That list covers the metadata structures that the allocators track, not everything that the process holds. Extended attributes, POSIX ACLs, locks, open files, sessions, Storage Classes, and patterns are outside it. The list also leaves out the memory that the process uses for anything other than metadata. On a Leader, the total of the breakdown is therefore lower than the resident memory of the process. The resident memory is reported as a separate figure. On a Follower, the total of the breakdown can be the higher of the two, because a table that was reserved and never written is not resident.