Master Server RAM from the object count
The Master Server keeps the whole metadata set in the memory of one process. The machine that runs the Master Server is therefore sized against two counts, not against any amount of storage. These counts are the number of objects and the number of chunks that the instance will hold. For the same reason, the ceiling on how far a cluster can grow is the memory of that one machine, not the capacity of the disks. What metadata holds covers what the set contains. Metadata in RAM covers how that set is answered from memory.
Neither count depends on the size of a file. A node record is the same size whatever the length of the file, because the length is a field inside that record. The length decides how many chunks the file uses, one for each 64 MiB of the file. For that reason, the two counts change independently. Limits and scale lists both together with other things that grow with a cluster. Start from the workload covers how to get both counts from what you intend to store.
Each object costs a node record and a name
An object costs one node record and one name record for every directory that the object appears under. The size of the node record depends on the type of the object. A hard link is a second name, not a second object.
| Record | Bytes |
|---|---|
| Node of a directory | 112 |
| Node of a regular file | 71 |
| Node of a symbolic link | 62, and the target text on top of that, in steps of 8 bytes |
| Node of a block or character device | 54 |
| Node of a FIFO or a socket | 50 |
| One name under one parent | 80 for a name of 8 bytes or less, and 8 more for every further 8 bytes |
A chunk costs a registry record and its parts
A chunk costs one record in the registry. The chunk also costs one slot in the chunk table of every file that points at the chunk. Each part of the chunk that a Chunkserver holds costs one further record. The registry record holds the number of files that point at the chunk in a field of its own. This applies when no more than four files point at the chunk and all of these files share one Storage Class. A fifth file, or a second Storage Class among these files, turns that number into a list of eight-byte records. The list has one record for each Storage Class involved.
| Record | Bytes |
|---|---|
| Chunk in the registry | 48, and 40 in the Community Edition |
| One slot in the chunk table of a file | 8 up to a file of 16 chunks. Above that the table is rounded up to a multiple of 16 slots through 256 chunks, and of 256 slots through 4096. A file of 17 chunks therefore takes 32 slots. The step multiplies by 16 at each further tier |
| One stored part: one copy, or one part of an Erasure Coding chunk | 16 |
Nodes, names, and chunks each carry an index
Each of the three kinds of record is indexed by a hash table of eight-byte slots. The number of slots is a power of two and starts at 16 777 216. The table doubles as soon as the records outnumber the slots. Every slot is allocated whether a record is stored in it or not. An index therefore costs the size of its table, not the size of its entries. Up to 16 777 216 records of a kind, the table stays at its starting size of 128 MiB. Above that, the table costs 16 bytes per record right after a doubling. That figure falls toward 8 as records are added.
A hundred million one-chunk files cost 25 GiB
Take a hundred million regular files. Each file has a name of eight bytes or less and is one chunk long. Each chunk is kept in two copies. The records of one file come to 239 bytes: 71 for the node, 80 for the name, 8 for the slot in its chunk table, 48 for the chunk record, and 32 for the two copies. By that count, each of the three index tables has doubled three times, to 134 217 728 slots and 1 GiB apiece. The index tables add another 32.2 bytes to every file. One file therefore costs 271.2 bytes, and a hundred million of them 25.3 GiB. The directories that hold these files come on top of that, at 213.5 bytes each if their names stay short.
An empty instance costs 300 MiB, and a real file about 300 bytes
Compare that arithmetic with two readings from running 5.x instances. An instance that holds no file and no chunk and has 30 Chunkservers attached reports 299 MiB of memory for its Leader. 300 MiB is therefore the fixed cost of the process. The cost of the objects is added to this fixed cost. An instance of 232 867 005 files in 52 GiB of RAM comes to 238 bytes per file after the fixed cost is subtracted. That figure is below the 271.2 bytes that the example above gives. Snapshots lower that figure. They raise the file count more than they raise the cost, because the copies share their chunks.
Plan against 300 bytes per file rather than the 271.2 of the example. The file in that example has a name of eight bytes and occupies a single chunk. Few real files are like that. 300 bytes allows for a longer name, for more chunks, or for both. That figure is the minimum for a given count, so put a margin over that minimum.
The save needs room above the metadata
The Master Server writes its metadata image from a forked copy of its own process (metadata save states). Both processes read the same pages of memory until one of them writes to a page. Every page written while the save runs is copied. The machine needs memory above the size of the metadata for as long as the save lasts, whatever the result of the arithmetic on this page.
The amount of extra memory depends on how fast the metadata changes during the save, not on how much metadata there is. An instance with a large, rarely written set copies few pages while its image is written. A smaller instance whose files change constantly copies many pages. Doubling is the bound that the copying cannot pass. Every instance stays below that bound. No arithmetic turns a metadata size into the headroom to leave free, because the rate of change depends on the workload rather than on the stored set. Leave room. Read what a given instance uses from its own figures. Prepare the operating system covers what happens when the fork cannot be made.
The figure covers metadata records and nothing else
Extended attributes, POSIX ACLs, locks, open files, and sessions are records of their own and are outside this arithmetic. The behavior of the applications sets the size of those records, not the amount of data that you store. Extended attributes are the records that a workload can multiply furthest past its file count. Estimate without extended attributes. Treat what they add as unknown. One workload keeps a few bytes in an attribute, but another workload keeps a kilobyte. Nothing in the stored set shows which of the two you have. The memory that the Master Server uses for anything but metadata is outside this arithmetic as well. This includes the changelog that the Master Server keeps in RAM. Two settings in mfsmaster.cfg limit that changelog: CHANGELOG_PRESERVE_MB at 500 MiB and CHANGELOG_PRESERVE_SECONDS at 5000 seconds by default. The Master Server drops its oldest changes once either limit is passed. The smaller of the two therefore decides how much stays in memory. Changes that a Metalogger or another Master Server has not yet received stay in memory past both limits. The operating system and every other process on the machine need their own memory on top of all this. The hardware to buy therefore needs a margin over this arithmetic. Master Servers lists the figure to plan against.
Every Master Server needs the same memory
A Follower Master Server holds the whole metadata set the same way the Leader does PRO (PRO Edition only). The Follower applies the same stream of changes to structures of its own so that the Follower can take over the role. Size every machine in the group against the same figure. Each machine has to hold the whole set, whichever machine answers clients at a given moment.
A running cluster reports the figure itself
An instance that already exists answers this better than any estimate made in advance. The Master Server measures its own metadata structures. It reports them in the Memory Usage section of the Info tab of the GUI, and on the command line as mfscli -SMU. Read the allocated figure rather than the used figure. The allocated figure counts each index table whole and does not go down when objects are deleted. The allocated figure is therefore the highest point that the instance has reached. Deleting objects does not add to the allocated figure either. The objects created after a deletion reuse the structures that the deleted object leaves behind, so deletions cost nothing further. Only a restart of the Master Servers returns the memory to the system. Restart the Master Servers for this only when an instance is being made permanently smaller. The allocated figure is not what a replacement machine has to match. That figure sums the metadata structures and nothing else. The Master Server reports the memory of the whole process as a separate reading. That reading, plus the headroom that the save needs above it, sizes the machine. What metadata holds covers which structures the breakdown sums and which records are outside it.
The Master Server measures its own structures and reports them here, which beats any estimate made from the outside.