Data placement tasks
This page covers the tasks that control where MooseFS stores data. These tasks are giving Chunkservers labels, defining and assigning Storage Classes, moving data between disk tiers, turning on Erasure Coding (EC), and assigning classes by file name with patterns. The mechanism chapter Placement and policies covers how the Master Server turns a class into an actual placement. This page shows the commands.
Give the Chunkservers labels and reload them
Set the labels of a Chunkserver in its mfschunkserver.cfg, as letters separated by , or ;:
LABELS = A,S
Apply the change with a reload. The mfschunkserver process keeps running and keeps serving data:
mfschunkserver reload
The Chunkserver reads the new value and reports it to the Master Server at once. Verify in the Labels column on the Chunkservers tab of the GUI.
A fresh cluster ships with classes
A new instance comes with predefined classes. 2CP is the class of the root directory, so every file gets 2CP until you assign another class.
| Class | Definition |
|---|---|
2CP | 2 copies |
3CP | 3 copies |
EC4+1 | 2 copies while fresh, EC 4+1 in the ARCHIVE state after 24 hours from ctime, files under 512 KiB stay in copies |
EC4+2 PRO (PRO Edition only) | as EC4+1, but 3 copies and EC 4+2 |
EC8+1 | as EC4+1, but EC 8+1 |
EC8+2 PRO (PRO Edition only) | as EC4+1, but 3 copies and EC 8+2 |
You can modify, rename, and delete all of them, like any class that you define yourself. Classes named 1 through 9 are not among the predefined classes. Where such classes exist, they came from an upgrade of a cluster older than 4.57.0. These classes also behave like ordinary classes. To list the classes of your cluster with full definitions, use mfslistsclass:
mfslistsclass -l
Define your own class with mfsscadmin
mfsscadmin is the administration tool for class definitions. mfsscadmin(1) documents the tool. Every operation is available as a subcommand (create, modify, delete, clone, rename, list, import) and also as a standalone command name (mfscreatesclass, mfsmodifysclass, and so on). Run the tool inside a MooseFS mount, or name a mount with -M.
A class that keeps two copies on labeled Chunkservers, one on a server labeled A and one on a server labeled B:
mfsscadmin create -K 'A,B' tworooms
A class that keeps newly written data in two copies anywhere and converts the data to Erasure Coding 4+2 PRO (PRO Edition only) two days after the last data write:
mfsscadmin create -K '2*' -A '@4+2' -d 2d -o m ecarchive
-K, -C, -A, and -T take one label expression each, for the KEEP, CREATE, ARCHIVE, and TRASH storage states. Only -K is required. A state without its own expression uses a fallback. CREATE and ARCHIVE fall back to KEEP. TRASH falls back to the definition that governed the chunk when the file was deleted. The expression syntax, the STANDARD, STRICT, and LOOSE labels modes, and how an undefined state inherits are described in label expressions and matching. In the Community Edition, an EC expression uses one parity part (@4+1, @8+1). Two to nine parity parts need PRO.
The -m option reads only the first letter of its value: d sets STANDARD, s sets STRICT, l sets LOOSE. -m std therefore sets STRICT, not STANDARD. Give it a single letter.
Two more options restrict who may use a class. -a 1 restricts the class to administrators. -g puts the class in an export group. Only mounts whose export entry allows that group can then assign the class. Both restrictions are described in class priority and export groups. The same page covers -p, the class priority. When files share a chunk through snapshots and have different classes, the class with the highest priority governs the chunk. If snapshots cross class boundaries, give your classes distinct priorities.
mfsscadmin delete removes a class only when no file or directory has that class. A class still in use is refused with an error.
mfsscadmin create and mfsscadmin modify print the definition themselves, under storage class created: and storage class changed:. Read the definition to confirm what the Master Server accepted. Check the KEEP expression that you gave and, if you defined one, the ARCHIVE expression and its delay. mfslistsclass -l prints the same block for a class that already exists, with its class ID added. A deleted class no longer appears in that list.
A class defines placement once per location
On a cluster divided into locations PRO (PRO Edition only), every class has a separate definition for each location. -l names the location that the following switches belong to. You can repeat -l. Each occurrence is followed by at least one of -K, -C, -A, and -T. A definition written before any -l belongs to the default location. Two sites, two copies at each:
mfsscadmin create -l west -K 2 -l east -K 2 twosites
The definitions do not have to match. The redundancy that each site uses does not have to match either. One rule limits the asymmetry. In any location, an Erasure Coding expression is accepted in ARCHIVE and TRASH and nowhere else (Erasure Coding). A site that is to hold EC therefore holds it as its ARCHIVE definition. A class that keeps two copies at the working site and lets the backup site fall back to a stripe:
mfsscadmin create -l west -K 2 -l east -K 1 -A '@4+2' offsite
The fallback between states works inside a location. When a location has no CREATE or ARCHIVE definition of its own, that location's KEEP definition is used instead. A zero expression stops that fallback. -C 0 means that the location receives no new chunks, but its KEEP definition still holds copies that arrive by replication. Use this arrangement for a backup site whose link is too slow for direct writes:
mfsscadmin create -l west -K 2 -l backup -C 0 -K 1 results
Leaving a location out of the class altogether has the same effect as a zero for every state: nothing of that class is stored there. Use the explicit zero where the fallback would otherwise fill in a state. Leave the location out where the location is to hold none of the class at all.
Every state has to be reachable somewhere in the class: at least one location needs an explicit KEEP. CREATE, ARCHIVE, and TRASH need either a definition of their own or a location whose KEEP is used instead and is not zeroed. mfsscadmin(1) lists all the combinations.
To remove one location's definition without changing the rest, name the location in mfsscadmin delete:
mfsscadmin delete -l backup results
Without -l, the same command removes the whole class. To retire a location, start with this step. Location tasks describes the rest.
Assign a class with the mfssclass tools
The assignment tools mfssetsclass, mfsgetsclass, mfscopysclass, and mfsxchgsclass are documented in mfssclass(1). Assign a class to a directory and everything already in it:
mfssetsclass -r ecarchive /mnt/mfs/projects
A new object inherits the class of its parent directory when the object is created. Changing a directory's class without -r therefore affects only objects created afterwards. The full order of precedence between inheritance, patterns, and manual assignment is described in how policies are applied. To read an assignment, use mfsgetsclass. With -r on a directory, the tool counts the objects in the subtree per class. mfscopysclass copies the assignment from one object to other objects. mfsxchgsclass replaces one class with another class and changes only objects that have the source class:
mfsxchgsclass -r tworooms ecarchive /mnt/mfs/projects
The tools resolve a symbolic link named on the command line and act on the link's target. mfsgetsclass on a link therefore reports the class of the target. Recursive runs walk the directory tree on the Master Server and do not follow symbolic links. No object that a link points to is therefore counted or changed twice.
Assigning a class changes metadata only. The chunks move when the chunk loop next compares them with the new definition. Copies on wrongly labeled servers have the lowest of the replication priorities, so a large reassignment takes time to complete.
Verify the assignment with mfsgetsclass -r /mnt/mfs/projects. Every file and directory in the subtree is counted under the class that you set, and no other class is listed. That count comes from the metadata. The physical placement of the chunks needs a separate check, described in Check where the data actually lies.
One class moves data from SSD to HDD
Tiering by drive type needs one class, not a script. Set LABELS = S in mfschunkserver.cfg on the SSD Chunkservers and LABELS = H on the HDD Chunkservers. Reload these Chunkservers. Then define a class that keeps new data on the SSDs and archives the data to the HDDs:
mfsscadmin create -K '2S' -A '2H' -d 30d -o m ssd2hdd
mfssetsclass -r ssd2hdd /mnt/mfs/projects
New chunks are written to S servers, because CREATE follows the KEEP definition. Thirty days after a file's last data write (-o m counts from mtime), its chunks enter the ARCHIVE state and migrate to H servers. A write does not clear the archive flag of a chunk archived in copies, so a file that becomes active again stays on the H servers until its flag is cleared, by R in the class's archive mode or by hand with mfsclrarchive. The file loop and the chunk loop decide when the flag is set and when the chunks move. The delay is therefore a minimum, not an exact time. The flag's modes are described in Storage Class evaluation.
Verify with mfsfileinfo. New files list copies on the S-labeled Chunkservers. Files past the delay list copies on the H-labeled Chunkservers.
Erasure Coding starts at the ARCHIVE definition
An EC expression is accepted only in the ARCHIVE and TRASH definitions of a class. To turn EC on, give a directory a class whose ARCHIVE definition is EC. The shortest way is to use a predefined class:
mfssetsclass -r 'EC4+1' /mnt/mfs/archive
Files get and keep two copies. Twenty-four hours after their last ctime change, their chunks are converted to EC 4+1. For a format @N+X, the cluster needs at least N+2X Chunkservers to convert chunks to EC. It needs at least N+X Chunkservers to keep the chunks in EC. For EC4+1, that means six and five. The thresholds and the conversion mechanics are described in Erasure Coding: part layout.
To pick your own delay, timestamp, and small-file cutoff, define a class instead:
mfsscadmin create -K '2*' -A '@8+1' -d 12h -o c -s 1Mi archive12h
-s sets the minimum file length. A shorter file never enters the ARCHIVE state and stays in copies. This is why the predefined EC classes have a 512 KiB cutoff. For very small files, the parts use more space than the copies that they replace.
Verify after the delay has passed: mfsfileinfo lists one EC part per Chunkserver for an archived chunk, instead of whole copies.
Patterns assign a class by file name
A pattern assigns a class when a file is created or renamed, before anyone runs a command on the file. The tools are available both as subcommands of mfspatadmin and as the standalone commands used below. mfspatadmin(1) documents them:
mfscreatepattern -n '*.iso' -c ecarchive
mfslistpattern
-n accepts a glob that is matched against the file name alone, never against the path. A pattern can also assign trash retention (-t) and extra attributes (-f). You can restrict a pattern to a creating user or group (-u, -g). When several patterns match, the pattern with the highest -p priority (0 to 255, default 128) applies. A pattern does not change files that already exist. Reassign those with mfssetsclass. How policies are applied covers where a pattern is placed in the order between inheritance and manual assignment, and what a rename does. Remove a rule with:
mfsdeletepattern -n '*.iso'
Verify with mfslistpattern. It prints one line per rule, with the glob, the priority, and the class that the rule assigns. A new rule appears on that list. A removed rule no longer appears in it.
Check where the data actually lies
mfsgetsclass -r shows what is assigned. mfsfileinfo shows where the data is physically stored. It prints the Chunkserver host and port of every copy or EC part of each chunk. The Storage Classes subtab of Redundancy in the GUI lists every class with its definitions and with counters of the objects and chunks that use the class. The Chunkservers tab shows each server's labels. Right after a change, a mismatch between assignment and physical placement is caused by the repair lag of the chunk loop, not by a fault.