CDN and media
One place behind a content delivery network holds the whole library. That place is the origin that the edge caches fill from and the store that the media pipeline writes its finished files into. This page describes one such deployment. The deployment is a single MooseFS instance in one server room. The instance holds video renditions, images, and the manifests that point at them. The instance serves a read load many times larger than the writes that filled it. The figures on this page are round numbers chosen for the example, not measurements.
CDNs come in two shapes. They differ in what the locations in front of the origin hold. In the full replica shape, every location keeps a complete or near-complete copy of the content. This shape suits material that has to be reachable from many places at the same moment, such as a video release synchronized across time zones. The full replica shape also suits material that is popular everywhere, such as blockbuster films, operating system and mainstream software updates, and widely played games. In the caching shape, one central location holds the complete content. The other locations keep only the frequently requested part of the content, and only for as long as that part is requested. This shape suits content that users generate, such as small business sites, images, and short films. The caching shape also suits content that is valid in some of the regions that the CDN serves and not in others, such as a news or advertising service. Both shapes need a location that holds everything. The deployment described here is that location.
The workload writes once and reads forever
| What the workload does | The figure this example works from |
|---|---|
| Video renditions produced by the transcoder | 500 TiB, files of 100 MiB to 4 GiB, each written once and not modified afterwards |
| Images, thumbnails, and manifests | 100 million objects, most of them under 100 KiB |
| Ingest | about 2 TiB a day, arriving as whole new files |
| Playback | up to 200 Gbit/s leaving the Chunkservers at peak, drawn from a shifting small share of the library |
Two counts follow from that description: the objects, which one Master Server holds in memory, and the bytes per second, which the Chunkservers and the network carry. The two counts change independently. Start from the workload calculates both counts from a workload. Master Server RAM from the object count calculates the first.
This workload differs from a general-purpose file system in that nothing is rewritten and that the reads concentrate on whatever was published last. Every decision in this example follows from those two facts.
One cluster carries two Chunkserver tiers
| Role | Machines | What each one carries |
|---|---|---|
| Master Server | 3 | One Leader and two Followers PRO (PRO Edition only), each holding the whole metadata set |
Chunkserver labeled S | 6 | SSDs, carrying the content that is being watched now |
Chunkserver labeled H | 18 | Hard drives, carrying the back catalog |
| Origin server | 16 | The HTTP origin the edge caches pull from, with the file system mounted |
A label is set on a Chunkserver, not on a disk, so the two tiers are two sets of machines rather than two kinds of drive inside one machine. The SSD machines get LABELS = S in mfschunkserver.cfg. The rest get LABELS = H. Size the S tier for the content still in its hot window and the H tier for everything else. Raw space and user space calculates how many disks those sizes need. Data placement tasks lists the commands for labeling Chunkservers, defining classes, and assigning classes.
Both Follower Master Servers need as much memory as the Leader, because each Follower holds the same metadata. Nothing about that figure follows from the media traffic: the Master Server is sized by the object count, never by throughput.
Only the ARCHIVE and TRASH definitions accept Erasure Coding
MooseFS writes data as copies and accepts Erasure Coding (EC) only in the ARCHIVE and TRASH definitions of a Storage Class (storage states). You therefore decide "how long does a file stay hot" rather than "copies or parts". One class covers both answers.
Serving needs the hot window in copies anyway. A chunk held in copies is read from one Chunkserver, whichever of the copies the client picks. A chunk held in Erasure Coding is read from every data part at once, four or eight Chunkservers for one stream. A chunk missing one of its data parts gets no servers at all, because parity parts are not read. The client waits and retries while the Master Server rebuilds the part. The read fails if the retries run out first. The read path describes both cases. On a library where a failed read stalls the player, a copy that another server can serve justifies its disk space while the content is being watched. The copy stops justifying that space when nobody asks for the file any more. Choosing between copies and Erasure Coding compares the two formats on space and on repair traffic.
One class places data on both tiers and measures the last read
mfsscadmin defines the class and mfssetsclass assigns it to the library:
mfsscadmin create -K 'S,2H' -A '@8+2,H' -d 30d -o AR -s 1Mi cdn
mfssetsclass -r cdn /mnt/mfs/library
-K 'S,2H' keeps three copies of active content: one on the SSD tier and two on the disks. -A '@8+2,H' PRO (PRO Edition only) puts archived chunks on the disk tier as eight data parts and two parity parts. This scheme keeps the Redundancy Level unchanged and reduces the stored size from three times the data to a quarter more than the data. The scheme also sets a minimum machine count. The label expression applies that minimum to the H servers only: Chunkservers needed for an Erasure Coding scheme lists both counts. Four data parts instead of eight lower that minimum and cost a little space. More parity parts raise the minimum and let the class survive one more concurrent failure. The bigger the instance, the higher the chance that two or three failures overlap. A large archive tier therefore justifies a higher Redundancy Level than a small tier.
-o AR -d 30d follows from the ingest pattern. Ingest writes each file once, so ctime and mtime stop changing the moment the transcoder finishes. With either of those timestamps, every file would be archived thirty days after it arrived, whether it was watched or not. With atime, the count restarts every time the file is read, so a title is archived thirty days after its last viewer. R sends the title back to the SSD tier when a viewer returns. The reversal has a cost: the chunks are rebuilt as copies and moved back. Set the period long enough that the reversal happens to titles that are genuinely watched again, not to single hits. Under the default ATIME_MODE, atime changes at most once a day for a file whose ctime and mtime have stopped changing. That is enough resolution for a period counted in days and none at all for a period counted in hours (timestamps). Storage Class evaluation lists the full set of archive modes and explains that the period is a minimum, not a schedule.
-s 1Mi never archives a file shorter than a mebibyte, so the manifests and the thumbnails stay in copies permanently.
Origin servers read the SSD copy while it lasts
The class says where the copies are stored. The class says nothing about which copy an origin server reads from. An option of mfsmount makes that choice. The option applies only on the client:
mfsmount /mnt/mfs -o mfspreflabels=S
Reads then go to the S copy. They go to a disk copy only when the Master Server no longer lists the S copy, for example because its Chunkserver has disconnected from the Master Server. The archived part of the library is served from the disks, because nothing else holds that part. Preferred labels covers the ranking. Parts of an Erasure Coding chunk are an exception: they are read in fixed positions. The preference does not affect which parts are read. Every option the mount takes is listed under mount options.
One mount cannot carry an origin server's output
A mount carries at most what its FUSE channel carries. That ceiling is a limit of the client machine, not of the cluster (the per-mount throughput ceiling). An origin server expected to send tens of gigabits therefore mounts the file system several times and splits its document root across the mounts. Alternatively, the origin server links libmfsio and leaves the kernel out of the data path altogether. Each mount is a separate session with its own caches and worker threads, so the memory that an origin server spends on the client grows with the number of mounts.
Streaming suits the client caches. A read that goes through a file from the beginning reaches the higher read-ahead levels and then fetches far ahead of the application. A large sequential read benefits from exactly that behavior. Scattered small reads do not get it. The buffers, the levels, and the caps on both are in client cache and read-ahead. The origin's own object cache comes before all of it, so MooseFS receives the misses rather than the requests.
Manifests and thumbnails cost by count, not by size
The hundred million small objects hold a fraction of the bytes and a large share of the cost. Each chunk is charged a minimum of 72 KiB, whatever the size of the file inside it, and is charged that minimum once per copy (small files). At three copies, these objects are counted as about 20 TiB, however little data they contain. The small objects also decide the Master Server's memory. A node record has the same size whatever the file's length, so a hundred million thumbnails and manifests cost more metadata than the whole video library does.
Two things follow. Keep the small objects out of Erasure Coding. The class's -s cutoff already does that. The predefined classes do the same for the same reason. Under EC, each part is a chunk file with its own header, so the headers cost more than the parity saves. The object count decides the Master Server machine. The cheapest reduction is therefore on the pipeline's side: fewer objects, each of them larger.
The network is sized for the reads that leave the Chunkservers
File data never passes through the Master Server: it goes straight between a mount and a Chunkserver. A read crosses the network once. A write crosses it once per copy. Ingest at 2 TiB a day into a three-copy class therefore puts three times that on the links and nothing on the Master Server. Repair traffic uses the same links whenever a disk is lost or the archive converts. Repair traffic is the part of the budget that is easy to forget. Bandwidth and latency budget covers both sides. A separated backbone network stops the cluster's own traffic from competing with the origin servers for the same interfaces.
Latency to the Master Server matters for a different reason than throughput. Every metadata operation that the client's caches cannot answer is a round trip. A video stream makes a handful of them and then moves bytes. A page that assembles forty thumbnails opens forty files. Put the Master Servers on the same low-latency network as the origin servers. Read mfstopology.cfg into the cluster if the racks are far enough apart for the distance between them to matter (topology mapping and distance).
A full replica takes a patch stream, not writes
This deployment serves one region from one room. Serving a second region from a second instance does not mean writing everything twice. Take a snapshot of the library on this side after each publishing round. Give the previous snapshot and the current snapshot to the MooseFS patch tools PRO (PRO Edition only). mfscreatepatch produces the set of operations that turns the older tree into the newer one. mfsapplypatch runs those operations on the remote instance, against a directory that is on a MooseFS mount (Master Server operations). mfsdumppatch prints what a patch will do before the patch is applied. A title published as an update to content that is already there is sent as the difference, not as a new copy of everything. The tools are listed under patches. The cost of a snapshot to the Master Server is in snapshots and copy-on-write.
Two instances synchronized this way are two file systems: each has its own namespace, its own metadata, and its own record of what the library holds between patches. When the second site has to access the same file system rather than a copy, use locations PRO (PRO Edition only). This mechanism divides one cluster across sites and maintains redundancy in each of them. Planning a cluster over several sites covers what that costs on the long line and when it is the right shape.
Caching satellites hold only what is hot
In this shape, the instance described on this page is the central instance and holds the complete content. The locations in front of it keep a cache of what their viewers ask for most. The two sides are more independent than in the full replica shape, because the whole content does not have to be synchronized. Each instance is the backend of the CDN software that fills it. That software decides what a satellite fetches and what it drops.
The archive rule is then set differently on each side. On the central instance, the rule can be set for space rather than for serving, because the reads arrive at the satellites. A shorter period turns more of the library into parts sooner. A satellite has the opposite traffic, and a great deal of it: content arrives and is deleted again all day. Give the satellite a class whose archive delay is longer than the time content usually stays there. Short-lived data is then never converted. Files popular enough to stay are turned into parts and stop costing three copies. The delay is the only thing that differs between the two definitions.