Preferred labels
The Storage Class of a chunk decides where its copies are stored. The Master Server enforces that decision (Storage Class evaluation). Each client decides separately which of those copies it contacts. A mount can be given a list of preferred labels. The client then sorts every list of Chunkservers that the Master Server sends to it, so that the preferred servers come first. The Master Server is never told about the preference. The preference does not cause any chunk to be placed or moved, so two mounts of the same file system can prefer different servers without affecting each other.
The option exists for that separation. Consider a class that keeps one copy of every chunk on Chunkservers labeled S (SSDs) and a second copy on Chunkservers labeled H (hard drives). Such a class gives each chunk a fast copy and a cheap copy. However, the class says nothing about which copy is read. -o mfspreflabels=S on the mount decides which copy is read. Reads go to the SSD copy. They fall back to the H copy only when the Master Server no longer lists the SSD copy. The same works geographically, with a copy of everything in each of two server rooms labeled A and B. Mounts in room A prefer A. Mounts in room B prefer B. The mounts in each room read their own copy of the same files.
- The preferred servers only come first. When none of them answers, the client asks the Master Server for a new list and retries the read. The other room's copy is still on that list, because the ranking reorders the list of Chunkservers and never shortens it. The retried read takes that copy only when the new list no longer names the preferred servers. The Master Server drops a Chunkserver from the list when the Chunkserver disconnects from the Master Server, not when only the mount cannot reach it.
- The Master Server never receives the preference. The client asks for a chunk by inode and chunk index. The answer includes each Chunkserver's labels, and the mount then sorts the servers by those labels on its own.
- Every Chunkserver in a room has the room's label. The class names the label, not the machine, so the Master Server still decides which server in a room holds the copy.
One copy in each server room. The class puts a copy in each room, but the mount decides which room's copy it reads. Follow the two arrows that meet on every Chunkserver: the one from the Master Server is the class, the one from the mount the preference. Nothing runs from a mount back to the Master Server, so the two halves have to be set to agree.
The preference is a mount option, kept client-side
The option is -o mfspreflabels=LABELEXPR. LABELEXPR is an ordered list of up to nine label expressions separated by commas, most preferred first. The list uses the same syntax as a Storage Class. If the list cannot be parsed, mfsmount stops before anything is mounted. The block device client mfsbdev accepts the same option. Command line tools covers how mount options are given and where they can be preset.
The client ranks each server by the first expression that the server matches
The reply that gives the client a chunk's Chunkservers also includes the labels that each of those servers has. The client then ranks every server in that reply by the first expression in the client's own list that the server's labels satisfy. A server that satisfies the first expression outranks a server whose first match is the second expression. A server that satisfies none of the expressions comes after every server that satisfies any. Ranking never shortens the list. A server that matches nothing is still read from or written to when the matching servers are no longer on the list. When no server matches anything, I/O proceeds exactly the same way as without the option.
Reads go to the best-ranked server, and ties go to the least busy server
For a read, the client sorts the list and reads from the first server on it. Servers of equal rank are ordered by the number of operations that this mount has running against each of them at that moment. Reads and writes are counted together. A mount that keeps many requests in flight therefore spreads them across equally preferred copies. Without mfspreflabels, every rank is equal, so this count alone decides the order. After a failed read, the client does not continue through the rest of the old list. The client asks the Master Server for a new list and sorts that list. The client keeps no record of which server failed. The Master Server lists every connected Chunkserver that holds a copy. A preferred server that is connected to the Master Server but that the mount cannot reach therefore comes first on every new list. The read retries that server until the read runs out of retries, and then fails. The read moves to a server that ranks lower only after the preferred server disconnects from the Master Server.
Writes build the chain in the same order
A chunk write reaches every copy through a chain. The client sends data to one Chunkserver, which forwards the data to the next Chunkserver (write path). The sorted list is that chain, so the client itself connects to the best-ranked server. The remaining copies follow in rank order. Servers of equal rank keep the order in which the Master Server sent them. Without the option, the chain is the Master Server's list, unchanged.
Erasure-coded parts keep fixed positions and are not sorted
When the Master Server answers a read with the parts of an Erasure Coding chunk instead of whole copies (Erasure Coding layout), the client reads from the data parts. The client reads each part from the Chunkserver named for that part, in positions fixed by part number. Preferred labels play no role in such a read. When the answer is a list of whole copies, the ranking applies, whatever format the chunk's class requires.
The client's ranking takes precedence over distance
The list that the client sorts is already ordered when the client receives it. The Master Server sorts the list before sending it. On a cluster divided into locations PRO (PRO Edition only), the Chunkservers of the client's own location come first. Nearer servers come before farther servers according to the topology map. Remaining ties are broken at random. The client's ranking is the primary sort key on top of that order. A read therefore goes to a matching far server rather than to a non-matching near server. For writes, the Master Server's order remains the tie-break.
On a cluster divided into locations, there is one more interaction. The Master Server reports that a Chunkserver outside the client's location has no labels at all. An expression made of letters therefore matches only servers in the mount's own location. * still matches every server. A negated expression matches a server that has no labels.