Choosing between copies and Erasure Coding
A chunk is stored either as whole copies or, under Erasure Coding (EC), as data parts and parity parts. The format is part of a Storage Class. The Storage Class sets the format separately for each storage state. Only the ARCHIVE and TRASH definitions accept EC. Data is therefore always written as copies. Before you buy hardware, decide what happens to the data once the data stops changing. The format that you choose has a cost in disk space, in Chunkservers, and in the time that a read or a write takes afterwards.
At equal redundancy, parity costs a fraction of copies
A copy definition requires one to nine copies. Each copy is the whole chunk. An EC definition is written @N+X. N is the number of data parts, four or eight. X is the number of parity parts: one to nine PRO (PRO Edition only) and one in the Community Edition. Every part, parity included, holds one Nth of a chunk, so a chunk in @N+X occupies (N+X)/N of its own size. K copies occupy K times that size.
The two formats count the failures that they survive in different ways. K copies survive K-1 lost disks or Chunkservers, because one copy has to remain. @N+X survives X lost disks or Chunkservers, in any mix of data and parity parts, because any N of the parts rebuild the rest. That count is the Redundancy Level. Only the Redundancy Level makes the two formats comparable.
| Format | Redundancy Level | Usable share of raw space |
|---|---|---|
| 2 copies | 1 | 50% |
| 3 copies | 2 | 33% |
| 4 copies | 3 | 25% |
@4+1 | 1 | 80% |
@4+2 PRO (PRO Edition only) | 2 | 67% |
@4+3 PRO (PRO Edition only) | 3 | 57% |
@8+1 | 1 | 89% |
@8+2 PRO (PRO Edition only) | 2 | 80% |
@8+3 PRO (PRO Edition only) | 3 | 73% |
The share is 1/K for K copies and N/(N+X) for @N+X. The table rounds the share to the nearest percent. The share applies to one definition. Raw space and user space calculates the space that is left for you in a cluster that holds several classes at once.
The scheme sets a minimum number of Chunkservers
Both N and X raise the number of Chunkservers that a scheme needs. A cluster with fewer Chunkservers than that number keeps the data in copies instead, at the cost of copies, whatever the Storage Class requires. Chunkservers needed for an Erasure Coding scheme lists the counts that each scheme requires.
EC saves space at the cost of traffic and waiting
A read reaches one Chunkserver per 256 KiB region of the chunk that the read covers, up to N Chunkservers. But a chunk in copies is read from one Chunkserver whatever the length of the read. A write to a chunk held in EC waits for a join. The join rebuilds a whole copy by reading a full set of data parts over the network. Restoring redundancy writes back a single part rather than a whole chunk. Computing a lost part reads a full set of parts. A copy is turned into EC either by a local split or by a transfer. A local split happens inside the Chunkserver that already holds the copy. A transfer sends each part to a Chunkserver of its own. A local split ends with the same result, because ordinary replication then spreads the parts over other servers. Erasure Coding: part layout describes each of these operations.
Part headers give the saving back on small files
Under EC, every part is a chunk file in its own right. Every part has its own header. A chunk that is small enough for the headers to matter therefore grows under EC rather than shrinking. The extra headers add more than the parity saves on data. Small files covers what a chunk costs at minimum, and why. A Storage Class can also set a minimum file length and leave every shorter file in copies. This rule is applied during Storage Class evaluation.
Changing data belongs in copies and settled data in EC
Data that keeps receiving occasional writes after it has been archived belongs in copies. Every such write costs a join and returns the chunk to copies. The chunk is turned into parts again the next time it is archived.
EC is for the rest of the data: large files, written once, then read whole or left alone. On that data, EC saves the difference between K times a chunk and (N+X)/N of that chunk. EC saves that difference on every chunk of the archive.