Skip to main content

Chunk and chunk size

A chunk is the fragment of file data that MooseFS stores, places, and protects as one piece.

What it is​

MooseFS divides every file into chunks of at most 64 MiB. The first 64 MiB of the file form its first chunk, the next 64 MiB the second, and so on. A file smaller than 64 MiB has one chunk. The last chunk of any file holds only the data that remains at the end of the file. Each chunk is stored on a Chunkserver as an ordinary file on one of the Chunkserver's disks.

Inside a chunk, data is addressed in blocks of 64 KiB, so a full chunk is 1024 blocks. The block is the unit of integrity checking. Each block has a checksum that the Chunkserver verifies against the data that it reads.

Both sizes are constants of the storage format. No configuration parameter changes them.

Why it matters​

A Storage Class is assigned to a file, but copies and Erasure Coding parts exist per chunk. Each chunk is placed on its own set of Chunkservers. Two consequences follow. A file larger than one chunk is spread over several servers and is read and written through all of them. One file can also hold chunks in different formats at the same time: the older chunks converted to Erasure Coding, and the chunk still being appended to kept in copies.

Chunks are also the unit that the cluster is counted in. The Master Server holds the chunk table of every file in RAM. A Chunkserver holds an entry for every chunk on its disks. Memory therefore follows the number of chunks rather than the number of terabytes stored. A file far smaller than a chunk still has a fixed per-chunk cost. Small files calculates that cost.

How MooseFS implements it​

The Master Server gives each chunk a 64-bit chunk ID and a version number. The chunk ID is taken from a counter that only increases. The version number starts at 1. Not every write raises the version. Before a modification that could leave a stale copy behind, the Master Server raises the chunk's version. The Master Server then tells the Chunkservers that hold the chunk to move to the new version (chunk versions and states lists when a raise is due). A copy left behind at the earlier version is not treated as a valid copy of that chunk. On a Chunkserver's disk, the chunk ID and the version together form the name of the chunk file.

With copy-based redundancy, every copy is a complete chunk with the same ID and version. Each copy is stored on a different Chunkserver. Under Erasure Coding, the chunk exists as data parts and parity parts. Each part is stored as a file of its own on a Chunkserver. The Master Server tracks all the parts under the same chunk ID.

  • Chunkserver: the component that stores chunks and verifies their checksums.
  • Master Server: the component that assigns chunk IDs and versions and records where every chunk is.
  • Metadata and namespace: the file structure that maps each file to its chunks.