Skip to main content

Migrating data in from another storage system

This page describes how to fill a MooseFS instance with data from an existing storage system. It covers what to decide on the target directory before the copy starts, how to run the copy, and how to check that everything arrived. Use it when a new cluster takes over a data set that is still stored on another system. Use it again whenever you move another tree in later. It expects a deployed cluster with a verified mount.

The data enters through the mount​

MooseFS presents a POSIX file system through the mount, so the mount is the import path. The tools that move data between local file systems (cp, rsync, and tar) move the data into MooseFS unchanged. There is no separate import tool to install or learn, because the mount already is the import interface. The commands MooseFS installs run and inspect the cluster. The mount transfers the bytes.

Set the target directory's policy before the copy starts​

A file gets its Storage Class, trash retention, and extra attributes from the directory that it is created in. The file gets them in the operation that creates the file (how policies are applied). A matching pattern overrides the inheritance by name in the same operation. A class set on the target directory before the copy therefore applies to every incoming file when the file is created. A class changed afterwards does not apply to any existing file by itself. You have to apply the class to every file already written. That second pass covers the whole tree and also moves the chunks (data placement tasks).

Settle two more attributes in the same step:

  • Trash retention. When the migration is rerun, a file that is deleted and copied again waits in the trash for its retention time. The file uses its space for that whole time. Give staging or scratch directories a retention measured in hours, with mfssettrashretention or through a pattern.
  • Quotas. If the migrated tree is to be capped, set the quota before the data is copied. Managing quotas is covered in space management.

A tree of small files costs chunks, not bytes​

Calculate the cost of the workload before you choose the class. Every chunk costs a fixed minimum, once per copy, whatever the size of the file in that chunk. On a tree of many small files, the chunk count, not the byte count, therefore decides how much of the cluster the data uses. See small files for the figure and raw space and user space for the counting. Erasure Coding (EC) does not reduce that cost. Under EC, a small chunk grows instead of shrinking (choosing between copies and Erasure Coding).

Copy with a tool that carries the metadata​

MooseFS stores what a POSIX file system stores: owner, group, permission bits, timestamps, and hard links (POSIX compliance), along with extended attributes and ACLs. This metadata is not copied automatically. The copying tool reads the metadata at the source and writes it at the target only when you request it. With rsync, the command is as follows, with your source tree in place of /export/data/:

rsync -aHAX /export/data/ /mnt/mfs/data/

-a copies owners, permissions, and timestamps. -H reproduces hard links as hard links instead of separate files. -A copies the ACLs and -X the extended attributes. A copy run without these options creates plain files. The copy drops the rest of the metadata without reporting an error.

Several streams move the data faster than one​

A single-threaded copy is bounded twice. Every file that it creates costs several exchanges with the Master Server. The copy waits for each exchange to finish before the next one starts (the latency budget). Everything that the copy writes passes through one mount, which has a throughput ceiling of its own. On millions of files, the waiting dominates. The speed of the copy depends on the round-trip time to the Master Server, not on the bandwidth of any link.

Split the tree into subtrees and run one copier per subtree. Parallel copiers overlap the metadata round trips. Several mounts multiply the data streams, on one machine or on many (multiple mounts). The metadata path does not multiply. The operations of every mount go to the one Master Server, so more streams add load there without adding capacity. Past that point, a copy with more streams stops getting faster. For the largest single streams, a copier built on libmfsio leaves the kernel and the mount's ceiling out of the data path.

Verify the copy against the source​

mfsdirinfo summarizes a tree from the counters of the Master Server: the number of inodes, directories, files, and chunks, then length, size, and realsize. Compare the file count and length, the sum of file lengths, with a count taken at the source. size and realsize have no counterpart at the source. They measure the cost of chunks and redundancy. Quota explains the difference between the three figures. By default, mfsdirinfo reads fast internal counters and counts a file once per hard link. mfsdirinfo -p walks the tree and counts every node and chunk once. Use this mode to check a copy made with -H.

To check a copy made with rsync, also run the same command again with -n -i added. The dry run lists every difference between source and target. A run that prints nothing confirms the copy.