Backups aren't simple

See also: John Salvatier's excellent blog, Reality, has a surprising amount of detail

TLDR: A real backup is more than a copy of your files. It needs snapshots, rotation, deduplication, an offsite copy, and restores you actually test. Building all of that yourself gets out of hand fast, which is why tools like Borg and Restic exist.

I read a comment somewhere that stuck with me, which went something like this: "There are two types of people: those who have suffered a catastrophic loss of data, and those who will."

Trying to find the source for this blog, I found that every other sysadmin has his rehashed version of the quote, but the gist is the same everywhere. Data loss happens more often than we'd hope, and most of us are woefully unprepared for when it hits (almost always at the worst possible time). So here's what a proper backup actually takes, and why it gets complicated faster than you'd think.

The day our family photos disappeared

I can confirm that I had a similar experience once. When I was little, we pulled all our family photos from our home laptops and PCs onto an external hard drive to free up space. This worked beautifully until one day my dad wanted to use the drive to store our TV set-top box (one of those old things) and was prompted to format it. He went ahead, and the disk was reformatted. The index of files was deleted, and we were stuck with a nominally empty drive.

It would be easy to blame him for screwing up, but it takes beginning a career in tech to realize that there is a series of errors that lead to this kind of mistake. First, we put all our photos in one place and didn't bother with backups. Secondly, most consumer-facing software usually has bold disclaimers telling you that formatting a disk means losing data (which the set-top box didn't; terrible UI). Besides, why would you expect a non-technical person to know any of this?

Thankfully we were able to get the photos restored, and it turned out to be a cheap lesson in handling data. You never keep important things in only one place. There's about a million things that can go wrong. Your drive could die; it could be stolen; bits could rot in cold storage (hard drives have magnetic particles which can inexplicably shift, and SSDs are made of NAND transistors which leak electricity and, over time, corrupt your data).

Why RAID 1 isn't a backup

So our first principle is to have a backup, i.e., a copy of your files someplace else. So far so good.

This doesn't cover the headaches a plugged-in drive can cause. Ransomware could encrypt your files, and you could do anything from an honest mistake like deleting the wrong file, up to catastrophic mistakes like running a script that overwrites everything with zeroes.

So our backup should not be a mirror of the first drive, because we also want to be able to go back in time if we mess up. Importantly, this means that mirroring your disk with something like RAID 1 is out. We need some other method that snapshots things.

How often should you back up?

How often do we want to take snapshots? Maybe, in our case with the photos, we should run a backup every week. If we lose 6 days and 23 hours of data, that's fine, and we can live with it. This is what's called a Recovery Point Objective (RPO) in IT, and in real cases, it ranges from <30 seconds for critical financial institutions, which really can't afford to lose data, to 24 hours or more for some small enterprises (if they even have a disaster recovery strategy).

Taking snapshots means that we have an increasing burden on our storage. With an RPO of 24 hours, you will end up having 7 snapshots per week. 30 per month. 365 per year, if you don't prune your snapshots. So you need to rotate your backups.

GFS backup rotation

Let's say I go with the naive approach and decide to keep 14 days' worth of snapshots. When I take a new snapshot, I delete the oldest one, and I add the new one. Pretty simple, but this now forces me to have a watchful eye. Maybe I keep lots of data and can't be bothered to check if something got corrupted in the past two weeks? But then again, I can't just store a year's worth of backups, and they're simply not relevant to me. What happened between day 2 and day 3 of the year has almost no significance when it's day 364. So the granularity of our backups must change. The closer we are to today, the more frequent the snapshots. The further back, the less frequent the snapshots.

So maybe we rotate our daily backups every 14 days, but also take weekly backups that we rotate every 7 weeks, and monthly backups that we rotate every 12 months. This should be much more efficient. But again, our complexity grows. We now have something called a GFS-rotated, snapshot-based backup. This list of adjectives will continue growing, as we'll see in a bit.

Incremental and deduplicated backups

Maybe then you take a look at how MPEG compresses video, and get fascinated by how a calm scene in a movie, where the protagonist speaks but otherwise doesn't move against a completely still background, can be used for compressing video. You notice that videos are composed of frames that are mostly similar to each other, changing only with a certain movement that can be represented as a vector, saving a fraction of the storage.

This leads you to the very logical conclusion that your snapshots also follow the same pattern! Even more, it turns out that file changes follow a fat-tailed distribution, so over a given period most files don't change at all, and a tiny minority change all the time.

So it becomes obvious that we shouldn't store identical copies of files, but rather deduplicate. We can use hard links when we need to reference an already existing file. This way, we store one file on disk, and then reference it from each of our snapshots. This also survives backup rotation because we never delete files; we only delete directory entries. This exact approach is used by rsnapshot, and is best described as an incremental backup, because we store only the changes between two adjacent snapshots, instead of all the changes since the latest full backup (these are called differential backups and are more robust when restoring, but I won't get into it for the sake of brevity).

The storage savings are not the only benefit we get from doing this. We briefly mentioned in the beginning that we don't store everything on a single machine. Obviously, networking is also involved in this process, since we need to transfer the files from one machine to another. Deduplicated backups save a lot of bandwidth, which is especially important if you use a cloud service as your second machine. It directly affects you financially.

To sum up, by this time we have created an incremental, deduplicated, GFS-rotated, snapshot-based backup. We can use rsync to pull the files from the main machine, and cron jobs to run our backup scripts. We can run backups on as many machines as we'd like, and adding another one is trivial. Even better, file metadata is preserved, so access permissions and file ownership stay intact.

Backing up Docker containers and databases

Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason is that many Docker containers like to create root-owned files, and if you're not careful, you can create a cron job running as the default user.

To make matters worse, almost every web app uses some kind of database. Databases sometimes store things in memory and flush them to disk in batches to improve performance. This practically means that restoring from backup will fail due to data corruption if we are unlucky. So we make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes. That should make it work!

The 3-2-1 backup rule and offsite storage

Then you read about incidents in which a model of hard drive had famously high failure rates, and start to wonder if you should maybe store your backups on two machines with different types of media. That way, a hardware-specific failure would be unlikely to wipe out your backups. And while we're on the topic of physical security, have one offsite backup in the cloud or on a machine at a family member's house. This way you make it really unlikely that a power surge, flood, or fire will destroy everything. This is where the 3-2-1 backup gets its name: 3 copies, on 2 different types of media, with 1 offsite.

Let's say you decide on a cloud provider for your offsite backup. Specifically, object storage like Amazon S3. You quickly find out that our current setup won't work because, one, files lose their metadata when you upload them to S3, and two, uploading many small files to S3 is punitively expensive (file sizes also follow a fat-tailed distribution). These two facts make it best for you to stick many files into a tarball. That way you retain both your file metadata and your low costs. But the question is, how do you do that? Do you stick everything in one giant tarball? Obviously not. Then your incremental backups with the hard links stop making sense. Best to split everything into clean 50MB chunks, but good luck with doing that in a way that is verifiably safe!

Why Borg or Restic beats DIY scripts

Up until this point, rolling your own backups sounded like something you should be able to do in an afternoon, but this is where I'd give up. It simply isn't worth the mental load. Instead, you just use tried-and-tested tools like Borg or Restic, which handle all this and much more (encryption, chunk-level deduplication, checksums). And give your utmost thanks to the wonderful open-source community for building, maintaining, and live-testing these tools, while respecting how much trial and error it took to get to the point where all this complexity is abstracted away for us.

Obviously, none of this is worth anything if you don't actually test restores. So that's also a little digital hygiene article that you will have added to your to-do list. So, as long as you run restores every 6 months, you can enjoy your:

encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution

Also, make sure not to run backups at 2 AM or 3 AM, or things may get scary.
____________________________________________________________________

Updates after the discussion

A big thank you to everyone for discussing this topic with me. Here are some important things that either slipped my mind, I wasn't aware of, or otherwise left out of the main article:

1. You shouldn't run cron jobs at or between 2 AM and 3 AM. Obviously, when we skip an hour, we skip the whole hour. Or have it twice, when we skip back.
2. With all this talk on backups, I should have mentioned ZFS. This is an entire filesystem built around enforcing data integrity and snapshots.
3. I will quote a Hacker News user crabbone directly:

Consistency groups. The article kinda almost gets there, but stops right before it realizes there's a problem. If you have multiple programs accessing the storage, e.g., a database (that you are snapshotting) and the application using the database, then you risk an inconsistent snapshot when the database state you wrote is ahead of or behind the application state. You need a mechanism to freeze all I/O except the snapshot to ensure the snapshot data is useful.

ACLs. And similar external mapping to the storage being used, eg., what if the users come from an LDAP server? Similar problem with symlinks coming from/going to outside the snapshot.

Sparse file support. Let's say you have a few VM images in your snapshot: if your tool doesn't support sparse files, it will likely greatly inflate the image sizes.

Bottom line: either use the product's own snapshot functionality (ZFS has those), or switch to the product that does.