How to backup your LXD Containers

Continuing the discussion from Running Containers with LXD:

Container backup solution

If you’ve read the article on LXD containers then you may be looking for an efficient backup solution. There are many options out there, however the combination of features provided by this option may be of interest.

Features

• Should provide consistency without needing to stop the container or it’s contents
• Backups are source encrypted, so can be stored anywhere
• Backups are compressed to save storage space
• Backups are de-duplicated, so duplicate blocks are removed, which saves even more space
• Backups are incremental by nature, so only changes since the last backup are saved
• Backups are sent directly to a remote host (or can be)
• It’s all free software

Additional Resources

The software involved is called borgbackup. In order to use it, you’ll need a client (in this example it would be your container host) and another server configured as a borg backup server. I may do an article on setting up your own backup server later, but in the meantime if you don’t have one handy (!) then you can get 10G of storage from these guys as a part of their free tier, or if you want to pay for more, it’s something like $2/month for 250Gb. (this is not a sponsored add / article, they’ve just proven to be good and really cheap)

Setting up

On Ubuntu you will need to install the borg software, the stock repositories carry a reasonably current version;

apt install borgbackup

This is a relatively low-level tool, so on top of this I recommend a script for managing backups, currently I’m using something called borgmatic. Unfortunately the Ubuntu repositories are a little out of date, so this example is based on installing from the Python repositories;

apt install python3-pip  # IF you don't already have "pip"
pip3 install borgmatic --break-system-packages

If you’re working on a system that’s dedicated to running containers as per the previous container article, don’t worry too much about the “–break-system-packages” flag. It’s Ubuntu being overly paranoid (IMO). If you’re working on a pre-existing system with lots of other Python code running, you might want to read up on what this flag is for.

The backup config

So, here we go, this is the script that’s going to do all the work. You will want one config for each container you wish to back up, each one goes into /etc/borgmatic.d/ as a yaml file. So this is the config to backup my madpenguin container; /etc/borgmatic.d/madpenguin.yaml

constants:
  instance: madpenguin
  borghost: borg
  borguser: wp

source_directories:
  - /mnt/snapshots/{instance}

repositories:
  - path: ssh://{borguser}@{borghost}./.backups/{instance}
    label: {instance}

one_file_system: true
patterns:
  - 'R /'
  - '+ /mnt/snapshots/{instance}'
  - '- /mnt/snapshots/*'

encryption_passphrase: "my_pass_phrase"
checkpoint_interval: 1800
compression: zstd,3
ssh_command: 'ssh -i ~/.ssh/id_rsa'

keep_secondly: 60
keep_minutely: 60
keep_hourly: 24
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 1

checks:
  - name: repository
  - name: archives

before_backup:
  - lvcreate -n {instance}_snapshot --snapshot default/containers_{instance} &&
    lvchange -ay -K default/{instance}_snapshot &&
    mkdir -p /mnt/snapshots/{instance} &&
    mount /dev/default/{instance}_snapshot /mnt/snapshots/{instance}

after_backup:
  - echo "Complete.........................." &&
    umount /mnt/snapshots/{instance} &&
    lvremove -y default/{instance}_snapshot

The backup procedure

On the first run you will need to initialise a repository. With your own server it will look something like this;

$ borgmatic init --encryption repokey

It might be a little different if you’re using borgbase but the borgbase instructions will guide you. In this instance as I’m backing up to a local server, we’re telling borg to store the encryption / decryption keys with the backed up data. (so it just requires the passphrase to recover the backup) If you are storing the backup on somebody else’s server (like borgbase) then you will probably want to store the key on locally for more security. If this worked then you should be ready to try a backup. Note that you will need to tweak the configuration to suit yourself, in particular the constants listed at the top of the script. Also the encryption_passphrase, anything log(ish) so long as it’s not “my_pass_phrase” (!) You may also need to tweak the ssh path depending on your server.

Running a full backup

# Note that in this example I'm using '-v1' for more verbosity
$ borgmatic -v1 -nc --stats -c /etc/borgmatic.d/madpenguin.yaml
/etc/borgmatic.d/madpenguin.yaml: Running command for pre-backup hook
  Logical volume "madpenguin_snapshot" created.
madpenguin: Creating archive
Creating archive at "ssh://wp@borg/./.backups/madpenguin::wp-2023-09-23T14:02:23.026340"
------------------------------------------------------------------------------
Repository: ssh://wp@borg/./.backups/madpenguin
Archive name: wp-2023-09-23T14:02:23.026340
Archive fingerprint: 211df698006a9b9dd2193fea15c0411faf50895dd465ea82df06ea5e197c28f1
Time (start): Sat, 2023-09-23 14:02:25
Time (end):   Sat, 2023-09-23 14:03:51
Duration: 1 minutes 25.45 seconds
Number of files: 91774
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                5.04 GB              2.10 GB              1.87 GB
All archives:                5.04 GB              2.10 GB              1.87 GB
                       Unique chunks         Total chunks
Chunk index:                   63183                83035
------------------------------------------------------------------------------
/etc/borgmatic.d/madpenguin.yaml: Running command for post-backup hook
Complete..........................
  Logical volume "madpenguin_snapshot" successfully removed.
madpenguin: Pruning archives
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:                5.04 GB              2.10 GB              1.87 GB
                       Unique chunks         Total chunks
Chunk index:                   63183                83035
------------------------------------------------------------------------------
madpenguin: Compacting segments
Remote: compaction freed about 1.12 kB repository space.
madpenguin: Running consistency checks
Remote: Starting repository check
Remote: finished segment check at segment 12
Remote: Starting repository index check
Remote: Index object count match.
Remote: Finished full repository check, no problems found.
Starting archive consistency check...
Analyzing archive wp-2023-09-23T14:02:23.026340 (1/1)
Archive consistency check complete, no problems found.

summary:
/etc/borgmatic.d/madpenguin.yaml: Successfully ran configuration file

So looking at the stats, it looks like I have 5GB of used space in the container, which compresses down to 2.1GB, which de-duplicates down to 1.87GB. So the backup is nearly down to a third of the actual container size, and duration looks like a minute and a half.

Running an Incremental backup

So that was a full backup, if I run it again it should give me an incremental backup;

$ borgmatic time -nc --stats -c /etc/borgmatic.d/madpenguin.yaml
  Logical volume "madpenguin_snapshot" created.
------------------------------------------------------------------------------
Repository: ssh://wp@borg:2222/./.backups/madpenguin
Archive name: wp-2023-09-23T15:15:05.051430
Archive fingerprint: 791a2de94df522af9a3e950057db15d19ede4cdce9714a851e421f0c2f7a4968
Time (start): Sat, 2023-09-23 15:15:07
Time (end):   Sat, 2023-09-23 15:15:21
Duration: 14.07 seconds
Number of files: 93030
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                5.11 GB              2.12 GB             28.57 MB
All archives:               10.15 GB              4.21 GB              1.90 GB
                       Unique chunks         Total chunks
Chunk index:                   64148               167315
------------------------------------------------------------------------------
Complete..........................
  Logical volume "madpenguin_snapshot" successfully removed.
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:               10.15 GB              4.21 GB              1.90 GB
                       Unique chunks         Total chunks
Chunk index:                   64148               167315
------------------------------------------------------------------------------

So an incremental in 14 seconds and an additional 28MB, seems fairly efficient!

Restoring backups

I would recommend NOT relying on backups telling you they’ve worked. Always verify a backup, at the very least every time you change your backup configuration or update your backup software. The only way to properly verify a backup is to restore it onto a test system and to make sure it works and contains the data you expect.

Tales from the Crypt

Once upon a time there was a London firm who were ranked at the top of their profession. One day a maintenance engineer arrived to update the Operating System on their main computer server. Before starting, as you might expect, he asked “do you have a current backup”, to be told, “we just did one before you arrived - as instructed - here it is”. The engineer proceeded to update the Operating System on their main server. Upon completion he found the application the firm ran that handled all of their data, was incompatible with the new version of the Operating System. At this point he concluded he would have to restore the most recent backup and report back that there was a mismatch between the Operating System and the application. Unfortunately the most recent backup (tape) failed to restore. The engineer was a little perplexed and went to the IT managed to request the previous backup. Fortunately, because the firm relied so heavily on the data there was a complex system of daily, weekly, monthly and annual backups. So there were lots of backups. Unfortunately the previous backup didn’t work either. Nor the one before that. Nor the one before that. Before they knew it, they were calling the Managing Director asking for the copy in his home safe from three months prior. That didn’t work either. Turns out in trying to restore the first backup, the engineer had wiped the system, and none of the backup tapes would restore. When the disaster recovery engineer was called to establish what had gone wrong, it turns out two parts of the backup plan had been ignored. The bit that said always verify a backup, and the bit that said replace tapes every year because they wear out. Tapes being tapes will write quite happily, but then not be able to see or report errors until they are read back. Of course at this point the system had been running for 5 years … … anyone want to ask me if they lived happily ever after? 1 post - 1 participant Read full topic