Containers on a ZFS filesystem

What and Why

There are many ways to host data for containers. Earlier I covered containers using LVM, so each container would sit on a separate thinly provisioned Logical Volume managed by LVM. In this instance I’m going to try to do the same, but with each container sitting on a volume on a ZFS pool. Operationally this should be relatively transparent, but under the hood it does expose some more interesting options, specifically compression and encryption. Typically you would do this on a real machine, however for ease of documentation I’m running inside a KVM based Virtual Machine.

To Start

I have a machine installed with Ubuntu Server 23.04, it has 200G of storage, partitioned with a 25G root partition and two empty partitions of 25G for 150G. The larger partition will be a storage pool for container instances, and the smaller will be a compressed and encrypted pool for private data. So, the following will all be done as the root user;

$ fdisk -l /dev/vda
Device         Start       End   Sectors  Size Type
/dev/vda1       2048      4095      2048    1M BIOS boot
/dev/vda2       4096  52432895  52428800   25G Linux filesystem
/dev/vda3   52432896 104861695  52428800   25G Linux filesystem
/dev/vda4  104861696 419428351 314566656  150G Linux filesystem

The first thing we need to do is install ZFS which includes the kernel modules, tools, and associated libraries.

$ apt install zfs-dkms

This should install ZFS and associated dependencies of which there will be a quite a few. It needs to compile and generate kernel modules so it may take a few minutes to complete.

Creating the ZFS Pools

Next we need to create the desired pools. The container pool is relatively straightforward, all we need is;

$ zpool create default /dev/vda4 -m legacy
$ zpool list
NAME      SIZE  ALLOC   FREE  FRAG   CAP  DEDUP    HEALTH
default   149G   100K   149G  0%     0%   1.00x    ONLINE
$ zfs list
NAME      USED  AVAIL     REFER  MOUNTPOINT
default   118K   144G       24K  legacy

It gets a little more interesting as we add a pool with compression and encryption;

$ zpool create -O encryption=on -O keyformat=passphrase \
               -O keylocation=prompt -o compatibility=off \
               -o feature@encryption=enabled -m legacy \
               private /dev/vda3
#
# It should then prompt for your passphrase, this should be
# secure and at least 14 characters.
#
Enter new passphrase: ....
Re-enter new passphrase: ....

$ zfs list
NAME      USED  AVAIL     REFER  MOUNTPOINT
default   118K   144G       24K  legacy
private   198K  23.7G       98K  legacy
#
# Now turn on compression for the pool called "private"
#
$ zfs set compression=gzip private

And we should be done. The state of private with regards to encryption is persistent, so once the pool has been unlocked, so it will remain so until either explicitly locked or until the machine is shut down.

Testing ZFS compression

Just to make sure we’re actually getting some compression on the private pool, I’m going to try a simple test, bearing in mind all pools with get de-duplication by default. (so just copying a file full of zero’s won’t actually show compression because de-duplication will have already taken all the blanks out)

$ tar cf archive.tar /usr
$ ls -lh
total 2.6G
-rw-r--r-- 1 root root 2.6G Oct 16 12:16 archive.tar
#
# If we create a temporary volume in "private"
#
$ zfs create private/tmp -o mountpoint=/mnt/tmp
#
# Then move our test archive onto it ..
#
$ zfs list
NAME          USED  AVAIL     REFER  MOUNTPOINT
default       118K   144G       24K  legacy
private       975M  22.8G       98K  legacy
private/tmp   975M  22.8G    =>975M  /mnt/tmp

So the archive when stored on the normal ext4 root filesystem is 2.6G in size, however when moved to the encrypted / compressed filesystem on the private zfs pool, it’s only actually consuming 975M, which seems pretty reasonable. Just to see how much of this is compression and how much is de-duplication, I’ll move it over to the uncompressed default pool;

$ zfs create default/tmp -o mountpoint=/mnt/tmp2
$ mv /mnt/tmp/archive.tar /mnt/tmp2
$ zfs list
NAME          USED  AVAIL     REFER  MOUNTPOINT
default      2.49G   142G       24K  legacy
default/tmp  2.49G   142G   =>2.49G  /mnt/tmp2
private       442K  23.7G       98K  legacy
private/tmp    98K  23.7G       98K  /mnt/tmp

So although we’re getting a little bit of de-duplication saving (2.46G vs 2.6) the majority is coming from the compression we applied to the private volume. We could apply compression to default, however this would slow our containers down somewhat and in this instance I’m not too worries about storage space.

Note compression is typically applied to either an entire pool, or a volume, whereas encryption is typically enabled for an entire pool.

Accessing our encrypted data

If you now reboot your machine, when it comes back up you should see that it has auto-mounted /mnt/tmp2, which is a filesystem we created on the default pool, but not the /mnt/tmp we created on the private pool. (this is because the pool is locked by default) To get access to the private pool we can do;

$zfs load-key -a
Enter passphrase for 'private': ...
1 / 1 key(s) successfully loaded
$ zfs mount -a

Now if you take a look at df you should see it has unlocked the private pool and automatically mounted the /mnt/tmp volume.

Adding containers into the mix

Adding containers was previously covered here; But to go over it again in a little less detail;

$ snap install lxd
$ lxd init
Would you like to use LXD clustering? (yes/no) no 
Do you want to configure a new storage pool? yes 
Name of the new storage pool: default
Name of the storage backend to use: zfs
Create a new ZFS pool? (yes/no): no
Name of the existing ZFS pool or dataset: default
Would you like to connect to a MAAS server? no
Would you like to create a new local network bridge? yes
What should the new bridge be called? lxdbr0 
What IPv4 address should be used? auto
What IPv6 address should be used? auto
Would you like the LXD server available over the network? yes
Address to bind LXD to (not including port): all
Port to bind LXD to: 8443
Would you like cached images to be updated automatically? yes
Would you like a YAML "lxd init" preseed to be printed? no

Now enable the LXD user interface;

$ snap set lxd ui.enable=true
$ systemctl reload snap.lxd.daemon

If you point your browser at the machine’s port 8443 (for from the machine itself; https://localhost:8443) and follow the instructions, you should be able to install the appropriate client certificates to get the GUI working.

Note I have found the process of installing client certificates in the browser for LXD to, on occasion, be problematic. If you end up with a strange text (JSON) response instead of a web page, you might like to try the following fix which has worked for me;

$ mkdir lxd-api-access-cert-key-files
$ cd lxd-api-access-cert-key-files
$ openssl genrsa -out lxd-webui.key 4096
$ openssl req -new -key lxd-webui.key -out lxd-webui.csropenssl x509 -req -days 3650 -in lxd-webui.csr -signkey lxd-webui.key -out lxd-webui.crt
$ openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -in $ lxd-webui.crt -inkey lxd-webui.key -out lxd-webui.pfx -name "LXD WebUI"
$ lxc config trust add lxd-webui.crt
# Now download the lxd-webui.pfx file. Locally.
# Import the file to the browser.

IF the issue still persists (note, this will destroy any containers you’ve created), try;

$ snap remove --purge lxd
$ snap install lxd
# At this point you will need to remove all the ZFS volumes
# from **default** because "init" will try to recreate them
$ lxd init

Creating a ZFS based container

So assuming we now have a working UI on https://localhost:8443, we should be seeing something like this; So, if we click on create instance … And follow the yellow brick road, we end up with a running container. Then if you click on storage you it should show you your (ZFS) storage pool with details of space used and space remaining; So our first container has consumed a total of 659Mb of storage, which is probably what you might expect for a basic server installation. On further inspection however, what it’s actually done is to create an immutable base image for the version of Linux you’ve selected, and then a second (copy-on-write) volume containing differences to the base image.

$ zfs list -r default/images
NAME                           USED  AVAIL     REFER  MOUNT
default/images                 617M   144G       24K  legacy
default/images/5c0f660608...   617M   144G      617M  legacy

$ zfs list -r default/containers
NAME                      USED  AVAIL     REFER  MOUNTPOINT
default/containers       21.3M   144G       24K  legacy
default/containers/zfs1  10.6M   144G      622M  legacy

So the base image is consuming 617M, but the container itself is only using 10.6M. The useful thing to note is that if we create a second container using the same version of Linux, it can use the same base instance. So whereas the first container consumes 617M + 10.6M of space, the second (and subsequent) will only consume 10.6M of space. (which makes them incredibly space efficient, even before you start to look at de-duplication or compression) Just to prove the point, if I create a second instance; Then go back an look at storage consumption in the default pool;

$ zfs list -r default/images
NAME                           USED  AVAIL     REFER  MOUNT
default/images                 617M   144G       24K  legacy
default/images/5c0f660608...   617M   144G      617M  legacy

$ zfs list -r default/containers
NAME                      USED  AVAIL     REFER  MOUNTPOINT
default/containers       21.3M   144G       24K  legacy
default/containers/zfs1  10.6M   144G      622M  legacy
default/containers/zfs2  10.6M   144G      622M  legacy

Summary

An alternative pool based storage system for LXD based containers.

• One or more host managed storage pools
• Access to ZFS options such as compression, encryption, RAID etc
• Out of the box de-duplication
• Lazy space allocation / re-allocation
• Easy access to snapshots for backing up individual containers
• Fully integrated into LXD’s infrastructure and UI

1 post - 1 participant Read full topic