systemd / zpool import / zfs mount / dependencies

On getting regular ZFS mount points to work with systemd dependency ordering.

ZFS on Ubuntu is nice. And so is systemd. But getting them to play nice together sometimes requires a little extra effort.

A problem we were facing was that services would get started before their respective mount points had all been made available. For example, for some setups, we have a local-storage ZFS zpool that holds the /var/lib/docker directory.

If there is no dependency ordering, Docker may start before the /var/lib/docker directory is mounted. Docker will start just fine, but it will start writing files in the wrong location. This is extremely inconvenient. So we want to force docker.service to start first after /var/lib/docker has been mounted.

Luckily we can have systemd handle the dependency ordering for us. We'll have to tell the specific service, in this case docker.service, to depend on one or more mount points. That might look like this:

# /etc/systemd/system/docker.service.d/override.conf
[Unit]
RequiresMountsFor=/data/kubernetes/static-provisioner
RequiresMountsFor=/var/lib/docker

These unit file directives specify that the Docker service can start first when these two paths have both been mounted. To this end, systemd will look for definitions of data-kubernetes-static\x2dprovisioner.mount and var-lib-docker.mount.

The mount unit file format is described in systemd.mount(5). The filename must adhere to the output of systemd-escape --path --suffix=mount MOUNTPOINT. So for the above two mount points, you might make these two unit files:

# /etc/systemd/system/data-kubernetes-static\x2dprovisioner.mount
[Unit]
Documentation=https://github.com/ossobv/vcutil/blob/main/mount.zfs-non-legacy
After=zfs-mount.service
Requires=zfs-mount.service

[Mount]
Where=/data/kubernetes/static-provisioner
What=local-storage/kubernetes/static-provisioner
Type=zfs-non-legacy

# /etc/systemd/system/var-lib-docker.mount
[Unit]
Documentation=https://github.com/ossobv/vcutil/blob/main/mount.zfs-non-legacy
After=zfs-mount.service
Requires=zfs-mount.service

[Mount]
Where=/var/lib/docker
What=local-storage/docker
Type=zfs-non-legacy

Observe: that Type=zfs-non-legacy requires some extra magic.

You might have been tempted to set Type=zfs, but that only works for so-called legacy mounts in ZFS. For those, you can do mount -t zfs tank/dataset /mointpoint. But for regular ZFS mounts, you cannot. They are handled by zfs mount and the mountpoint and canmount properties.

Sidenote: usually, you'll let zfs-mount.service mount everything. It will zfs mount -a, which works if the zpool was also correctly/automatically imported. But because you have no guarantees there, it is nice to manually force the dependency on the specific directories you need, as done above.

To complete the magic, we add a helper script as /usr/sbin/mount.zfs-non-legacy. This gets the burden of ensuring that the zpool is imported and that the dataset is mounted. That script then basically looks like this:

#!/bin/sh
# Don't use this script, use the better version from:
# https://github.com/ossobv/vcutil/blob/main/mount.zfs-non-legacy
name="$1"  # local-storage/docker
path="$2"  # /var/lib/docker

zpool import "${name%%/*}" || true  # import poool
zfs mount "${name}" || true

# Quick hack: mount again, it should be mounted now. If it isn't,
# we have failed and report that back to mount(8).
zfs mount "${name}" 2>&1 | grep -qF 'filesystem already mounted'
# (the status of grep is returned to the caller)

That allows systemd to call mount -t zfs-non-legacy local-storage/docker /var/lib/docker and that gets handled by the script.

A better version of this script can be found as mount.zfs-non-legacy from ossobv/vcutil. This should get you proper moint point dependencies and graceful breakage when something fails.