automounted zsys system snapshots prevent deletion

[aieri]

Describe the bug
After upgrading to Lunar, I noticed the garbage collector being unable to clean up old states due to the dataset is busy error.

Somehow new states snapshots get automatically mounted under /root/.zfs/snapshot, so the garbage collector cannot do its job until I manually umount them. I don't believe this was the case in Kinetic.

Quick workaround:

$ awk '/autozsys/ {print "sudo umount "$1}' /proc/mounts
$ # if happy, pipe the above to sh
$ zsysctl service gc -v

To Reproduce

root@laptop:~# zsysctl state save -s
ZSys is adding automatic system snapshot to GRUB menu
Successfully saved as "autozsys_felpum"
root@laptop:~# zsysctl state remove -s felpum
ERROR couldn't remove system state felpum: Couldn't remove state rpool/ROOT/ubuntu_y3atsu@autozsys_felpum: Couldn't destroy rpool/ROOT/ubuntu_y3atsu@autozsys_felpum: couldn't destroy "rpool/ROOT/ubuntu_y3atsu@autozsys_felpum" and its children: cannot destroy dataset "rpool/ROOT/ubuntu_y3atsu@autozsys_felpum": dataset is busy 
root@laptop:~# mount | grep felpum
rpool/ROOT/ubuntu_y3atsu@autozsys_felpum on /root/.zfs/snapshot/autozsys_felpum type zfs (ro,relatime,xattr,posixacl)
root@laptop:~# umount rpool/ROOT/ubuntu_y3atsu@autozsys_felpum
root@laptop:~# zsysctl state remove -s felpum
ZSys is adding automatic system snapshot to GRUB menu

Installed versions:

• OS: 23.04 (Lunar Lobster)
• Zsysd running version:

zsysctl	0.5.9
zsysd	0.5.9

[aieri]

I've noticed this happens at least when zsys takes a snapshot:

1. start with no snapshots being mounted
2. remove or install a package
3. all of your previous system states are now mounted under /root/.zfs/snapshot/

[ddnexus]

the garbage collector cannot do its job until I manually umount them

or at next reboot

[ddnexus]

@aieri The package removal/install triggers an update-grub, and in particular triggers the code in /etc/grub.d/10_linux_zfs, which mounts everything and does not seem to umount it correctly.

That seems also to cause the history entries in the grub menu to fail (i.e. there is no grub history anymore), at least on Lunar. @aieri plese, could you check whether your grub history appears in the grub menu or not?

I have another laptop with a previous Ubuntu version that does not have any problem with mount or grub history, even though it uses an identical /etc/grub.d/10_linux_zfs.

The problem seems to be related to openzfs/zfs#9958 and its workaround in the /etc/grub.d/10_linux_zfs and something that has been changed in zfs/zsys in the latest version.

[aieri]

That seems also to cause the history entries in the grub menu to fail (i.e. there is no grub history anymore), at least on Lunar. @aieri plese, could you check whether your grub history appears in the grub menu or not?

@ddnexus that's right, dataset history is no longer displayed in the grub menu of my Lunar machine.

[ddnexus]

dataset history is no longer displayed in the grub menu of my Lunar machine

Thank you for confirming it.

I put a few echo in the /etc/grub.d/10_linux_zfs to get some feedback from both versions, and indeed the /root/.zfs/... get mounted by it and never unmounted in Lunar. I have no idea about whether that is supposed to accomplish anything or it's just a by-product of something else, nor which code triggers it, nor if the old version even use that mountpoints. Debugging it will cost too much time to non-maintainers like myself, so I give up.

I wasted a huge amount of time just managing the quirks and bugs of zsys during the past years. On the other hand - ironically - I never had a single hardware failure on my laptops during that time. In retrospective, and looking at the behaviour of Canonical for this project (and others), adopting zsys has been a bad decision on my part, and continuing with it would be worse now, when the project is obviously in abandoned state.

Luckily there are compelling alternatives that appear to work for my needs and have some decent level of maintenance.

👀