I first observed this in my test virtual machine that goes from non-LTS version to other. Going from 25.04 to 25.10 made it crash in a really weird way. It took X with it and left the system in a half-installed state. I restored it from snapshot and tried again with with no X running. It still crashes, the text console session just dies and I'm left in a login screen. If I log in, the session lasts couple of seconds and just dies. I can't see the dmesg as su and sudo neither work, suggesting a very unfortunate spot where the update failed. I booted off a live CD image, mounted the system in a chroot and examined what happens if I just attempt to continue the interrupted dpkg. Much to my surprise the system recovered just fine and after manually cleaning up the old unsupported packages, it worked just fine. This same system was later updated from 25.10 to 26.04 which caused no issues.
I chalked this up as typical Ubuntu non-LTS behavior, but I soon revisited the issue when I started to update my actual desktop installations from 24.04 to 26.04. I encountered the exact same kind of crash. I'm aware the encrypted ZFS setup is not that common, but I've found it to be least offensive maintaining-wise compared to stacking blockdevices over blockdevices with the alternatives of having a fully encrypted system.
In the encrypted root setup by Ubuntu's installer, the encryption key for the root volume is stored in a zvol named keystore under the rpool. This luks-encrypted volume has a file called system.key, which is the actual key. To chroot the encrypted system, we need to mount the zvol with the key somewhere, use that to unlock the rest of it and after that we're ready to recover the system as usual.
In short, we make directories for the key's mount point and the root system. Then we import the rpool into a foreign system, unlock it, then import the boot filesystem bpool and eventually mount it all. Boot off something that has ZFS built-in, the current Ubuntu live installer image works.
mkdir /mnt/target
mkdir /mnt/key
zpool import -N -f -R /mnt/target rpool
cryptsetup open /dev/zvol/rpool/keystore zfs-key
mount -v /dev/mapper/zfs-key /mnt/key/
zfs load-key -L file:///mnt/key/system.key rpool
zpool import -N -f -R /mnt/target bpool
zfs mount -a
After this we need to have the live system's proc and other special mounts available in the upcoming chrooted directory, which is at /mnt/target.
mount -v -o bind /dev /mnt/target/dev
mount -v -o bind /dev/pts /mnt/target/dev/pts
mount -v -o bind /proc /mnt/target/proc
mount -v -o bind /sys /mnt/target/sys
After that we switch into the chroot. Continuing dpkg's unconfigured packages processing will recover the system. Note that if you have any packages that download something in their install scripts, such as certain fonts, you may temporarily need to edit the chrooted environment's /etc/resolv.conf to have a valid name server.
chroot /mnt/target /bin/bash
dpkg --configure -a
Once it's done, you may exit the chroot, the system should be intact. Do note that when the system boots for the first time, ZFS will remember that it was mounted by someone else and it will die at bootloader and drop you into the emergency shell. You need to just import the rpoool and bpool with the force option, which changes their ownership back to the current system from the live image. Exiting the emergency shell should make it ask for the encryption password and the system will boot normally. It will also do so from now on automatically.
zpool import -f rpool
zpool import -f bpool
exit
No comments:
Post a Comment