Convert Debian (or Ubuntu) system from BIOS boot to EFI

When upgrading hardware, some newer hardware tends to be allergic towards old BIOS booting. Fortunately converting is relatively simple once you figure out all the pieces needed.

The main points are:

1. Boot off a rescue CD, add EFI partition, mount filesystems

2. Remove grub-pc

3. Install grub-efi

Here's a closer look at it:

 

1. Boot off a rescue CD

I use SystemRescueCD 9 here, but probably any version works. Important: boot the CD image in EFI mode! Otherwise you won't be able to add grub-efi to the EFI boot menu and the system won't boot.
 
After it has booted, we need a directory for our root partition. I'll use /mnt/target here, the system's root is assumed /dev/sda1, use tools like lsblk and blkid to identify the correct one in yours.
 
We will need access to EFI parameter store, so we first need to modprobe efivarfs.
 
First verify with a partition editor, such as cfdisk that the disk uses GPT partition scheme. If not, you have to convert it first in-place with  gdisk /dev/sda. Gdisk will load the MBR partition table, after which you can use the w-command to write the GPT partition table over it. This does not affect the partitions themselves or the data on them.
 
Grub's EFI parts needs a vfat partition to live in. It only needs some megabytes, 200 megabyte partition is enough. Partition type for it is EFI System. One quick hack is to make the swap partition this much smaller. It can be then formatted to vfat with mkfs.vfat. The freshly created partition will be /dev/sda3 here.
 
Next we mount the system root directory and bind-mount the system directories from the live CD's ramdisk root. Without these bind mounts grub-install won't work. We also mount efivarfs.

mount /dev/sda1 /mnt/target
mkdir -v /mnt/target/boot/efi
mount /dev/sda3 /mnt/target/boot/efi
mount -o bind /sys /mnt/target/sys
mount -o bind /proc /mnt/target/proc
mount -o bind /dev /mnt/target/dev
mount -o bind /dev/pts /mnt/target/dev/pts
mount -t efivarfs none /mnt/target/sys/firmware/efi/efivars
 
Now that we've gotten everything mounted, we chroot into the real system.
 
chroot /mnt/target /bin/bash

Hopefully we now find ourselves inside the target system. First thing we do here is to add the new efi partition to the /etc/fstab.

/dev/sda3  /boot/efi  vfat  umask=0077  0  0
 

2. Remove grub-pc

The old grub conflicts with the grub-efi, so we get rid of it:

apt --purge remove grub-pc

It'll most likely remove its dependencies as well.

3. Install grub-efi

apt install grub-efi shim-signed
 
It'll also install couple of handy utilities for poking EFI, like efibootmgr. The shim-signed package makes it possible to use secure boot.

Next we need to install grub's files on the freshly created vfat partition. Note that this command does not take the partition, but the device where the root drive is installed. Since the root here was /dev/sda1, it's just /dev/sda.
 
grub-install /dev/sda
 
Look at carefully what it says. If it errors out, there are couple of things that might have been wrong.  First thing to check with errors is if you actually booted in EFI mode or BIOS. If it complains about not being able to modify EFI variables, the efivarfs module is missing or it was not mounted.

If everything went okay, the files EFI need are now placed on the vfat partition, but we still need the menu entry for the OS. Update-grub does this for us. We also need to update the existing initramfs.

update-grub
update-initramfs -u

It should mention that it adds the boot menu entry. We can then use efibootmgr to investigate:

efibootmgr

Spot the Debian or Ubuntu entry in the list and see that it's listed in the BootOrder. The other options vary depending on the motherboard or virtualization platform.

If the efibootmgr shows the entry, you're now ready to boot. Exit from the chroot and reboot the machine. You should be greeted with grub-efi's boot menu now.

Speedtest.net's own cli client vs. Ubuntu's and Debian's

I monitor a remote location with a 4G link and one of the metrics I'm interested in is the weather's effect to the quality of the connection. Debian and Ubuntu carry speedtest.net's console client, but no matter what I do, it's outcome is random at best. The same connection over the web browser or app version shows a rather stable 50-something megs, while the speedtest-cli fluctuates wildly between 20 and 50 megs between the tests.

I had assumed the Linux client offered by speedtest.net was exactly same thing carried by distributions, but apparently I was wrong. With their own command line client I can finally see what I believe to be realistic values.

Installation instructions for it are here: https://www.speedtest.net/apps/cli

lm-sensors coretemp under XCP-ng 8.2.1

Xen appears to do something funny and the default coretemp-module for Intel's CPUs refuses to co-operate in domain 0. To make lm-sensors work under XCP-ng, there's a package called "coretemp-module-alt" which works under Xen. Just install and modprobe it.

Convert Windows 10 virtual from MBR and BIOS boot to GPT and UEFI

I had an old Windows 10 virtual I use to test out stuff and it's been around from the times when you could upgrade from Windows 8 for free. That again was an upgraded Windows 7, so you get the picture, UEFI in Linux KVM didn't work when this thing was installed. Nowdays it works and I want to upgrade this virtual to boot from UEFI instead. Windows 10 comes with an utility to do the conversion on an existing installation, so no need to reinstall. However, it'd be a very good idea to take a snapshot of the virtual before doing this!

First Windows needs to be rebooted into rescue mode. Settings -> Update & Security -> Recover -> Advanced startup.

Next I selected Troubleshoot -> Advanced Options -> Command prompt. This made Windows reboot, ask me to log in and then let me into recovery environment command line. The utility is in \windows\system32 directory, which should be where we land when the recovery console starts. Converting takes two commands:

mbr2gpt /validate

mbr2gpt /convert

Once finished, the command line window can be closed and virtual powered off. Afterwards the boot firmware can be configured from BIOS to UEFI and secure boot can be turned on.

Hetzner Storage Box sub account CIFS path

Hetzner's storage boxes are relatively cheap way to dump virtual machine images or whatever. I wanted to create an ISO SR for XCP-ng as a sub user for the storage box. I would have expected this be pretty obvious thing but no. So, I created a sub-user in Hetzner's Robot interface, shared a directory called "iso" and the resulting path eventually was:

\\u123456.your-storagebox.de\u123456-sub1

Just heads up if you expected it to end with \backup\iso or something.

Copying Linux root filesystem on file level

Years ago I was curious about btrfs and converted one of my virtuals machines to use it. The virtual in question had used ext4 before and was converted in-place into btrfs. Later on it was learned this was in fact a bad idea as the btrfs conversion tool seems to be a gamble on data integrity.

Anyway, this system ran with converted btrfs for years, during which time I had learned that btrfs created from scratch was pretty decent for some things, but the converted ones wouldn't pass btrfs' fsck even when they would pass online scrub. This virtual was one of those cases and I decided to migrate the existing filesystem's content onto a freshly created one on file level.

Just out of curiosity, I ran btrfs check on it and it was just what I had encountered before:


# btrfs check /dev/xvdb1
Opening filesystem to check...
Checking filesystem on /dev/xvdb1
UUID: 8c7512e2-3613-474d-9234-835a57b6f896
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 5 inode 1177668 errors 8000, inline file extent too large
root 5 inode 1177752 errors 8000, inline file extent too large
--- bazillion lines cut ---
root 5 inode 1831684 errors 8000, inline file extent too large
root 5 inode 2878563 errors 8000, inline file extent too large
ERROR: errors found in fs roots
found 7227060224 bytes used, error(s) found
total csum bytes: 6377088
total tree bytes: 159498240
total fs tree bytes: 132104192
total extent tree bytes: 16334848
btree space waste bytes: 40630997
file data blocks allocated: 120313016320
 referenced 6451007488

While the errors are numerous and scary, there's apparently no data loss. I created a fresh btrfs filesystem, mounted them both and then created one monster of a rsync command:

rsync -axHAWXS --numeric-ids --progress /mnt/source/ /mnt/target/

After rsync was finished, I just had to use grub-install to write a new boot track on the freshly created virtual disk and this Debian 11 based system booted off fine and continues collecting data with Munin.

FreeNAS 12, rsync code 10 and compression

FreeNAS (or TrueNAS core nowadays) started to misbehave with its rsync. My backups jobs had started to fail inexplicably with error code 10. 

A closer inspection revealed they had deprecated the old -z (same as --compress) option compression. Changing it to -zz (--new-compress) made it work again.

Hetzner, IPv6 and routing with PfSense

Hetzner has a peculiar way of handling IPv6. They'll point your IPv6 block onto one MAC address and it's up to you how you deal with it. There's numerous examples out there how to do it with KVM based systems, such as just plain Linux installations running KVM virtuals or dedicated KVM distress, such as Proxmox. But what if you run VMWare ESXi, Citrix XenServer or its excellent open source version, XCP-ng?

Hetzner's go-to answer is installing a Linux virtual, pointing the IPv6 block to it and then letting the virtual handle the routing. I'm sure this works just fine, but I saw the whole concept as an overkill as there's dedicated firewall distributions that run with lean resources and are very easy to back up and maintain.

So, I thought PfSense would have been documented in a million places. Nope, just those lone forum posts of "Does anyone know how to do this?" and then either zero replies, or replies that didn't contribute anything. So, here's how I did it and it works just fine. I make the assumption that ESXi or Xen is already installed and you know how to use it.

First, go into Hetzner's Robot interface and get yourself an extra IPv4 address. Once it's set up, request a separate MAC address for it (little network card and screen icon next to the IP). Make a note of the new MAC address you get.

Create a new virtual machine. Set it up for FreeBSD template and install PfSense on it. For WAN interface, use the MAC address you got previously from Robot interface.

Configure the IPv4 address you got from Hetzner, then get decide what'll be the new router PfSense's IP. I'm going to use 123:456:678::1 here. Select 64bit netmask. Use fe80::1 as a gateway for IPv6.

From PfSense's settings, go to Advanced -> Networking and make sure Allow IPv6 is checked.

Next go to Firewall -> Aliases and create a new alias, type "Host(s)". I'll call it HETZNER_VIRTUALS here. Add all the IPs your virtual machines will use here. For this example, I'll use 123:456:678::20.

Next go Firewall -> Rules and create three rules for the WAN interface:

Rule 1: "Block Hetzner virtuals  to firewall"

Action: Reject 

Address Family: IPv6

Source: Single host or alias -> HETZNER_VIRTUALS

Destination: This firewall (self)


Rule 2: "Allow ANY to Hetzner virtuals"

Action: Pass 

Address Family: IPv6

Source: ANY

Destination: Single host or alias -> HETZNER_VIRTUALS

 

Rule 3: "Allow Hetzner virtuals to ANY"

Action: Pass

Address Family: IPV6 

Source: Single host or alias -> HETZNER_VIRTUALS

Destination: ANY

 

If you're wondering, you could also use the whole block for the rules, but I personally preferred to just route to the machines I have. At this point you should be able to use ping6 to ping hosts outside from PfSense's shell.

Now what PfSense is set up, install your guest OS. I'll use 123:456:678::20 for my example and 123:456:678::1 as its gateway. Note that you may need to create reverse DNS for the IP for it to actually work. 

After the installation traceroute6 should show it goes through 123:456:678::1 on its way out and the same thing the other direction.