Xubuntu and horrible touchpad experience on Lenovo machines

I've used Thinkpads and other Lenovo machines since 486 days on Linux. I used to pretty much ignore their touchpad then those surfaced along the trackpoint, but at some the touchpads grew so big my palm started to touch them in the typical typing position.

I could not find any reasonable way to either enable palm rejection or make it sane any other way under XFCE. People like to parrot "use the disable touchpad when typing" feature as some kind of a solution, but they do not see the problem: the issue begins BEFORE I start typing, as the thing clicks all around the screen, often focusing the wrong thing before first keypress is done.

I toyed around with synclient, trying to create a proper palm rejection profile, but to no avail. The big discovery came as I realized that there's multiple drivers available for the touchpad and the Synaptics one is loaded by default, which may seem like a sane thing. But the libinput driver not only seem to handle it, it seems to handle it in a fashion we've come to expect from a Windows or Mac computer's touchpad.

I literally had to just remove the package xserver-xorg-input-synaptics and it was instant bliss afterwards. The only thing I did in XFCE mouse settings was to configure how I wanted the scroll and mouse button emulation to work and there it was. After years of pain, a touchpad experience close to the OSes for which the official drivers were written for.

Ubuntu update from 24.04 to 26.04 with encrypted ZFS root crashes

I first observed this in my test virtual machine that goes from non-LTS version to other. Going from 25.04 to 25.10 made it crash in a really weird way. It took X with it and left the system in a half-installed state. I restored it from snapshot and tried again with with no X running. It still crashes, the text console session just dies and I'm left in a login screen. If I log in, the session lasts couple of seconds and just dies. I can't see the dmesg as su and sudo neither work, suggesting a very unfortunate spot where the update failed. I booted off a live CD image, mounted the system in a chroot and examined what happens if I just attempt to continue the interrupted dpkg. Much to my surprise the system recovered just fine and after manually cleaning up the old unsupported packages, it worked just fine. This same system was later updated from 25.10 to 26.04 which caused no issues.

I chalked this up as typical Ubuntu non-LTS behavior, but I soon revisited the issue when I started to update my actual desktop installations from 24.04 to 26.04. I encountered the exact same kind of crash. I'm aware the encrypted ZFS setup is not that common, but I've found it to be least offensive maintaining-wise compared to stacking blockdevices over blockdevices with the alternatives of having a fully encrypted system.

In the encrypted root setup by Ubuntu's installer, the encryption key for the root volume is stored in a zvol named keystore under the rpool. This luks-encrypted volume has a file called system.key, which is the actual key. To chroot the encrypted system, we need to mount the zvol with the key somewhere, use that to unlock the rest of it and after that we're ready to recover the system as usual.

In short, we make directories for the key's mount point and the root system. Then we import the rpool into a foreign system, unlock it, then import the boot filesystem bpool and eventually mount it all. Boot off something that has ZFS built-in, the current Ubuntu live installer image works.

mkdir /mnt/target
mkdir /mnt/key
zpool import -N -f -R /mnt/target rpool
cryptsetup open /dev/zvol/rpool/keystore zfs-key
mount -v /dev/mapper/zfs-key /mnt/key/
zfs load-key -L file:///mnt/key/system.key rpool
zpool import -N -f -R /mnt/target bpool
zfs mount -a 

After this we need to have the live system's proc and other special mounts available in the upcoming chrooted directory, which is at /mnt/target. 

mount -v -o bind /dev /mnt/target/dev
mount -v -o bind /dev/pts /mnt/target/dev/pts
mount -v -o bind /proc /mnt/target/proc
mount -v -o bind /sys /mnt/target/sys 

After that we switch into the chroot. Continuing dpkg's unconfigured packages processing will recover the system. Note that if you have any packages that download something in their install scripts, such as certain fonts, you may temporarily need to edit the chrooted environment's /etc/resolv.conf to have a valid name server.

chroot /mnt/target  /bin/bash
dpkg --configure -a 

Once it's done, you may exit the chroot, the system should be intact. Do note that when the system boots for the first time, ZFS will remember that it was mounted by someone else and it will die at bootloader and drop you into the emergency shell. You need to just import the rpoool and bpool with the force option, which changes their ownership back to the current system from the live image. Exiting the emergency shell should make it ask for the encryption password and the system will boot normally. It will also do so from now on automatically.

zpool import -f rpool
zpool import -f bpool
exit

 

XCP-ng PCIe error messages

XCP-ng runs fine on cheap desktops that make good, low power homelabbing virtualization hosts. I came across this issue with a Fujitsu Esprimo small form factor machines. The dom0 ring buffer is flooded with messages about PCIe bus error. The problem looks like this:

pcieport 0000:00:1c.7: AER: Corrected error received: 0000:00:1c.7

pcieport 0000:00:1c.7: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)

pcieport 0000:00:1c.7:   device [8086:a33f] error status/mask=00001000/00002000

pcieport 0000:00:1c.7:    [12] Timeout  

The device in question appears to be the bridge itself:

00:1c.7 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #8 (rev f0)

It seems the PCIe power management does not play ball with the dom0 kernel in XCP-ng. The issue can be fixed by just turning the power management feature off, after which the errors stop. It's done by adding pcie_aspm=off in dom0 kernel parameters (not Xen!)

This was apparently fixed in later kernel versions, but XCP-ng 8.3 still has relatively old dom0 kernel, so this was still needed.

Debian screen garbled during boot under XCP-ng

This is an annoyance I've had with Xen Linux guests for a while. The symptoms are: when booting in EFI mode, GRUB appears normally, but after the initial messages about loading the kernel and initramdisk, the screen looks like this:


After a while during the boot the screen eventually becomes normal before the login script. This is a small annoyance, except if the system drops into an emergency shell and you can't read the screen.

Debian's GRUB configuration in /etc/default/grub has an option GRUB_GFXMODE, I uncommented that and set the 1024x768. I observed the framebuffer to be a bit more readable, but still garbled. Since the resolution was correct, I assumed it has to do with color depth and tried setting the color depth too, which wasn't instantly obvious in the configuration file examples.

But in my case setting GRUB_GFXMODE=1024x768x32 finally made it look normal right from the start.

I determined the resolution and color depth from virtual_size and bits_per_pixel files in /sys/class/graphics/fb0.

Seafile web GUI getting stuck at random with Apache

I have ran my Seafile server with an Apache front end under Debian for closer to a decade now. I've never had any meaningful issues with it, but I encountered an issue with freshly set up instance where the web GUI would get stuck loading at random. It happened often just after login, but would also happen when moving in directories and viewing files.

After very careful comparison to a working one, I noticed the working one used the Apaches prefork MPM and the problematic one again the event MPM. I remembered a case from a distant past, where FastCGI had issues with the worker MPM and the event MPM is a close relative to it.

After switching the problematic instance from event to prefork MPM, Seafile's web GUI issues went away.

Huge (or tiny) mouse cursor in Firefox Snap under Ubuntu.

Ever since Ubuntu 22.04 was released, Firefox's mouse cursor has been acting out with XFCE. I have a custom cursor theme which the snap version of Firefox can't seem to use. After some investigation, it seems Firefox pulls the cursor size from whatever XFCE's mouse cursor settings control, but the locally created mouse cursor doesn't scale.

To get the mouse cursor in in Firefox onto a bit more manageable size (mine was like 2cm tall in a HD resolution display!), go to XFCE's Mouse and Touchpad settings, theme tab and adjust it to more saner size there.

Can't enter UEFI setup with F2 on an Intel NUC

Repurposing a stack of old NUCs I came across an odd thing. The NUCs' keyboards worked fine under Linux, but none of the boot options presented in the UEFI splash screen worked.

After some experimentation I discovered that UEFI only sees USB keyboards from the two front ports. Other devices such as USB drives for booting work just fine.

Raspberry Pi OS 12 and static IP address

Previous versions of Raspberry Pi Os (previously Raspbian) uses dhcpcd to configure static IP address. From the version 12 onward they defaulted to Network Manager instead. The official documentation unhelpfully suggests using your router to give it a static IP, but in various homelab use cases you might want the device to come up with a correct IP even if the DHCP server or something else failed upstream to the Raspberry Pi.

Many are familiar with Network Manager's GUI configuration, but its command line tool nmcli may be a new acquaintance.

After the default installation you will have a network configuration called "Wired connection 1" and it uses DHCP. We are going to replace that with static example IP of 192.168.123.50, using router .1 and the Google public DNS 8.8.8.8. Note that you either need to be root or sudo the following commands:

nmcli con mod "Wired connection 1" ipv4.address "192.168.123.50/24" ipv4.gateway "192.168.123.1" ipv4.dns "8.8.8.8" ipv4.method manual

Unless it gives you an error, you can activate this new configuration by:

nmcli con up "Wired connection 1"

The actual configuration is stored under /etc/NetworkManager/system-connections and can be edited with a standard text editor.


Roborock S8 won't connect

I was asked to look into freshly bought Roborock S8 in a case where the device's initial setup could not be completed. The owner had been running circles resetting anything from the vacuum cleaner to their phone's settings and had reached a total frustration with their tech support.

This device is technically curious. It's very expensive and it starts with claiming to be incompatible with pretty much anything, including 5GHz WiFi, longer WiFi passwords or anything but letters and numbers in them and the list goes on.

Because the vacuum cleaner couldn't handle 5GHz network, even with same named 2.4GHz network available, a separate network with just the wanted frequency range was set up for it. That didn't help. When setting up the device, the process progressed to the point where the vacuum cleaner was connecting to the WiFi and then it just stopped there.

A closer examination from Unifi dash board and on the firewall revealed curious behavior: the device did connect to its WiFi, got an IP, but then it just sat there. I assumed it would have wanted to call home for firmware upgrade, but there was no meaningful traffic outwards or towards the cellphone running its control app.

Looking at the frustratingly long shotgun approach the technical support had given them, one item caught my eye. They claimed the device is not compatible with networks that have their router at 192.168.8.1. Well that's certainly a curious limitation. This was also exactly what the owner had as their LAN address space. For the laughs, I changed the LAN's IP range to something else and would you look at that: the vacuum cleaner instantly called home and worked.

CUPS won't print in color

After replacing my old color laser, I came across an issue where no matter what, I couldn't print in color. The printing software was set to use color, local CUPS was set to color, the CUPS server was set to color. Even if added as a local printer to my workstation's CUPS, the color was still MIA.

I learned that CUPS has hidden secondary setting for colors, that is not exposed to its web GUI. The actual master switch for color or monochrome is hidden away under "other options" in CUPS' graphical GUI. If you're administrating a CUPS server without access to the graphical GUI, the setting is in /etc/cups/printers.conf, as Option printer-color-mode. It defaults to monochrome.

The reason it defaults to color appears to be a bug report against CUPS by a large school's administrator, who complained that the students waste so much color because of the default. I can only assume this change also created the huge confusion for us, who have used CUPS since forever, but have never encountered this kind of a hidden default master switch value.

To fix this, you have two options: through a GUI that reveals the "Other Options" or just editing the printers.conf directly.

The GUI in many GTK based things appears to be a fast way, find it in your settings dialog or just run system-config-printer from a terminal.

Graphical settings dialog
















The setting is well hidden under Job Options, towards the bottom under "more" and Other Options, as print-color-mode. Flip that to "color" and save.

Other option is to stop the CUPS daemon and editing /etc/cups/printers.conf.

Look for "Option printer-color-mode monochrome" and change "monochrome" to "color", then start the CUPS daemon.

The world is no longer in greyscale.



Ubuntu, Debian and Raspbian (or Raspberry Pi OS) custom boot screen

This appears to be a topic where you find a lot of misinformation on the web. Most include modifying the existing Pix Plymouth theme in-place as no one giving all the advice seems to know how to actually create new ones. The outcome with modifying the Pix theme in-place is that the next time the Plymouth themes package updates, your changes will be overwritten. Also there's plenty of neat themes out there, waiting to be installed.

This is how you actually clone the Pix-theme and make your own splash screen. Pix is a good theme for an easy hack, as it's virtually nothing but a scaled picture, with a text underneath saying what's currently starting up. You'll find it from Raspbian's rdp-plym-splash -package and it'll work on anything that has Plymouth.

If you already have a theme, but want to change it, scroll down to the second part.

Creating your own theme

The themes live in /usr/share/plymouth/themes. If you're on Raspbian, you can copy the "pix" directory into a new name. If you're not on Raspbian, download the package mentioned earlier and extract the pix-directory from it and put it in the theme directory, then rename it. We are going to call our new theme "demo" here.

After copying the pix directory as demo, go into it and rename the pix.plymouth and pix.script as demo.plymouth and demo.script.

Edit demo.plymouth, change the name-parameter to demo and fix the ImageDir and ScriptFile to point to the freshly created directory and renamed script file.

When you're done, replace the splash.png with whatever .png image you like, while keeping the name. Your custom theme is now complete.

Changing the theme on Debian or Ubuntu

Debian and Ubuntu manage the active Plymouth theme via update-alternatives, as a alternative called default.plymouth. For your new custom theme to show up, you first need to add it, then change to it.

As root or with sudo, run:

update-alternatives --install /usr/share/plymouth/themes/default.plymouth default.plymouth /usr/share/plymouth/themes/demo/demo.plymouth 100


update-alternatives --config default.plymouth

Select the freshly added theme, them rebuild the initramfs, which will contain the boot splash.

update-initramfs -u

On the next boot you'll see your new splash screen.

Changing the theme on Raspberry Pi OS

Raspberry Pi OS uses its own script instead of update-alternatives. First verify it can see your freshly added theme with:

plymouth-set-default-theme --list

If it's visible, switch to it:

plymouth-set-default-theme demo

On the next boot your new splash screen will be seen.

Lenovo ST50 and Unqualified DIMM error message

I have various low-end servers running my home lab stuff. After IBM killed their X3100 series, I went with Lenovo ThinkServers and they have done the job for over a decade in that purpose.

Starting with ThinkServer TS150 and DDR4 DIMMs they added a feature that checks the DIMMs for Lenovo signature and not finding it logs an error in the system log about using unqualified DIMMs. This used to be a minor inconvenience.

Starting with ThinkSystem ST50, they finally did what I had feared: it's no longer just a note in the system log, it will complain about it on every boot, requiring the user to press F2 to acknowledge the huge injustice of using non-Lenovo sanctioned DIMMs.

Save some hardware hack that emulates USB keyboard and sends an F2 key press or stabbing the firmware, there's no way around this.

Keep this in mind if you plan to use one for you homelab.

Ubiquiti UNVR disk size upgrade

I installed an Ubiquiti UNVR with couple of 4T disks I had at hand while waiting for the proper sized ones to be back in stock. Once the new drives arrived, I had originally intended to swap them out at once, nuking the recordings in the process. However there's an undocumented way to upgrade with no data loss at all.

The trick is to replace the disks one by one, letting the system rebuild the RAID between the disk swaps, then reboot the system after it has synced the last disk. Before the reboot the array appears to match the size of the old disks, even when the storage configuration page shows the correct size. A quick investigation over SSH confirmed this; the RAID device had only used the old disks' amount of space out of the new disks. Rebooting the system triggered it to grow the RAID and the EXT4 filesystem on it.

This works at least for UNVR and UNVR Pro and was tested on UniFi OS 3.0.16.

Updating BackupPC from series 3 to series 4

A lot of Linux distributions carried BackupPC v3 for a long time even when the 4-series had been stable for years. As a result systems that got installed before Debian 11 or Ubuntu server 22.04 were also using the older BackupPC variant.

BackupPC had a major overhaul of the backup storage format between versions 3 and 4, one of the big selling points being a lot better performance which was a very welcome thing for all of us who manage multi-terabyte installations.

BackupPC 4 is backwards compatible with the BackupPC 3 format and new backups are made with the improved version. However the existing backups won't be converted automatically. So if you were running Debian 10 or Ubuntu server 20.04 and upgraded to a current version, here's how to deal with BackupPC.

BackupPC processes and file ownership are for user "backuppc" in Debian and Ubuntu. You can not run any of the commands as root, switch to the backuppc user before running any of them. BackupPC daemon also can not run at the same time. The v4 format uses a lot more inodes than the older, so make you you either have enough free inodes with ext4 (df -i, see you're below 40% or so) or use a filesystem where preallocated inode count is not an issue (such as ZFS, btrfs and so forth). 

1. Preparations

If you're running BackupPC in a virtual machine and you have enough space to snapshot it, this might be a good spot to do so.

You may want to remove incremental backups from all hosts. I have come across several cases where full backups migrate just fine and incrementals cause the migration script to error out. YMMV.

Stop the BackupPC service:

systemctl stop backuppc

Switch to backuppc user. Since it does not have a password set by default, you have to go through the root user. 

2. Test with one host

Select one of the backup hosts as a test subject and make a dry run on it. In this example the -m switch causes the migration script to just print what it would do and -h specifies the host name. 

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -m -h example_host

This will list the backups and file counts for that host that exist in the v3 storage. If it looks good, migrate the host by omitting the -m switch.

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -h example_host

Since this will include copying files over, it will take from hours to days. Note that the first one will always be the slowest, as it will not need to copy files over if the file already exists in the new pool.

The outcome looks something like this:

BackupPC_migrateV3toV4: migrating host example_host backup #966 to V4 (approx 591362 files)
example_host #966: 0.0%
example_host #966: 0.3%
...
example_host #966: 99.6%
example_host #966: 99.8%
BackupPC_migrateV3toV4: converted backup in /var/lib/backuppc/pc/example_host/966; removing /var/lib/backuppc/pc/example_host/966.old

Once you're done with your test host and it looks good, it's time to convert the rest of them.

3. Migrate all hosts

Depending on your backup history, converting a huge system may easily take days. Take this in account as you won't be making new backups while the migration is running.

Migrating all hosts happens with the -a switch:

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -a

If you're accessing the system remotely, it's an extremely good idea to run this in a screen.

4. Cleaning up

After the migration start BackupPC daemon again:

systemctl start backuppc

Still using the backuppc user, force a full nighly maintenance, so it cleans up everything that was left in the old pool:

/usr/share/backuppc/bin/BackupPC_nightly 0 255

Once it's complete, go into the BackupPC web GUI, navigate to Edit Config -> Server and uncheck PoolV3Enabled. Save settings.

Navigate to Admin Options and click Reload.

The pool is now migrated.

Raspberry Pi OS 11 (2022 onwards) headless setup

Raspbian used to have its default user "pi" with a well known password. Naturally people were expected to nuke the user and create their own or at least change the password. Unfortunately we now have approximately bazillion Raspberries with public IPs and the default password in place, so the developers finally retired the concept of default user when they went from Debian 10 to 11.

Raspbian 10's renamed successor, Raspberry Pi OS expects you to use their GUI tool to write the card images. It creates the default user among couple of other options. But under the hood it doesn't actually add users to the Linux system, it just creates couple of text files that are used to configure the freshly installed system on its first boot. This approach makes it possible to make the image writer software portable, as all the major OSes support the FAT-partition Raspberry uses to boot.

If you don't want to use the official tool, just download the image, write it onto a SD card and then mount the first partition on a temporary directory. You can create two extra files here to create your user on the first boot: an empty file called ssh (or ssh.txt) to enable SSH by default and userconf (or userconf.txt) to create a new user. The userconf file is in the format of "username:password hash" and can be generated with an oneliner such as this:

(echo -n "pi:" ; openssl passwd -6 -stdin <<< 'raspberry') > userconf

Change the name and password as wanted, in the example they are "pi" and "raspberry".


Ubuntu 22.04 and held back packages

I came across a situation in Ubuntu 22.04 where a typical apt upgrade upgraded some packages, but left some packages hanging with the error "The following packages have been kept back". For years, a go-to way to deal with this is apt dist-upgrade which would install the packages that need upgrades, but would also pull in the new dependencies in them.

Unlike with previous Ubuntu versions, this is actually a feature called phased updates. The long story short, only a number of systems receive the updates when released and if they don't cause problems, some days later everyone gets it. You can use apt-cache policy with the package name as a parameter to see this in action, the output should list the newest version as "phased" with percentage on it.

You can force the installation of the held packages with apt -o APT::Get::Always-Include-Phased-Updates=true upgrade, but keep in mind that this is intended feature and that you will get the packages couple of days later, but with an added peace of mind that they didn't didn't cause problems for the test group.

Dealing with stuck iCloud Drive upload

With MacOS 11 and 12 iCloud Drive seems to get stuck at random when the computer sleeps. The symptom is that Finder shows stuck progress pie as iCloud Drive's status and clicking it opens a window that shows 1 or more items stuck at uploading or downloading. A common suggestion that usually helps is rebooting the computer, but it appears that only one process needs restarting to recover. The process' name is "bird".

Locate the process with:

pgrep bird

The command should output the process' PID number. Use the kill command to stop it and it'll restart automatically within couple of seconds. No need for sudo, it runs as your own user.

kill (PID_NUMBER_HERE)


Convert Debian (or Ubuntu) system from BIOS boot to EFI

When upgrading hardware, some newer hardware tends to be allergic towards old BIOS booting. Fortunately converting is relatively simple once you figure out all the pieces needed.

The main points are:

1. Boot off a rescue CD, add EFI partition, mount filesystems

2. Remove grub-pc

3. Install grub-efi

Here's a closer look at it:

 

1. Boot off a rescue CD

I use SystemRescueCD 9 here, but probably any version works. Important: boot the CD image in EFI mode! Otherwise you won't be able to add grub-efi to the EFI boot menu and the system won't boot.
 
After it has booted, we need a directory for our root partition. I'll use /mnt/target here, the system's root is assumed /dev/sda1, use tools like lsblk and blkid to identify the correct one in yours.
 
We will need access to EFI parameter store, so we first need to modprobe efivarfs.
 
First verify with a partition editor, such as cfdisk that the disk uses GPT partition scheme. If not, you have to convert it first in-place with  gdisk /dev/sda. Gdisk will load the MBR partition table, after which you can use the w-command to write the GPT partition table over it. This does not affect the partitions themselves or the data on them.
 
Grub's EFI parts needs a vfat partition to live in. It only needs some megabytes, 200 megabyte partition is enough. Partition type for it is EFI System. One quick hack is to make the swap partition this much smaller. It can be then formatted to vfat with mkfs.vfat. The freshly created partition will be /dev/sda3 here.
 
Next we mount the system root directory and bind-mount the system directories from the live CD's ramdisk root. Without these bind mounts grub-install won't work. We also mount efivarfs.

mount /dev/sda1 /mnt/target
mkdir -v /mnt/target/boot/efi
mount /dev/sda3 /mnt/target/boot/efi
mount -o bind /sys /mnt/target/sys
mount -o bind /proc /mnt/target/proc
mount -o bind /dev /mnt/target/dev
mount -o bind /dev/pts /mnt/target/dev/pts
mount -t efivarfs none /mnt/target/sys/firmware/efi/efivars
 
Now that we've gotten everything mounted, we chroot into the real system.
 
chroot /mnt/target /bin/bash

Hopefully we now find ourselves inside the target system. First thing we do here is to add the new efi partition to the /etc/fstab.

/dev/sda3  /boot/efi  vfat  umask=0077  0  0
 

2. Remove grub-pc

The old grub conflicts with the grub-efi, so we get rid of it:

apt --purge remove grub-pc

It'll most likely remove its dependencies as well.

3. Install grub-efi

apt install grub-efi shim-signed
 
It'll also install couple of handy utilities for poking EFI, like efibootmgr. The shim-signed package makes it possible to use secure boot.

Next we need to install grub's files on the freshly created vfat partition. Note that this command does not take the partition, but the device where the root drive is installed. Since the root here was /dev/sda1, it's just /dev/sda.
 
grub-install /dev/sda
 
Look at carefully what it says. If it errors out, there are couple of things that might have been wrong.  First thing to check with errors is if you actually booted in EFI mode or BIOS. If it complains about not being able to modify EFI variables, the efivarfs module is missing or it was not mounted.

If everything went okay, the files EFI need are now placed on the vfat partition, but we still need the menu entry for the OS. Update-grub does this for us. We also need to update the existing initramfs.

update-grub
update-initramfs -u

It should mention that it adds the boot menu entry. We can then use efibootmgr to investigate:

efibootmgr

Spot the Debian or Ubuntu entry in the list and see that it's listed in the BootOrder. The other options vary depending on the motherboard or virtualization platform.

If the efibootmgr shows the entry, you're now ready to boot. Exit from the chroot and reboot the machine. You should be greeted with grub-efi's boot menu now.

Speedtest.net's own cli client vs. Ubuntu's and Debian's

I monitor a remote location with a 4G link and one of the metrics I'm interested in is the weather's effect to the quality of the connection. Debian and Ubuntu carry speedtest.net's console client, but no matter what I do, it's outcome is random at best. The same connection over the web browser or app version shows a rather stable 50-something megs, while the speedtest-cli fluctuates wildly between 20 and 50 megs between the tests.

I had assumed the Linux client offered by speedtest.net was exactly same thing carried by distributions, but apparently I was wrong. With their own command line client I can finally see what I believe to be realistic values.

Installation instructions for it are here: https://www.speedtest.net/apps/cli

lm-sensors coretemp under XCP-ng 8.2.1

Xen appears to do something funny and the default coretemp-module for Intel's CPUs refuses to co-operate in domain 0. To make lm-sensors work under XCP-ng, there's a package called "coretemp-module-alt" which works under Xen. Just install and modprobe it.