Ubiquiti UNVR disk size upgrade

I installed an Ubiquiti UNVR with couple of 4T disks I had at hand while waiting for the proper sized ones to be back in stock. Once the new drives arrived, I had originally intended to swap them out at once, nuking the recordings in the process. However there's an undocumented way to upgrade with no data loss at all.

The trick is to replace the disks one by one, letting the system rebuild the RAID between the disk swaps, then reboot the system after it has synced the last disk. Before the reboot the array appears to match the size of the old disks, even when the storage configuration page shows the correct size. A quick investigation over SSH confirmed this; the RAID device had only used the old disks' amount of space out of the new disks. Rebooting the system triggered it to grow the RAID and the EXT4 filesystem on it.

This works at least for UNVR and UNVR Pro and was tested on UniFi OS 3.0.16.

Updating BackupPC from series 3 to series 4

A lot of Linux distributions carried BackupPC v3 for a long time even when the 4-series had been stable for years. As a result systems that got installed before Debian 11 or Ubuntu server 22.04 were also using the older BackupPC variant.

BackupPC had a major overhaul of the backup storage format between versions 3 and 4, one of the big selling points being a lot better performance which was a very welcome thing for all of us who manage multi-terabyte installations.

BackupPC 4 is backwards compatible with the BackupPC 3 format and new backups are made with the improved version. However the existing backups won't be converted automatically. So if you were running Debian 10 or Ubuntu server 20.04 and upgraded to a current version, here's how to deal with BackupPC.

BackupPC processes and file ownership are for user "backuppc" in Debian and Ubuntu. You can not run any of the commands as root, switch to the backuppc user before running any of them. BackupPC daemon also can not run at the same time. The v4 format uses a lot more inodes than the older, so make you you either have enough free inodes with ext4 (df -i, see you're below 40% or so) or use a filesystem where preallocated inode count is not an issue (such as ZFS, btrfs and so forth). 

1. Preparations

If you're running BackupPC in a virtual machine and you have enough space to snapshot it, this might be a good spot to do so.

You may want to remove incremental backups from all hosts. I have come across several cases where full backups migrate just fine and incrementals cause the migration script to error out. YMMV.

Stop the BackupPC service:

systemctl stop backuppc

Switch to backuppc user. Since it does not have a password set by default, you have to go through the root user. 

2. Test with one host

Select one of the backup hosts as a test subject and make a dry run on it. In this example the -m switch causes the migration script to just print what it would do and -h specifies the host name. 

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -m -h example_host

This will list the backups and file counts for that host that exist in the v3 storage. If it looks good, migrate the host by omitting the -m switch.

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -h example_host

Since this will include copying files over, it will take from hours to days. Note that the first one will always be the slowest, as it will not need to copy files over if the file already exists in the new pool.

The outcome looks something like this:

BackupPC_migrateV3toV4: migrating host example_host backup #966 to V4 (approx 591362 files)
example_host #966: 0.0%
example_host #966: 0.3%
...
example_host #966: 99.6%
example_host #966: 99.8%
BackupPC_migrateV3toV4: converted backup in /var/lib/backuppc/pc/example_host/966; removing /var/lib/backuppc/pc/example_host/966.old

Once you're done with your test host and it looks good, it's time to convert the rest of them.

3. Migrate all hosts

Depending on your backup history, converting a huge system may easily take days. Take this in account as you won't be making new backups while the migration is running.

Migrating all hosts happens with the -a switch:

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -a

If you're accessing the system remotely, it's an extremely good idea to run this in a screen.

4. Cleaning up

After the migration start BackupPC daemon again:

systemctl start backuppc

Still using the backuppc user, force a full nighly maintenance, so it cleans up everything that was left in the old pool:

/usr/share/backuppc/bin/BackupPC_nightly 0 255

Once it's complete, go into the BackupPC web GUI, navigate to Edit Config -> Server and uncheck PoolV3Enabled. Save settings.

Navigate to Admin Options and click Reload.

The pool is now migrated.

Raspberry Pi OS 11 (2022 onwards) headless setup

Raspbian used to have its default user "pi" with a well known password. Naturally people were expected to nuke the user and create their own or at least change the password. Unfortunately we now have approximately bazillion Raspberries with public IPs and the default password in place, so the developers finally retired the concept of default user when they went from Debian 10 to 11.

Raspbian 10's renamed successor, Raspberry Pi OS expects you to use their GUI tool to write the card images. It creates the default user among couple of other options. But under the hood it doesn't actually add users to the Linux system, it just creates couple of text files that are used to configure the freshly installed system on its first boot. This approach makes it possible to make the image writer software portable, as all the major OSes support the FAT-partition Raspberry uses to boot.

If you don't want to use the official tool, just download the image, write it onto a SD card and then mount the first partition on a temporary directory. You can create two extra files here to create your user on the first boot: an empty file called ssh (or ssh.txt) to enable SSH by default and userconf (or userconf.txt) to create a new user. The userconf file is in the format of "username:password hash" and can be generated with an oneliner such as this:

(echo -n "pi:" ; openssl passwd -6 -stdin <<< 'raspberry') > userconf

Change the name and password as wanted, in the example they are "pi" and "raspberry".


Ubuntu 22.04 and held back packages

I came across a situation in Ubuntu 22.04 where a typical apt upgrade upgraded some packages, but left some packages hanging with the error "The following packages have been kept back". For years, a go-to way to deal with this is apt dist-upgrade which would install the packages that need upgrades, but would also pull in the new dependencies in them.

Unlike with previous Ubuntu versions, this is actually a feature called phased updates. The long story short, only a number of systems receive the updates when released and if they don't cause problems, some days later everyone gets it. You can use apt-cache policy with the package name as a parameter to see this in action, the output should list the newest version as "phased" with percentage on it.

You can force the installation of the held packages with apt -o APT::Get::Always-Include-Phased-Updates=true upgrade, but keep in mind that this is intended feature and that you will get the packages couple of days later, but with an added peace of mind that they didn't didn't cause problems for the test group.

Dealing with stuck iCloud Drive upload

With MacOS 11 and 12 iCloud Drive seems to get stuck at random when the computer sleeps. The symptom is that Finder shows stuck progress pie as iCloud Drive's status and clicking it opens a window that shows 1 or more items stuck at uploading or downloading. A common suggestion that usually helps is rebooting the computer, but it appears that only one process needs restarting to recover. The process' name is "bird".

Locate the process with:

pgrep bird

The command should output the process' PID number. Use the kill command to stop it and it'll restart automatically within couple of seconds. No need for sudo, it runs as your own user.

kill (PID_NUMBER_HERE)


Convert Debian (or Ubuntu) system from BIOS boot to EFI

When upgrading hardware, some newer hardware tends to be allergic towards old BIOS booting. Fortunately converting is relatively simple once you figure out all the pieces needed.

The main points are:

1. Boot off a rescue CD, add EFI partition, mount filesystems

2. Remove grub-pc

3. Install grub-efi

Here's a closer look at it:

 

1. Boot off a rescue CD

I use SystemRescueCD 9 here, but probably any version works. Important: boot the CD image in EFI mode! Otherwise you won't be able to add grub-efi to the EFI boot menu and the system won't boot.
 
After it has booted, we need a directory for our root partition. I'll use /mnt/target here, the system's root is assumed /dev/sda1, use tools like lsblk and blkid to identify the correct one in yours.
 
We will need access to EFI parameter store, so we first need to modprobe efivarfs.
 
First verify with a partition editor, such as cfdisk that the disk uses GPT partition scheme. If not, you have to convert it first in-place with  gdisk /dev/sda. Gdisk will load the MBR partition table, after which you can use the w-command to write the GPT partition table over it. This does not affect the partitions themselves or the data on them.
 
Grub's EFI parts needs a vfat partition to live in. It only needs some megabytes, 200 megabyte partition is enough. Partition type for it is EFI System. One quick hack is to make the swap partition this much smaller. It can be then formatted to vfat with mkfs.vfat. The freshly created partition will be /dev/sda3 here.
 
Next we mount the system root directory and bind-mount the system directories from the live CD's ramdisk root. Without these bind mounts grub-install won't work. We also mount efivarfs.

mount /dev/sda1 /mnt/target
mkdir -v /mnt/target/boot/efi
mount /dev/sda3 /mnt/target/boot/efi
mount -o bind /sys /mnt/target/sys
mount -o bind /proc /mnt/target/proc
mount -o bind /dev /mnt/target/dev
mount -o bind /dev/pts /mnt/target/dev/pts
mount -t efivarfs none /mnt/target/sys/firmware/efi/efivars
 
Now that we've gotten everything mounted, we chroot into the real system.
 
chroot /mnt/target /bin/bash

Hopefully we now find ourselves inside the target system. First thing we do here is to add the new efi partition to the /etc/fstab.

/dev/sda3  /boot/efi  vfat  umask=0077  0  0
 

2. Remove grub-pc

The old grub conflicts with the grub-efi, so we get rid of it:

apt --purge remove grub-pc

It'll most likely remove its dependencies as well.

3. Install grub-efi

apt install grub-efi shim-signed
 
It'll also install couple of handy utilities for poking EFI, like efibootmgr. The shim-signed package makes it possible to use secure boot.

Next we need to install grub's files on the freshly created vfat partition. Note that this command does not take the partition, but the device where the root drive is installed. Since the root here was /dev/sda1, it's just /dev/sda.
 
grub-install /dev/sda
 
Look at carefully what it says. If it errors out, there are couple of things that might have been wrong.  First thing to check with errors is if you actually booted in EFI mode or BIOS. If it complains about not being able to modify EFI variables, the efivarfs module is missing or it was not mounted.

If everything went okay, the files EFI need are now placed on the vfat partition, but we still need the menu entry for the OS. Update-grub does this for us. We also need to update the existing initramfs.

update-grub
update-initramfs -u

It should mention that it adds the boot menu entry. We can then use efibootmgr to investigate:

efibootmgr

Spot the Debian or Ubuntu entry in the list and see that it's listed in the BootOrder. The other options vary depending on the motherboard or virtualization platform.

If the efibootmgr shows the entry, you're now ready to boot. Exit from the chroot and reboot the machine. You should be greeted with grub-efi's boot menu now.

Speedtest.net's own cli client vs. Ubuntu's and Debian's

I monitor a remote location with a 4G link and one of the metrics I'm interested in is the weather's effect to the quality of the connection. Debian and Ubuntu carry speedtest.net's console client, but no matter what I do, it's outcome is random at best. The same connection over the web browser or app version shows a rather stable 50-something megs, while the speedtest-cli fluctuates wildly between 20 and 50 megs between the tests.

I had assumed the Linux client offered by speedtest.net was exactly same thing carried by distributions, but apparently I was wrong. With their own command line client I can finally see what I believe to be realistic values.

Installation instructions for it are here: https://www.speedtest.net/apps/cli

lm-sensors coretemp under XCP-ng 8.2.1

Xen appears to do something funny and the default coretemp-module for Intel's CPUs refuses to co-operate in domain 0. To make lm-sensors work under XCP-ng, there's a package called "coretemp-module-alt" which works under Xen. Just install and modprobe it.

Convert Windows 10 virtual from MBR and BIOS boot to GPT and UEFI

I had an old Windows 10 virtual I use to test out stuff and it's been around from the times when you could upgrade from Windows 8 for free. That again was an upgraded Windows 7, so you get the picture, UEFI in Linux KVM didn't work when this thing was installed. Nowdays it works and I want to upgrade this virtual to boot from UEFI instead. Windows 10 comes with an utility to do the conversion on an existing installation, so no need to reinstall. However, it'd be a very good idea to take a snapshot of the virtual before doing this!

First Windows needs to be rebooted into rescue mode. Settings -> Update & Security -> Recover -> Advanced startup.

Next I selected Troubleshoot -> Advanced Options -> Command prompt. This made Windows reboot, ask me to log in and then let me into recovery environment command line. The utility is in \windows\system32 directory, which should be where we land when the recovery console starts. Converting takes two commands:

mbr2gpt /validate

mbr2gpt /convert

Once finished, the command line window can be closed and virtual powered off. Afterwards the boot firmware can be configured from BIOS to UEFI and secure boot can be turned on.

Hetzner Storage Box sub account CIFS path

Hetzner's storage boxes are relatively cheap way to dump virtual machine images or whatever. I wanted to create an ISO SR for XCP-ng as a sub user for the storage box. I would have expected this be pretty obvious thing but no. So, I created a sub-user in Hetzner's Robot interface, shared a directory called "iso" and the resulting path eventually was:

\\u123456.your-storagebox.de\u123456-sub1

Just heads up if you expected it to end with \backup\iso or something.

Copying Linux root filesystem on file level

Years ago I was curious about btrfs and converted one of my virtuals machines to use it. The virtual in question had used ext4 before and was converted in-place into btrfs. Later on it was learned this was in fact a bad idea as the btrfs conversion tool seems to be a gamble on data integrity.

Anyway, this system ran with converted btrfs for years, during which time I had learned that btrfs created from scratch was pretty decent for some things, but the converted ones wouldn't pass btrfs' fsck even when they would pass online scrub. This virtual was one of those cases and I decided to migrate the existing filesystem's content onto a freshly created one on file level.

Just out of curiosity, I ran btrfs check on it and it was just what I had encountered before:


# btrfs check /dev/xvdb1
Opening filesystem to check...
Checking filesystem on /dev/xvdb1
UUID: 8c7512e2-3613-474d-9234-835a57b6f896
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 5 inode 1177668 errors 8000, inline file extent too large
root 5 inode 1177752 errors 8000, inline file extent too large
--- bazillion lines cut ---
root 5 inode 1831684 errors 8000, inline file extent too large
root 5 inode 2878563 errors 8000, inline file extent too large
ERROR: errors found in fs roots
found 7227060224 bytes used, error(s) found
total csum bytes: 6377088
total tree bytes: 159498240
total fs tree bytes: 132104192
total extent tree bytes: 16334848
btree space waste bytes: 40630997
file data blocks allocated: 120313016320
 referenced 6451007488

While the errors are numerous and scary, there's apparently no data loss. I created a fresh btrfs filesystem, mounted them both and then created one monster of a rsync command:

rsync -axHAWXS --numeric-ids --progress /mnt/source/ /mnt/target

Do note the lack if last slash there in the target! After rsync was finished, I just had to use grub-install to write a new boot track on the freshly created virtual disk and this Debian 11 based system booted off fine and continues collecting data with Munin.

FreeNAS 12, rsync code 10 and compression

FreeNAS (or TrueNAS core nowadays) started to misbehave with its rsync. My backups jobs had started to fail inexplicably with error code 10. 

A closer inspection revealed they had deprecated the old -z (same as --compress) option compression. Changing it to -zz (--new-compress) made it work again.

Hetzner, IPv6 and routing with PfSense

Hetzner has a peculiar way of handling IPv6. They'll point your IPv6 block onto one MAC address and it's up to you how you deal with it. There's numerous examples out there how to do it with KVM based systems, such as just plain Linux installations running KVM virtuals or dedicated KVM distress, such as Proxmox. But what if you run VMWare ESXi, Citrix XenServer or its excellent open source version, XCP-ng?

Hetzner's go-to answer is installing a Linux virtual, pointing the IPv6 block to it and then letting the virtual handle the routing. I'm sure this works just fine, but I saw the whole concept as an overkill as there's dedicated firewall distributions that run with lean resources and are very easy to back up and maintain.

So, I thought PfSense would have been documented in a million places. Nope, just those lone forum posts of "Does anyone know how to do this?" and then either zero replies, or replies that didn't contribute anything. So, here's how I did it and it works just fine. I make the assumption that ESXi or Xen is already installed and you know how to use it.

First, go into Hetzner's Robot interface and get yourself an extra IPv4 address. Once it's set up, request a separate MAC address for it (little network card and screen icon next to the IP). Make a note of the new MAC address you get.

Create a new virtual machine. Set it up for FreeBSD template and install PfSense on it. For WAN interface, use the MAC address you got previously from Robot interface.

Configure the IPv4 address you got from Hetzner, then get decide what'll be the new router PfSense's IP. I'm going to use 123:456:678::1 here. Select 64bit netmask. Use fe80::1 as a gateway for IPv6.

From PfSense's settings, go to Advanced -> Networking and make sure Allow IPv6 is checked.

Next go to Firewall -> Aliases and create a new alias, type "Host(s)". I'll call it HETZNER_VIRTUALS here. Add all the IPs your virtual machines will use here. For this example, I'll use 123:456:678::20.

Next go Firewall -> Rules and create three rules for the WAN interface:

Rule 1: "Block Hetzner virtuals  to firewall"

Action: Reject 

Address Family: IPv6

Source: Single host or alias -> HETZNER_VIRTUALS

Destination: This firewall (self)


Rule 2: "Allow ANY to Hetzner virtuals"

Action: Pass 

Address Family: IPv6

Source: ANY

Destination: Single host or alias -> HETZNER_VIRTUALS

 

Rule 3: "Allow Hetzner virtuals to ANY"

Action: Pass

Address Family: IPV6 

Source: Single host or alias -> HETZNER_VIRTUALS

Destination: ANY

 

If you're wondering, you could also use the whole block for the rules, but I personally preferred to just route to the machines I have. At this point you should be able to use ping6 to ping hosts outside from PfSense's shell.

Now what PfSense is set up, install your guest OS. I'll use 123:456:678::20 for my example and 123:456:678::1 as its gateway. Note that you may need to create reverse DNS for the IP for it to actually work. 

After the installation traceroute6 should show it goes through 123:456:678::1 on its way out and the same thing the other direction.