Seafile web GUI getting stuck at random with Apache

I have ran my Seafile server with an Apache front end under Debian for closer to a decade now. I've never had any meaningful issues with it, but I encountered an issue with freshly set up instance where the web GUI would get stuck loading at random. It happened often just after login, but would also happen when moving in directories and viewing files.

After very careful comparison to a working one, I noticed the working one used the Apaches prefork MPM and the problematic one again the event MPM. I remembered a case from a distant past, where FastCGI had issues with the worker MPM and the event MPM is a close relative to it.

After switching the problematic instance from event to prefork MPM, Seafile's web GUI issues went away.

Huge (or tiny) mouse cursor in Firefox Snap under Ubuntu.

Ever since Ubuntu 22.04 was released, Firefox's mouse cursor has been acting out with XFCE. I have a custom cursor theme which the snap version of Firefox can't seem to use. After some investigation, it seems Firefox pulls the cursor size from whatever XFCE's mouse cursor settings control, but the locally created mouse cursor doesn't scale.

To get the mouse cursor in in Firefox onto a bit more manageable size (mine was like 2cm tall in a HD resolution display!), go to XFCE's Mouse and Touchpad settings, theme tab and adjust it to more saner size there.

Can't enter UEFI setup with F2 on an Intel NUC

Repurposing a stack of old NUCs I came across an odd thing. The NUCs' keyboards worked fine under Linux, but none of the boot options presented in the UEFI splash screen worked.

After some experimentation I discovered that UEFI only sees USB keyboards from the two front ports. Other devices such as USB drives for booting work just fine.

Raspberry Pi OS 12 and static IP address

Previous versions of Raspberry Pi Os (previously Raspbian) uses dhcpcd to configure static IP address. From the version 12 onward they defaulted to Network Manager instead. The official documentation unhelpfully suggests using your router to give it a static IP, but in various homelab use cases you might want the device to come up with a correct IP even if the DHCP server or something else failed upstream to the Raspberry Pi.

Many are familiar with Network Manager's GUI configuration, but its command line tool nmcli may be a new acquaintance.

After the default installation you will have a network configuration called "Wired connection 1" and it uses DHCP. We are going to replace that with static example IP of 192.168.123.50, using router .1 and the Google public DNS 8.8.8.8. Note that you either need to be root or sudo the following commands:

nmcli con mod "Wired connection 1" ipv4.address "192.168.123.50/24" ipv4.gateway "192.168.123.1" ipv4.dns "8.8.8.8" ipv4.method manual

Unless it gives you an error, you can activate this new configuration by:

nmcli con up "Wired connection 1"

The actual configuration is stored under /etc/NetworkManager/system-connections and can be edited with a standard text editor.

Roborock S8 won't connect

I was asked to look into freshly bought Roborock S8 in a case where the device's initial setup could not be completed. The owner had been running circles resetting anything from the vacuum cleaner to their phone's settings and had reached a total frustration with their tech support.

This device is technically curious. It's very expensive and it starts with claiming to be incompatible with pretty much anything, including 5GHz WiFi, longer WiFi passwords or anything but letters and numbers in them and the list goes on.

Because the vacuum cleaner couldn't handle 5GHz network, even with same named 2.4GHz network available, a separate network with just the wanted frequency range was set up for it. That didn't help. When setting up the device, the process progressed to the point where the vacuum cleaner was connecting to the WiFi and then it just stopped there.

A closer examination from Unifi dash board and on the firewall revealed curious behavior: the device did connect to its WiFi, got an IP, but then it just sat there. I assumed it would have wanted to call home for firmware upgrade, but there was no meaningful traffic outwards or towards the cellphone running its control app.

Looking at the frustratingly long shotgun approach the technical support had given them, one item caught my eye. They claimed the device is not compatible with networks that have their router at 192.168.8.1. Well that's certainly a curious limitation. This was also exactly what the owner had as their LAN address space. For the laughs, I changed the LAN's IP range to something else and would you look at that: the vacuum cleaner instantly called home and worked.

CUPS won't print in color

After replacing my old color laser, I came across an issue where no matter what, I couldn't print in color. The printing software was set to use color, local CUPS was set to color, the CUPS server was set to color. Even if added as a local printer to my workstation's CUPS, the color was still MIA.

I learned that CUPS has hidden secondary setting for colors, that is not exposed to its web GUI. The actual master switch for color or monochrome is hidden away under "other options" in CUPS' graphical GUI. If you're administrating a CUPS server without access to the graphical GUI, the setting is in /etc/cups/printers.conf, as Option printer-color-mode. It defaults to monochrome.

The reason it defaults to color appears to be a bug report against CUPS by a large school's administrator, who complained that the students waste so much color because of the default. I can only assume this change also created the huge confusion for us, who have used CUPS since forever, but have never encountered this kind of a hidden default master switch value.

To fix this, you have two options: through a GUI that reveals the "Other Options" or just editing the printers.conf directly.

The GUI in many GTK based things appears to be a fast way, find it in your settings dialog or just run system-config-printer from a terminal.

The setting is well hidden under Job Options, towards the bottom under "more" and Other Options, as print-color-mode. Flip that to "color" and save.

Other option is to stop the CUPS daemon and editing /etc/cups/printers.conf.

Look for "Option printer-color-mode monochrome" and change "monochrome" to "color", then start the CUPS daemon.

The world is no longer in greyscale.

Ubuntu, Debian and Raspbian (or Raspberry Pi OS) custom boot screen

This appears to be a topic where you find a lot of misinformation on the web. Most include modifying the existing Pix Plymouth theme in-place as no one giving all the advice seems to know how to actually create new ones. The outcome with modifying the Pix theme in-place is that the next time the Plymouth themes package updates, your changes will be overwritten. Also there's plenty of neat themes out there, waiting to be installed.

This is how you actually clone the Pix-theme and make your own splash screen. Pix is a good theme for an easy hack, as it's virtually nothing but a scaled picture, with a text underneath saying what's currently starting up. You'll find it from Raspbian's rdp-plym-splash -package and it'll work on anything that has Plymouth.

If you already have a theme, but want to change it, scroll down to the second part.

Creating your own theme

The themes live in /usr/share/plymouth/themes. If you're on Raspbian, you can copy the "pix" directory into a new name. If you're not on Raspbian, download the package mentioned earlier and extract the pix-directory from it and put it in the theme directory, then rename it. We are going to call our new theme "demo" here.

After copying the pix directory as demo, go into it and rename the pix.plymouth and pix.script as demo.plymouth and demo.script.

Edit demo.plymouth, change the name-parameter to demo and fix the ImageDir and ScriptFile to point to the freshly created directory and renamed script file.

When you're done, replace the splash.png with whatever .png image you like, while keeping the name. Your custom theme is now complete.

Changing the theme on Debian or Ubuntu

Debian and Ubuntu manage the active Plymouth theme via update-alternatives, as a alternative called default.plymouth. For your new custom theme to show up, you first need to add it, then change to it.

As root or with sudo, run:

update-alternatives --install /usr/share/plymouth/themes/default.plymouth default.plymouth /usr/share/plymouth/themes/demo/demo.plymouth 100

update-alternatives --config default.plymouth

Select the freshly added theme, them rebuild the initramfs, which will contain the boot splash.

update-initramfs -u

On the next boot you'll see your new splash screen.

Changing the theme on Raspberry Pi OS

Raspberry Pi OS uses its own script instead of update-alternatives. First verify it can see your freshly added theme with:

plymouth-set-default-theme --list

If it's visible, switch to it:

plymouth-set-default-theme demo

On the next boot your new splash screen will be seen.

Lenovo ST50 and Unqualified DIMM error message

I have various low-end servers running my home lab stuff. After IBM killed their X3100 series, I went with Lenovo ThinkServers and they have done the job for over a decade in that purpose.

Starting with ThinkServer TS150 and DDR4 DIMMs they added a feature that checks the DIMMs for Lenovo signature and not finding it logs an error in the system log about using unqualified DIMMs. This used to be a minor inconvenience.

Starting with ThinkSystem ST50, they finally did what I had feared: it's no longer just a note in the system log, it will complain about it on every boot, requiring the user to press F2 to acknowledge the huge injustice of using non-Lenovo sanctioned DIMMs.

Save some hardware hack that emulates USB keyboard and sends an F2 key press or stabbing the firmware, there's no way around this.

Keep this in mind if you plan to use one for you homelab.

Ubiquiti UNVR disk size upgrade

I installed an Ubiquiti UNVR with couple of 4T disks I had at hand while waiting for the proper sized ones to be back in stock. Once the new drives arrived, I had originally intended to swap them out at once, nuking the recordings in the process. However there's an undocumented way to upgrade with no data loss at all.

The trick is to replace the disks one by one, letting the system rebuild the RAID between the disk swaps, then reboot the system after it has synced the last disk. Before the reboot the array appears to match the size of the old disks, even when the storage configuration page shows the correct size. A quick investigation over SSH confirmed this; the RAID device had only used the old disks' amount of space out of the new disks. Rebooting the system triggered it to grow the RAID and the EXT4 filesystem on it.

This works at least for UNVR and UNVR Pro and was tested on UniFi OS 3.0.16.

Updating BackupPC from series 3 to series 4

A lot of Linux distributions carried BackupPC v3 for a long time even when the 4-series had been stable for years. As a result systems that got installed before Debian 11 or Ubuntu server 22.04 were also using the older BackupPC variant.

BackupPC had a major overhaul of the backup storage format between versions 3 and 4, one of the big selling points being a lot better performance which was a very welcome thing for all of us who manage multi-terabyte installations.

BackupPC 4 is backwards compatible with the BackupPC 3 format and new backups are made with the improved version. However the existing backups won't be converted automatically. So if you were running Debian 10 or Ubuntu server 20.04 and upgraded to a current version, here's how to deal with BackupPC.

BackupPC processes and file ownership are for user "backuppc" in Debian and Ubuntu. You can not run any of the commands as root, switch to the backuppc user before running any of them. BackupPC daemon also can not run at the same time. The v4 format uses a lot more inodes than the older, so make you you either have enough free inodes with ext4 (df -i, see you're below 40% or so) or use a filesystem where preallocated inode count is not an issue (such as ZFS, btrfs and so forth). 

1. Preparations

If you're running BackupPC in a virtual machine and you have enough space to snapshot it, this might be a good spot to do so.

You may want to remove incremental backups from all hosts. I have come across several cases where full backups migrate just fine and incrementals cause the migration script to error out. YMMV.

Stop the BackupPC service:

systemctl stop backuppc

Switch to backuppc user. Since it does not have a password set by default, you have to go through the root user. 

2. Test with one host

Select one of the backup hosts as a test subject and make a dry run on it. In this example the -m switch causes the migration script to just print what it would do and -h specifies the host name. 

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -m -h example_host

This will list the backups and file counts for that host that exist in the v3 storage. If it looks good, migrate the host by omitting the -m switch.

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -h example_host

Since this will include copying files over, it will take from hours to days. Note that the first one will always be the slowest, as it will not need to copy files over if the file already exists in the new pool.

The outcome looks something like this:

BackupPC_migrateV3toV4: migrating host example_host backup #966 to V4 (approx 591362 files)

example_host #966: 0.0%

example_host #966: 0.3%

...

example_host #966: 99.6%

example_host #966: 99.8%

BackupPC_migrateV3toV4: converted backup in /var/lib/backuppc/pc/example_host/966; removing /var/lib/backuppc/pc/example_host/966.old

Once you're done with your test host and it looks good, it's time to convert the rest of them.

3. Migrate all hosts

Depending on your backup history, converting a huge system may easily take days. Take this in account as you won't be making new backups while the migration is running.

Migrating all hosts happens with the -a switch:

/usr/share/backuppc/bin/BackupPC_migrateV3toV4 -a

If you're accessing the system remotely, it's an extremely good idea to run this in a screen.

4. Cleaning up

After the migration start BackupPC daemon again:

systemctl start backuppc

Still using the backuppc user, force a full nighly maintenance, so it cleans up everything that was left in the old pool:

/usr/share/backuppc/bin/BackupPC_nightly 0 255

Once it's complete, go into the BackupPC web GUI, navigate to Edit Config -> Server and uncheck PoolV3Enabled. Save settings.

Navigate to Admin Options and click Reload.

The pool is now migrated.

Raspberry Pi OS 11 (2022 onwards) headless setup

Raspbian used to have its default user "pi" with a well known password. Naturally people were expected to nuke the user and create their own or at least change the password. Unfortunately we now have approximately bazillion Raspberries with public IPs and the default password in place, so the developers finally retired the concept of default user when they went from Debian 10 to 11.

Raspbian 10's renamed successor, Raspberry Pi OS expects you to use their GUI tool to write the card images. It creates the default user among couple of other options. But under the hood it doesn't actually add users to the Linux system, it just creates couple of text files that are used to configure the freshly installed system on its first boot. This approach makes it possible to make the image writer software portable, as all the major OSes support the FAT-partition Raspberry uses to boot.

If you don't want to use the official tool, just download the image, write it onto a SD card and then mount the first partition on a temporary directory. You can create two extra files here to create your user on the first boot: an empty file called ssh (or ssh.txt) to enable SSH by default and userconf (or userconf.txt) to create a new user. The userconf file is in the format of "username:password hash" and can be generated with an oneliner such as this:

(echo -n "pi:" ; openssl passwd -6 -stdin <<< 'raspberry') > userconf

Change the name and password as wanted, in the example they are "pi" and "raspberry".

Ubuntu 22.04 and held back packages

I came across a situation in Ubuntu 22.04 where a typical apt upgrade upgraded some packages, but left some packages hanging with the error "The following packages have been kept back". For years, a go-to way to deal with this is apt dist-upgrade which would install the packages that need upgrades, but would also pull in the new dependencies in them.

Unlike with previous Ubuntu versions, this is actually a feature called phased updates. The long story short, only a number of systems receive the updates when released and if they don't cause problems, some days later everyone gets it. You can use apt-cache policy with the package name as a parameter to see this in action, the output should list the newest version as "phased" with percentage on it.

You can force the installation of the held packages with apt -o APT::Get::Always-Include-Phased-Updates=true upgrade, but keep in mind that this is intended feature and that you will get the packages couple of days later, but with an added peace of mind that they didn't didn't cause problems for the test group.

Dealing with stuck iCloud Drive upload

With MacOS 11 and 12 iCloud Drive seems to get stuck at random when the computer sleeps. The symptom is that Finder shows stuck progress pie as iCloud Drive's status and clicking it opens a window that shows 1 or more items stuck at uploading or downloading. A common suggestion that usually helps is rebooting the computer, but it appears that only one process needs restarting to recover. The process' name is "bird".

Locate the process with:

pgrep bird

The command should output the process' PID number. Use the kill command to stop it and it'll restart automatically within couple of seconds. No need for sudo, it runs as your own user.

kill (PID_NUMBER_HERE)