Category: Journal

Experiences with building a Gentoo virtualisation host

As part of my work to set up infrastructure for a few projects that I hope to launch with some mates in the coming months, I needed to set up a KVM virthost using Gentoo. I decided to write up the process for FOSS Friday! This setup was performed on a Hetzner AMD server running the latest musl stage3, but glibc should be roughly the same.

Hetzner’s AMD offerings are some of the lowest cost dedicated servers with actual support and decent cross connects. All three of these factors are important to the projects that will be using this server.

Gentoo was chosen so that packages could be built with the exact configuration required. There are no extraneous dependencies that can cause vulnerabilities without even being needed or utilised by the actual workload.

The goal is for the host and guest VMs to share the same on-disk kernel. This way, the kernel is only built and updated once. All VMs will automatically boot into the new kernel when the host is rebooted into the new kernel. As such, the guests do not need a /boot or GRUB at all.

Configuring the Host

I decided to have the host and guests share virtually all of their Portage configurations, though I have not set up a centralised Git repository for them to live in just yet. The CPU_FLAGS_X86 are straight from cpuid2cpuflags. USE is “-X -nls -vala verify-sig”, a conservative but useful global-USE for lightweight, hardened infra.

The base hardware additionally needed sys-kernel/linux-firmware for AMD microcode and TCP offloading. Right now, I’m using package.accept_keywords to accept the ~amd64-keyworded version 20240115-r3. It has a significant performance improvement over 20240115 as I tweaked which firmware files are installed using savedconfig.

For package.use, the base settings I find most useful include:

# prefer lighter
app-alternatives/bc -gnu gh
app-alternatives/cpio -gnu libarchive

# trim the fat, what we don’t need on a server
dev-python/pygobject -cairo
net-firewall/ebtables -perl
net-libs/glib-networking -gnome
net-libs/libsoup -brotli
net-misc/netifrc -dhcp
sys-boot/grub -fonts -themes

# eliminate circular dep
dev-libs/libsodium -verify-sig

# would pull CMake into the graph
net-misc/curl -http2

# Required USE for libvirt / virt-install
app-emulation/libvirt lvm
app-emulation/libvirt-glib introspection
net-dns/dnsmasq script
net-libs/gnutls pkcs11 tools
sys-fs/lvm2 lvm
sys-libs/libosinfo introspection

I then did a full world rebuild, followed by emerge -av eix vim sysklogd chrony libvirt virt-install.

Host-side Networking

I created a bridge interface for the guests to use, which will be a private network segment with no access to the outside world. They will still have access to the host itself, which can run a Portage rsync mirror and binpkg/distfiles host as well.

I did the configuration this way because these VMs will contain sensitive data including login information, and I wanted to be extra-paranoid about network traffic going in to them. It’s probably better to use libvirt’s NAT if possible for your use case.

I added the following stanza to my /etc/conf.d/net:

bridge_kvmbr0=""
config_kvmbr0="172.16.11.1/24"

This added an empty bridge interface, and set the guest network subnet as 172.16.11.0/24. The host will use .1. To be extra fancy, you could configure a private DNS server to listen on that IP which would allow guests to resolve each other and communicate via hostname.

Host-side Kernel Configuration

I’m using gentoo-kernel, so there wasn’t any actual Kconfig to be done, but there is the matter of setting up the “hassle-free” automatic update system that I described in the introduction.

What I did was to symlink /boot/vmlinuz-current and /boot/initramfs-current to the present version. We can set the guests to boot that, and simply update the symlinks when the kernel itself is updated.

Configuring the Guests

I used a full-disk LVM volume group on the Hetzner server’s second attached disk for guest storage. I created an LV for each guest machine, and then formatted the LV with XFS. Since the VMs don’t need a boot loader there is no reason to have a partition table at all. You can use your file system of choice; I used XFS for performance and consistency.

# lvcreate -n keycloak -L 40G hostvg
  Logical volume "keycloak" created
# mkfs.xfs /dev/hostvg/keycloak
[...]
# mount /dev/hostvg/keycloak /opt
# curl [stage 3 tarball] | tar -C /opt -xJf -
[Downloading and extracting the tarball]
# cd /opt
# mount -R /dev dev
# mount -t proc none proc
# mount -t sysfs none sys
# chroot /opt

We are now able to configure the guest environment as desired. Since there is no outbound network access, if you want network time you will need to run a network time server on the host. I personally tend to trust virtio’s RTC system as it rarely loses sync in my experience. With the present frequency of kernel and low-level system updates, it isn’t likely that any of these systems will have long enough uptimes to have tiny amounts of drift matter anyway.

We configure the guest-side networking to use the subnet we defined in the host bridge. For instance, on this VM I could use config_eth0="172.16.11.2/24". There is no reason to set routes_eth0 because the host system is not going to route packets out for it.

Setting up the Guest

Now it is time to run virt-install for the guest and boot it up. Make sure your SSH keys are installed and the chroot is unmounted first!

# virt-install --boot kernel=/boot/vmlinuz-current,initrd=/boot/initramfs-current,cmdline='console=tty0 console=ttyS0 ro root=/dev/vda net.ifnames=0' --disk /dev/hostvg/keycloak -n auth01 -r 8192 --vcpus=2 --cpuset=10-11 --cpu host --import --osinfo gentoo -w bridge=kvmbr0,mac=52:54:00:04:04:03 --graphics none --autostart

Let’s describe some of the fancier of these options. For a full description of the options used here and additional ones you can try, see the refreshingly coherent man page.

--boot kernel=…,initrd=…,cmdline=…
This sets up the guest to boot from the host kernel, as discussed previously.

--import
This tells virt-instal that we have already installed an OS to the disk provided, so it doesn’t need to perform any installation procedures. We’re “importing” an existing drive into libvirt.

-w bridge=kvmbr0,mac=52:54:00:…
This configures networking to use the bridge we set up previously. Note that the MAC for each guest must be unique, and for KVM VMs it must start with 52:54:00.

Enjoy!

This article showed the overview of how I’ve configured a Gentoo machine to serve as a virthost with a dedicated private LAN segment for guests and a way to have those guests share the same kernel as the host. We also looked at a way to “cheat” on storage by using an actual file system as the attached disk.

In the next set of articles, I plan to review:

  • Setting up WireGuard on the host to have pain-free access to the private LAN segment from my workstation for administration purposes
  • Leveraging the power of Gentoo overlays and profiles to have a consistent configuration for an entire fleet of servers
  • Sharing /var/db/repos and /var/cache/distfiles from the host to each guest, so there is only one copy – saving disk space, bandwidth, and time

Until then, happy hacking!

Dual-booting Windows 11 and Windows 7 on a Haswell

I have a Haskell-era MSI B85-G41 PC Mate motherboard and I decided to use it as a “mid-tier”-ish gaming PC and also as a TV set top box. I already had a WinTV-DCR-2650 dual-tuner CableCARD USB device, and I was gifted a Nvidia GeForce RTX 3070 for the project. The board had 32 GB RAM when I decommissioned it in 2019 as the Adélie x86_64 builder, so memory was not a concern.

My goal is to use Windows 11 for gaming, and Windows 7 Media Centre for the TV support (since Cox Oklahoma uses encryption for virtually all channels).

The problem is that Microsoft dropped support for Windows 7 long before this hardware existed, so it is difficult to boot on it. Also, Windows 11 doesn’t officially support Haswell, either.

Windows 11 was trivial to install in all honesty. I used Rufus to put the installer for Windows 11 on a USB disk, then followed the suggestions from this article in Tom’s Hardware and it installed quite nicely. It is performant, stable, and even still does Windows Update.

Windows 7 was significantly more difficult. I used Rufus again and ensured it used GPT and UEFI. It locked up early in boot. I found the UEFISeven project which seemed to make things somewhat better, but it never finished booting beyond “Starting Windows”. The Windows logo continued to pulse, but after 15 minutes I gave up. I found an issue on the UEFISeven tracker and despite my trepidation on running unknown binaries for booting, putting it in the USB stick managed to boot Windows 7’s installation environment successfully.

Next, while performing the installation, the system had a STOP 0x7E in HIDCLASS.SYS. This appears to be a very classic bug and it’s caused by using a Microsoft Wireless Keyboard/Mouse. (Irony as a MS hardware product crashes MS Windows…) Replacing them with (even more ironically) an Apple Pro Keyboard and Mouse allowed setup to continue.

The next problem was actually dual-booting. If I use the patched Windows 7 boot EFI application as BOOTMGFW.EFI, Windows 11 doesn’t boot; it seems to load all the files, but stays at a black screen. If I use Windows 11’s BOOTMGFW.EFI, Windows 7 no longer boots.

I’ve made a small batch script on the desktop of each one to reboot to the other. The 7->11 script renames BOOTMGFW.EFI to BOOTMGFW.7, then renames BOOTMGFW.11 to BOOTMGFW.EFI. The inverse is done for the 11->7 script. Note that you have to mount the ESP first, which is done (in both OSes) as “MOUNTVOL S: /S”. You can use any available drive letter.

I used LegacyUpdate.net to fetch and install all the needed updates for Windows 7. I still wouldn’t trust it unprotected on the “real” internet, but I’m comfortable enough with it sitting on my home network this way. Kudos to that team for making such a useful and valuable service for all retrocomputing enthusiasts!

Notes about the iBook G3 Clamshell

I’ve just repaired the hinge on my Indigo Clamshell. While I was in there, I also replaced the aging hard disk with a SD card adaptor. I wanted to write down a few notes about the process, both for posterity and so that others can benefit from my experience.

The standoffs for the hard disk caddy are brittle. I slightly over-tightened one and it snapped right off. Luckily, it snapped in a way that it would still stand solidly and hold the grounding wire of the charging board. When the Service Source manual says do not overtighten, it means it – as soon as there is the slightest resistance, stop: it’s tight.

I burned a copy of the iBook Software Restore CD from the fabulous archivists at the Garden, so that I could put the original software back on the SD card since it was empty. I used Verbatim CD-R 52x media and burned with an LG SP80NB80 on my Mac Studio.

The disc was readable by the iBook’s optical drive, but only barely; it took five minutes to show the Desktop. I’m not sure if it was the speed at which it was burned, the Verbatim media simply not agreeing with the iBook, or something about the power of the laser in the LG.

Errors from attempting to use a burned iBook Software Restore CD.

I regularly received “Some applications could not be quit.” when attempting to use Erase, and received “Restoring the software configuration iBook HD.img to volume Macintosh HD failed.” when attempting to use Restore.

I used my Power Mac G5 to read the CD and copy it to a USB key. Specifically, I used:

sudo dd if=/dev/disk3s1 of=/dev/disk2 bs=1048576

A mere 15 minutes later, I had a functional USB version of the iBook Software Restore. I then used a copy of Puma (Mac OS X 10.1.4) to install on the same partition, allowing me to dual-boot the system in both 9 and X. I have a second partition I plan to use to install Jaguar or Panther. I haven’t decided which one yet.

I’ll close with a photo of the iBook being a happy Puma. Until next time, be well!

My Indigo iBook G3 Clamshell, showing the introduction video from Mac OS X “Puma” 10.1.
Happy as a clam(shell)! 😁

An introduction to features

I would like to have themed “features” here on my blog, both as a sort of recurring writing prompt and so that I have more of a focus on the subject matter I should cover.

With that in mind, I won’t be able to write each feature each week. It’ll be a surprise! I do hope to cover more ground, and disseminate both my knowledge and my opinions better, with this new organisation.

Mac Monday

On Mondays, I’ll write about the Macintosh. This can include fun I’m having with the older Macs in my retro computer collection, productivity tips I’ve discovered on my modern Macs, and references and guides for how to accomplish tasks with a Mac.

FOSS Friday

On Fridays, I’ll write about libre software (FOSS). This includes news and opinions on what is going on in the greater Linux and BSD worlds, reviews of libre software that I use (for better or worse), and projects I am working on.

I really do want to focus more on reviewing libre software, as I feel highlighting the positives and negatives of the software we have can spur better development in future projects.

Caturday

On Saturdays, I’ll write about my cats! More than likely, most posts will be a single photo and a caption, but they have enough fun and cause enough mischief that I expect there will be a few long-form articles as well.

Thoughts

Other articles will be my thoughts on a specific subject, which is how this blog originally started. These will not be on any prescribed schedule, and will be infrequent.

Here’s to the future!