As part of my work to set up infrastructure for a few projects that I hope to launch with some mates in the coming months, I needed to set up a KVM virthost using Gentoo. I decided to write up the process for FOSS Friday! This setup was performed on a Hetzner AMD server running the latest musl stage3, but glibc should be roughly the same.
Hetzner’s AMD offerings are some of the lowest cost dedicated servers with actual support and decent cross connects. All three of these factors are important to the projects that will be using this server.
Gentoo was chosen so that packages could be built with the exact configuration required. There are no extraneous dependencies that can cause vulnerabilities without even being needed or utilised by the actual workload.
The goal is for the host and guest VMs to share the same on-disk kernel. This way, the kernel is only built and updated once. All VMs will automatically boot into the new kernel when the host is rebooted into the new kernel. As such, the guests do not need a /boot or GRUB at all.
Configuring the Host
I decided to have the host and guests share virtually all of their Portage configurations, though I have not set up a centralised Git repository for them to live in just yet. The CPU_FLAGS_X86
are straight from cpuid2cpuflags
. USE is “-X -nls -vala verify-sig
”, a conservative but useful global-USE
for lightweight, hardened infra.
The base hardware additionally needed sys-kernel/linux-firmware
for AMD microcode and TCP offloading. Right now, I’m using package.accept_keywords
to accept the ~amd64
-keyworded version 20240115-r3
. It has a significant performance improvement over 20240115
as I tweaked which firmware files are installed using savedconfig
.
For package.use
, the base settings I find most useful include:
# prefer lighter
app-alternatives/bc -gnu gh
app-alternatives/cpio -gnu libarchive
# trim the fat, what we don’t need on a server
dev-python/pygobject -cairo
net-firewall/ebtables -perl
net-libs/glib-networking -gnome
net-libs/libsoup -brotli
net-misc/netifrc -dhcp
sys-boot/grub -fonts -themes
# eliminate circular dep
dev-libs/libsodium -verify-sig
# would pull CMake into the graph
net-misc/curl -http2
# Required USE for libvirt / virt-install
app-emulation/libvirt lvm
app-emulation/libvirt-glib introspection
net-dns/dnsmasq script
net-libs/gnutls pkcs11 tools
sys-fs/lvm2 lvm
sys-libs/libosinfo introspection
I then did a full world rebuild, followed by emerge -av eix vim sysklogd chrony libvirt virt-install
.
Host-side Networking
I created a bridge interface for the guests to use, which will be a private network segment with no access to the outside world. They will still have access to the host itself, which can run a Portage rsync mirror and binpkg/distfiles host as well.
I did the configuration this way because these VMs will contain sensitive data including login information, and I wanted to be extra-paranoid about network traffic going in to them. It’s probably better to use libvirt’s NAT if possible for your use case.
I added the following stanza to my /etc/conf.d/net
:
bridge_kvmbr0=""
config_kvmbr0="172.16.11.1/24"
This added an empty bridge interface, and set the guest network subnet as 172.16.11.0/24. The host will use .1. To be extra fancy, you could configure a private DNS server to listen on that IP which would allow guests to resolve each other and communicate via hostname.
Host-side Kernel Configuration
I’m using gentoo-kernel
, so there wasn’t any actual Kconfig to be done, but there is the matter of setting up the “hassle-free” automatic update system that I described in the introduction.
What I did was to symlink /boot/vmlinuz-current
and /boot/initramfs-current
to the present version. We can set the guests to boot that, and simply update the symlinks when the kernel itself is updated.
Configuring the Guests
I used a full-disk LVM volume group on the Hetzner server’s second attached disk for guest storage. I created an LV for each guest machine, and then formatted the LV with XFS. Since the VMs don’t need a boot loader there is no reason to have a partition table at all. You can use your file system of choice; I used XFS for performance and consistency.
# lvcreate -n keycloak -L 40G hostvg
Logical volume "keycloak" created
# mkfs.xfs /dev/hostvg/keycloak
[...]
# mount /dev/hostvg/keycloak /opt
# curl [stage 3 tarball] | tar -C /opt -xJf -
[Downloading and extracting the tarball]
# cd /opt
# mount -R /dev dev
# mount -t proc none proc
# mount -t sysfs none sys
# chroot /opt
We are now able to configure the guest environment as desired. Since there is no outbound network access, if you want network time you will need to run a network time server on the host. I personally tend to trust virtio’s RTC system as it rarely loses sync in my experience. With the present frequency of kernel and low-level system updates, it isn’t likely that any of these systems will have long enough uptimes to have tiny amounts of drift matter anyway.
We configure the guest-side networking to use the subnet we defined in the host bridge. For instance, on this VM I could use config_eth0="172.16.11.2/24"
. There is no reason to set routes_eth0 because the host system is not going to route packets out for it.
Setting up the Guest
Now it is time to run virt-install
for the guest and boot it up. Make sure your SSH keys are installed and the chroot is unmounted first!
# virt-install --boot kernel=/boot/vmlinuz-current,initrd=/boot/initramfs-current,cmdline='console=tty0 console=ttyS0 ro root=/dev/vda net.ifnames=0' --disk /dev/hostvg/keycloak -n auth01 -r 8192 --vcpus=2 --cpuset=10-11 --cpu host --import --osinfo gentoo -w bridge=kvmbr0,mac=52:54:00:04:04:03 --graphics none --autostart
Let’s describe some of the fancier of these options. For a full description of the options used here and additional ones you can try, see the refreshingly coherent man page.
--boot kernel=…,initrd=…,cmdline=…
This sets up the guest to boot from the host kernel, as discussed previously.
--import
This tells virt-instal that we have already installed an OS to the disk provided, so it doesn’t need to perform any installation procedures. We’re “importing” an existing drive into libvirt.
-w bridge=kvmbr0,mac=52:54:00:…
This configures networking to use the bridge we set up previously. Note that the MAC for each guest must be unique, and for KVM VMs it must start with 52:54:00.
Enjoy!
This article showed the overview of how I’ve configured a Gentoo machine to serve as a virthost with a dedicated private LAN segment for guests and a way to have those guests share the same kernel as the host. We also looked at a way to “cheat” on storage by using an actual file system as the attached disk.
In the next set of articles, I plan to review:
- Setting up WireGuard on the host to have pain-free access to the private LAN segment from my workstation for administration purposes
- Leveraging the power of Gentoo overlays and profiles to have a consistent configuration for an entire fleet of servers
- Sharing /var/db/repos and /var/cache/distfiles from the host to each guest, so there is only one copy – saving disk space, bandwidth, and time
Until then, happy hacking!