…and the main Adélie Linux Web serving box went down, in a strange way.
Network access to all the KVM VMs running on our primary dedicated server suddenly dropped, and new connections were refused. Connecting to the host via BMC, I was greeted with possibly the oddest machine identification I’ve ever seen:
Adélie Linux 4.14.76-mc11-easy-p8/ppc64 on chloe
Sat Aug 6 3160 10:08:02 (hvc0; 0 users)
Upon login, it became even weirder:
chloe ~ # uptime
10:07:54 up 24855 days, 3:14, 1 user, load averages: 1.10, 0.25, 0.08
Note how time went backwards! It actually seemed to be stuck in a loop going from about 10:06 AM to 10:11 AM UTC on the 6th of August, 3160 AD. (We have a rack-mounted time machine?) It turns out that there is an issue with the 4.14 branch running KVM on 64-bit PowerPC. Upgrading the box to kernel 4.19.3 appears to have solved the issue.