The musl preprocessor debate

Today, I would like to discuss a project that I care very deeply about: the musl libc. One of the most controversial and long-standing debates in the musl community is that musl does not define a preprocessor macro.

What’s in a macro?

Simply put, preprocessor macros allow C code to build parts of itself conditionally. For example, the GNU libc defines the “__GLIBC__” macro. If your code needs to do something specific to function properly on systems using that library, it can conditionally build that code using “#ifdef __GLIBC__”.

The authors of musl have said that they will not add a preprocessor macro identifying the platform as musl because:

It’s a bug to assume a certain implementation has particular properties rather than testing.

Rich Felker, “Re: #define __MUSL__ in features.h”, 2013-03-29

I agree with this sentiment in theory, and in an idealised world this would hold up. However, I’d like to discuss why I think this may need to be reconsidered moving forward.

Sometimes you can’t test

One major reason this is an issue is that sometimes it is not possible to do what the authors consider the “correct” form of testing, which is compile-testing.

This practice requires you to build a small test program, determine whether it built properly, determine its runtime characteristics, and then use the results of that test to influence how your actual software is built. This is an alternative to using the conditional code with preprocessor macros.

However, there are many reasons you may not be able to successfully perform such testing. Cross compilation is a large gap here. In fact, many years ago when I was starting the Adélie project, this caused failures in the base image I was building.

The Bash shell could not perform any compile-time or run-time checks because it was being cross-compiled from a GNU libc system to a musl libc system. This caused it to use “fallback” code that worked improperly. If musl had defined a __MUSL__ macro, Bash would not have needed to assume it was running on a pre-POSIX system.

Similarly, the mailing list thread that made me feel strongly enough to write this article involves a header-only library. These types of libraries are meant to be “drop-in” and function without any changes to a developer’s build system. If header-only libraries start requiring you to use build-time tests, you lose the main reason to use them in the first place.

The author of this thread correctly points out that FreeBSD versions their API with a preprocessor macro. Any software that requires a certain API can simply ensure that __FreeBSD_version is defined as greater-or-equal than the versions that introduced that API.

The main reason that the musl project is fearful of this approach, at least to my observation, is that features or APIs (or indeed, bug fixes) can be backported to prior versions. I feel very strongly that this is not the responsibility of the libc.

If a distribution backports a feature, API, or patch to an older version of a library, it is that distribution’s responsibility to ensure that the software they build against it continues to function. When I backported an API from Qt 5.10 to 5.9 to ensure KDE continued building for Adélie, it was my responsibility as maintainer of those packages to keep them building properly. It certainly does not mean Qt should stop defining a preprocessor macro to determine the version being built against.

Additionally, some APIs are privileged. Determining whether these APIs work correctly using run-time testing can prevent CI/CD from working properly because the CI user does not have permission to use them.

A versioned macro like FreeBSD’s makes sense

I feel that the best way forward for musl is to define a macro like FreeBSD’s. It monotonically increases as APIs or features are added.

I agree that simple bug fixes, and even behavioural changes, probably should not be tracked with this macro. However, this would make it significantly easier to use new APIs as they are introduced.

It also makes builds more efficient. The cost of compile-time tests racks up quickly. On my POWER9 Talos workstation, typical ./configure runs take longer than the builds themselves. This is because fork+exec is still a slow path on POWER. It is similar on ARM, MIPS, and many other RISC architectures.

Macros like these don’t fully eliminate the need for ./configure, but they lessen the workload. Compile-time tests make sense for behaviour detection, but they do not make sense for API detection.

Clearing confusion regarding modern PowerPC endianness

I am having to correct, with alarming regularity, confusion regarding the endianness of modern PowerPC and POWER chips.  This article is going to answer a lot of those questions, with facts and citations.

What endianness are modern PowerPC / POWER CPUs, including POWER9?
Fact: All POWER Architecture processors since POWER3 support both big and little endian modes. This is because the PowerPC ISA defines an endian-switch bit in a processor control register (MSR), and all POWER processors since POWER3 implement the PowerPC ISA. The PowerPC ISA dates back to the 1990s, where AIX and Linux were exclusively big endian and Windows NT (yes, Windows NT) ran on PowerPC in little endian mode. Most POWER hardware, and most PowerPC computers, historically had firmware that only supported big endian mode. Reports are that POWER4 and POWER5 chips do not support setting the MSR because no firmware supports this mode, but I have no citation to confirm nor deny this. (My IBM POWER hardware starts at POWER6.) This has changed with POWER8, and now modern computers support both. POWER8 and POWER9 can run in either endian, though they still default to big endian during initial bootup (and the firmware services are still in big endian, requiring a byteswap for little endian OSes).
Isn’t Linux only being developed for PowerPC on little endian now?
Fact: The Linux kernel supports both endians equally.
Didn’t Debian drop support for big endian PowerPC with Jessie?
Fact: Debian still “actively supports” big endian 64-bit PowerPC; it is not a release architecture because it does not have enough dedicated maintainers. The port is still fully functional and is kept up to date.
When you buy a new POWER computer, aren’t your only choices of operating system little endian?
Fact: In addition to Debian’s big endian port, there are plenty of other operating systems that support big endian. Gentoo’s PPC64 profile is bi-endian in nature. FreeBSD and Adélie Linux are exclusively big endian, and support all the modern features of POWER9 including DARN, Radix MMU, and more. Devuan is currently adding PPC64 support for both endians.
Isn’t IBM (or OpenPOWER, or [another member organisation of OpenPOWER]) investing solely in little endian for the future?
Fact: OpenPOWER is dedicated to supporting development of both BE and LE.

Aren’t you stuck with one endian or the other?
Fact: Linux’s KVM hypervisor lets you run an environment with the opposite endianness of your host. You can freely run either endian on your host and still have the software of the other endianness available to you with no issues.

The Joys of Unix Programming: MAP_ANON(YMOUS)

I was trying to do a little late-night hacking last night on SuperGameHerm, the Game Boy emulator my friends and I are writing, and I hit an error in the memory mapper. Specifically, certain OSes that used to be named after cats don’t like calling mmap on /dev/zero (neither does Android). I thought it was odd that it was falling back to that code in the first place, though, because Apple’s Mac OS X — I mean, a certain cat themed OS — has always supported MAP_ANON, and I confirmed that by going to man mmap on a Mac.

What was going on? I dug deeper and saw MAP_ANON was guarded in sys/mman.h, so CMake wasn’t finding it and it was instead compiling our fallback code. And so I started digging up other issues related to big endian machines and realised that I had only tested OS X and FreeBSD on big endian, and never tested OS X or FreeBSD on little endian. So was my big mistake.

This is a comprehensive guide to How to Make MAP_ANON(YMOUS) Visible, for every OS I could find information on:

Mac OS X

On 10.3 and below, this is easy; it’s always there! It is not guarded by any #ifdef.

On 10.5 and above, it is slighly harder; you must define _DARWIN_C_SOURCE to cause MAP_ANON to be visible in a public scope.

On 10.4 only, it’s much harder! It is only protected by #ifndef _POSIX_C_SOURCE, so to use MAP_ANON against the 10.4 SDK, you must completely undefine _POSIX_C_SOURCE. You don’t have any other choice.

I suppose that means my overall advice then is to use the 10.5 SDK no matter what, if you have a Leopard computer handy, because it can target as low as 10.0. Otherwise, use the Panther SDK included with Tiger’s Xcode Tools. Don’t ever use Tiger’s SDK if you want MAP_ANON.

FreeBSD

Before 5.0, it’s always visible, just as in OS X 10.3. There are no preprocessor options to show or hide MAP_ANON.

On 5.0 or above, the only way to cause MAP_ANON to be visible is to define __BSD_VISIBLE somewhere. Undefining _POSIX_C_SOURCE won’t save you here.

Other BSDs (NetBSD, OpenBSD, DragonFly BSD)

It’s never guarded. MAP_ANON is always available.

Solaris

I could only get my hands on OpenSolaris, but considering the header having a copyright date of 1989 (by AT&T), I can’t imagine it’s any different on Real Solaris (or Oracle Solaris). There are no guards here, either; that’s to be expected since they invented the damn thing.

Linux

glibc: I don’t understand /usr/include/bits in the slightest. It seems to be always available no matter what options I toss to clang, but it is guarded by.. __USE_MISC? I presume this is some sort of feature macro buried deep in glibc that I don’t care about or understand.

musl: It’s always available, at least on 1.1.11 which is what I have on my test box.

Android: After searching through their spaghetti of includes to get to the actual file that defines constants, it appears they are all completely unguarded, though that isn’t surprising since it is Linux and embedded.

In conclusion

Perhaps it’s best to avoid anonymous mmap(2) in applications that you want to actually be portable.

 

FreeBSD on Apple MacBook Pro 8,2: Epilogue.

Why I left FreeBSD.

It is with a fairly heavy heart that I write I am no longer running FreeBSD on my MacBook Pro.

What happened to improving?

Part of the problem is that I finally received gainful employment in March, and that work is almost impossible to do on FreeBSD. A lot of it involves Chrome (which I still have been unable to run on FreeBSD), Qt5-based applications (which crash due to known bugs in libv8 that Google do not care to resolve), and some Python libraries that have truly terrible performance on FreeBSD.

Why not run Linux in a VM for work?

Sure, I could have, if VirtualBox ever worked…

Weren’t you excited to fix up FreeBSD?

I was. I still am, but something just feels different. For over a decade, FreeBSD has for me been the go-to operating system for any use case: servers, embedded projects, desktop systems, and more. But the current heading of development seems to strongly suggest this is no longer encouraged or desired.

When I first started out with Gentoo nine years ago, they were pretty much bent on making it for newer hardware only. Back then, Pentium computers were like the Pentium 4s of now – something you give your grandma or little sister for web browsing, but nothing too serious. And Gentoo developers did not really care if they broke compatibility with these older systems. I can understand that, given that compiling the entire system by hand is something that is pretty taxing for older hardware.

The nice thing about FreeBSD was their community never looked down on you for using these older machines, and realised they still have use. My first interactions with #FreeBSDHelp on EFnet were in 2006 and related to getting SLIP support working in sysinstall so I could remotely install FreeBSD 6 on my Pentium 90 laptop. They were happy to help.

The roles have largely reversed now. Running into issues with older hardware get me looks of disdain and “great, upgrade your hardware and try again” from the FreeBSD community. Meanwhile, the Gentoo team was happy to help me with an issue regarding my retro Intel486 box, in 2015. This computer has no business still functioning, and they were still willing to help me configure a kernel that would boot on it with its anaemic 20 MB RAM.

The other thing I have noticed is that even now, months later, none of my Ports bugs have been handled. In the same amount of time, I have filed three bugs against Portage packages… and all of them were closed within one week of being opened. I feel like my contributions matter to the Gentoo Linux team.

What have you learned?

FreeBSD is more fun to hack on than Gentoo. FreeBSD is harder to get things done on than Gentoo.

FreeBSD is lighter on resources than Gentoo. FreeBSD is heavier on bug backlog than Gentoo.