The musl preprocessor debate

Today, I would like to discuss a project that I care very deeply about: the musl libc. One of the most controversial and long-standing debates in the musl community is that musl does not define a preprocessor macro.

What’s in a macro?

Simply put, preprocessor macros allow C code to build parts of itself conditionally. For example, the GNU libc defines the “__GLIBC__” macro. If your code needs to do something specific to function properly on systems using that library, it can conditionally build that code using “#ifdef __GLIBC__”.

The authors of musl have said that they will not add a preprocessor macro identifying the platform as musl because:

It’s a bug to assume a certain implementation has particular properties rather than testing.

Rich Felker, “Re: #define __MUSL__ in features.h”, 2013-03-29

I agree with this sentiment in theory, and in an idealised world this would hold up. However, I’d like to discuss why I think this may need to be reconsidered moving forward.

Sometimes you can’t test

One major reason this is an issue is that sometimes it is not possible to do what the authors consider the “correct” form of testing, which is compile-testing.

This practice requires you to build a small test program, determine whether it built properly, determine its runtime characteristics, and then use the results of that test to influence how your actual software is built. This is an alternative to using the conditional code with preprocessor macros.

However, there are many reasons you may not be able to successfully perform such testing. Cross compilation is a large gap here. In fact, many years ago when I was starting the Adélie project, this caused failures in the base image I was building.

The Bash shell could not perform any compile-time or run-time checks because it was being cross-compiled from a GNU libc system to a musl libc system. This caused it to use “fallback” code that worked improperly. If musl had defined a __MUSL__ macro, Bash would not have needed to assume it was running on a pre-POSIX system.

Similarly, the mailing list thread that made me feel strongly enough to write this article involves a header-only library. These types of libraries are meant to be “drop-in” and function without any changes to a developer’s build system. If header-only libraries start requiring you to use build-time tests, you lose the main reason to use them in the first place.

The author of this thread correctly points out that FreeBSD versions their API with a preprocessor macro. Any software that requires a certain API can simply ensure that __FreeBSD_version is defined as greater-or-equal than the versions that introduced that API.

The main reason that the musl project is fearful of this approach, at least to my observation, is that features or APIs (or indeed, bug fixes) can be backported to prior versions. I feel very strongly that this is not the responsibility of the libc.

If a distribution backports a feature, API, or patch to an older version of a library, it is that distribution’s responsibility to ensure that the software they build against it continues to function. When I backported an API from Qt 5.10 to 5.9 to ensure KDE continued building for Adélie, it was my responsibility as maintainer of those packages to keep them building properly. It certainly does not mean Qt should stop defining a preprocessor macro to determine the version being built against.

Additionally, some APIs are privileged. Determining whether these APIs work correctly using run-time testing can prevent CI/CD from working properly because the CI user does not have permission to use them.

A versioned macro like FreeBSD’s makes sense

I feel that the best way forward for musl is to define a macro like FreeBSD’s. It monotonically increases as APIs or features are added.

I agree that simple bug fixes, and even behavioural changes, probably should not be tracked with this macro. However, this would make it significantly easier to use new APIs as they are introduced.

It also makes builds more efficient. The cost of compile-time tests racks up quickly. On my POWER9 Talos workstation, typical ./configure runs take longer than the builds themselves. This is because fork+exec is still a slow path on POWER. It is similar on ARM, MIPS, and many other RISC architectures.

Macros like these don’t fully eliminate the need for ./configure, but they lessen the workload. Compile-time tests make sense for behaviour detection, but they do not make sense for API detection.

Web browsers, music players, workarounds, and PulseAudio

As security researchers have discovered yet another horrible security bug in Chrome, and Google yet again decides to put off fixing it, I decided to finally give up Chrome entirely. I had dwindled down my usage of it from primary browser in 2009; to secondary browser for Flash and videos in 2013; and finally using it solely for streaming Google Play Music and Spotify, along with the occasional site compatibility test for my work, in 2015. Firefox’s inspector tools and Firebug are good enough, and I have a Mac running Safari if I need a WebKit test, so I decided it was no longer important to test on Chrome. That left the issue of music streaming.

Can’t handle it, can’t handle it

Google Play Music, however, has a fatal flaw. It is a mess of terrible “one page” JavaScript. After only a few hours of music streaming, it had already leaked 150 MB(!) worth of orphan DOM nodes, and 282 MB(!!!!) worth of uncollectable JavaScript objects. This basically means it created buttons, links, and so on, and didn’t properly remove them when it was done, so that memory is leaked out and I would have to restart Firefox to get that memory back. Restarting Firefox multiple times a day is not an option for me.

What’s worse is that one of Firefox’s best and most unknown features was also making my life worse. Every 10 seconds, it scans its memory to see if any of it can be reclaimed, to make sure that it does not use too much memory. Since Google Play Music’s interface had leaked so much memory, the scan was taking about 2 seconds – during which the browser became completely unresponsive. That means that for about 20% of the entire time it was open, it was unresponsive (frozen, locked, etc), all because Google has no idea how to write JavaScript.

My mother (bless her soul, she’s openly embraced Debian) suggested I try Rekonq, but it could not even load Google Play Music’s user interface. I also tried Opera Classic (pre-Blink), and it too could not load Google Play Music. At this point I am very upset at Google; why did you write such a cluster#*$@ of terrible code instead of writing a simple multi-page player like YouTube? YouTube does not suffer from any of these issues, and is a Google product!* Anyway, my next goal was to see what I could do for streaming music that did not require a Web browser.

* I am aware that YouTube has a single-page mode, but I found a way to disable it except while using playlists. It works great and does not leak half a gigabyte of memory.

Done, done, and I’m on to the next one

It turns out that Google Play Music has no official API and no non-browser clients. Even Spotify has unofficial ones that are of questionable quality and legality, but Google has done a very good job of making their API so hard to use that nobody bothers to even try with them. (Future project idea: Make one anyway.)

Then I realised their Android app is pretty reliable and certainly better than having my browser locked for 20% of the time it’s open. However, I still need to be able to hear other things on my computer (if someone links me to a video or presentation, for instance), and I don’t want to have to keep flipping back and forth between my phone and desktop.

My work-provided desktop did not come with a sound card (even though we use sound a lot internally…), so I am using a USB Griffin iMic as my sound “card”. It works fantastically in Linux/ALSA, but one thing I could not figure out was how to make it play line in as a monitor (i.e. playthrough, listening to line in/mic with headphones/out, whatever you like to call it). Thankfully, I found a very helpful blog post about this very issue, and a solution involving PulseAudio: pactl load-module module-loopback was all it took to listen to crystal-clear, low-latency, glorious Nexus audio on my desktop!

Final thoughts

  • While it certainly is great that PulseAudio offers the same great passthrough functionality that OS X had since Jaguar (and lost in Mavericks), they really need to document PulseAudio modules better.
  • Google needs to rethink making their music player in one page JavaScript. A native app would be amazing and make me a much happier catfox.
  • It just feels like… if Google hadn’t royally screwed up Chrome, and they hadn’t royally screwed up their music player, then hours of my life would have been saved because then I would not have had to learn how to monitor line in within Linux. It was interesting learning all this, but I still have this feeling that it should be entirely unnecessary, and like this is a very unclean workaround for what amounts to “Google is terrible at writing code”.

Oh well, at least Android 6.0 is good. (For now.)


Musings: More Python 3 compat, Project Sunrise, InspIRCd modules and Portage

Some good news: as I eix-sync’d this morning, I noticed that dev-python/ndg-httpsclient and dev-python/ipaddress now have Python 3 compatibility. That means two of the packages I had thought had no chance of being upgraded actually have been. As for my own efforts, I have been very busy with work and musl support patches lately, but I have been looking at fixing up the htop package next.

I’ve found Project Sunrise, a way for me to be able to contribute ebuilds to Gentoo in hopes of someday getting them in the master Portage repository. I’m hoping to add a few Python libraries first, then moving up to packaging SuperGameHerm and PyIRC once they’ve matured enough to be useable by external users.

While testing PyIRC, I needed to be able to use a few modules that are not a part of InspIRCd’s main package. Since Portage didn’t allow any way of including them in the installed package, I simply checked out the source code package, ran modulemanager to add the modules, then built only those modules. I copied them to the /usr/lib64/inspircd/modules directory and added them to modules.conf, and voila! Now I can do more IRCv3.2 testing.

Python 2 -> 3 upgrade: status update

More Python projects ported to Python 3.

This is a small update on my bringing packages in Gentoo to Python 3.

I haven’t had time to contribute as much to this effort as I had hoped, but I have successfully finished with two packages and the patches are now in the hands of upstream maintainers. I’ve been toying with the musl C library as an alternative to glibc (and I’ll be posting about my experiences with that later), which has distracted me a bit from Python 3 work.

Required a bit of effort. Debian #789753 filed. Maintainer seems happy enough with it, but it’s not in master yet.
This was simple enough; libevdev has had upstream support for Python 3 since 2013. Gentoo #553110 filed with a patch to update the ebuild accordingly. No response as of post time.