Category: Technology

The Joys of Unix Programming: MAP_ANON(YMOUS)

I was trying to do a little late-night hacking last night on SuperGameHerm, the Game Boy emulator my friends and I are writing, and I hit an error in the memory mapper. Specifically, certain OSes that used to be named after cats don’t like calling mmap on /dev/zero (neither does Android). I thought it was odd that it was falling back to that code in the first place, though, because Apple’s Mac OS X — I mean, a certain cat themed OS — has always supported MAP_ANON, and I confirmed that by going to man mmap on a Mac.

What was going on? I dug deeper and saw MAP_ANON was guarded in sys/mman.h, so CMake wasn’t finding it and it was instead compiling our fallback code. And so I started digging up other issues related to big endian machines and realised that I had only tested OS X and FreeBSD on big endian, and never tested OS X or FreeBSD on little endian. So was my big mistake.

This is a comprehensive guide to How to Make MAP_ANON(YMOUS) Visible, for every OS I could find information on:

Mac OS X

On 10.3 and below, this is easy; it’s always there! It is not guarded by any #ifdef.

On 10.5 and above, it is slighly harder; you must define _DARWIN_C_SOURCE to cause MAP_ANON to be visible in a public scope.

On 10.4 only, it’s much harder! It is only protected by #ifndef _POSIX_C_SOURCE, so to use MAP_ANON against the 10.4 SDK, you must completely undefine _POSIX_C_SOURCE. You don’t have any other choice.

I suppose that means my overall advice then is to use the 10.5 SDK no matter what, if you have a Leopard computer handy, because it can target as low as 10.0. Otherwise, use the Panther SDK included with Tiger’s Xcode Tools. Don’t ever use Tiger’s SDK if you want MAP_ANON.

FreeBSD

Before 5.0, it’s always visible, just as in OS X 10.3. There are no preprocessor options to show or hide MAP_ANON.

On 5.0 or above, the only way to cause MAP_ANON to be visible is to define __BSD_VISIBLE somewhere. Undefining _POSIX_C_SOURCE won’t save you here.

Other BSDs (NetBSD, OpenBSD, DragonFly BSD)

It’s never guarded. MAP_ANON is always available.

Solaris

I could only get my hands on OpenSolaris, but considering the header having a copyright date of 1989 (by AT&T), I can’t imagine it’s any different on Real Solaris (or Oracle Solaris). There are no guards here, either; that’s to be expected since they invented the damn thing.

Linux

glibc: I don’t understand /usr/include/bits in the slightest. It seems to be always available no matter what options I toss to clang, but it is guarded by.. __USE_MISC? I presume this is some sort of feature macro buried deep in glibc that I don’t care about or understand.

musl: It’s always available, at least on 1.1.11 which is what I have on my test box.

Android: After searching through their spaghetti of includes to get to the actual file that defines constants, it appears they are all completely unguarded, though that isn’t surprising since it is Linux and embedded.

In conclusion

Perhaps it’s best to avoid anonymous mmap(2) in applications that you want to actually be portable.

 

Musings: More Python 3 compat, Project Sunrise, InspIRCd modules and Portage

Some good news: as I eix-sync’d this morning, I noticed that dev-python/ndg-httpsclient and dev-python/ipaddress now have Python 3 compatibility. That means two of the packages I had thought had no chance of being upgraded actually have been. As for my own efforts, I have been very busy with work and musl support patches lately, but I have been looking at fixing up the htop package next.

I’ve found Project Sunrise, a way for me to be able to contribute ebuilds to Gentoo in hopes of someday getting them in the master Portage repository. I’m hoping to add a few Python libraries first, then moving up to packaging SuperGameHerm and PyIRC once they’ve matured enough to be useable by external users.

While testing PyIRC, I needed to be able to use a few modules that are not a part of InspIRCd’s main package. Since Portage didn’t allow any way of including them in the installed package, I simply checked out the source code package, ran modulemanager to add the modules, then built only those modules. I copied them to the /usr/lib64/inspircd/modules directory and added them to modules.conf, and voila! Now I can do more IRCv3.2 testing.

Foxtoo: Gentoo + musl C library on 100MHz Pentium laptop

The beginnings of Adélie Linux.

I haven’t yet finished up writing all the content I want to write about the process needed to get this working, but I do have a little teaser picture to share:

That is a Compaq LTE 5150 with a 100MHz Pentium CPU and 40 MB EDO RAM running Gentoo Linux! The kernel version is 4.2-rc1 (because I’m an incorrigible ricer), it was all built with GCC 4.9.3 and it is using the venerable musl C library instead of glibc. Boot up takes only about 15 seconds off the 5400 RPM laptop IDE disk, and once booted, the minimal kernel I have + bash use only about 3 MB of the 40 MB total.

It may not seem like a very useful thing to have done, but I had a lot of fun building it up, and I’ve ended up finding and closing various bugs in everything from procps to the kernel itself. So I feel that not only has this been a fun personal project across two weekends, but it has been productive for the entire community 🙂

Python 2 -> 3 upgrade: status update

More Python projects ported to Python 3.

This is a small update on my bringing packages in Gentoo to Python 3.

I haven’t had time to contribute as much to this effort as I had hoped, but I have successfully finished with two packages and the patches are now in the hands of upstream maintainers. I’ve been toying with the musl C library as an alternative to glibc (and I’ll be posting about my experiences with that later), which has distracted me a bit from Python 3 work.

app-misc/ca-certificates
Required a bit of effort. Debian #789753 filed. Maintainer seems happy enough with it, but it’s not in master yet.
dev-libs/evdev
This was simple enough; libevdev has had upstream support for Python 3 since 2013. Gentoo #553110 filed with a patch to update the ebuild accordingly. No response as of post time.