The Retro Lab: Introduction

Welcome to my latest series of articles, The Retro Lab, where I will be detailing my excursions into the art and hobby of retrocomputing.

This article will serve as a general overview of what I hope to accomplish, and a bit about my background and why this will be fun for me 🙂 If you just want to see the meat of this, skip to the section “And now, in the present”.

My history with computers

When I was very young, my family had a 386 running DOS and Windows 3.1. The only thing I cared about were the games, of course. I don’t remember a lot from this era, because I was so young.

When we were with my grandpa, I loved to play on his XT clone. It was a Leading Edge Model D with 20MB disk running DOS. I inherited this computer when he passed and it is still in my closet. I treasure it. Some of the games it had were a text adventure game called “CIA” and Wheel of Fortune. The real magic for me, though, was in the BASIC interpreter. It was amazing to type in some commands and see this big huge loud complex machine do what I tell it! This is what hooked me on programming and is why I chose the field I did.

The next computer we had was a Canon StarWriter Pro 5000. I’m not sure what hardware it has – and it’s also in my closet, so a careful tear down may be a future article! – but I know it ran GEOS. I loved to write little comedy skits and song lyrics as a child, and it was cool to have all of them on a single 3.5″ floppy disk instead of taking all the paper in the house.

The real life changing moment, however, came on February 22, 1997. It was the day we brought home The Pentium. What a beast of a computer: 133 MHz, 24 MB RAM, and an 8 speed CD drive! It was a Compaq Presario 4712, and it came with Encarta 97, Compton’s Interactive Encyclopaedia, The Yukon Trail, Magic Carpet, PGA Tour ’96, but most of all: 15 free hours of America Online.

AOL was amazing to a second grader. They had Nickelodeon online! You could download little sound clips from Nick and Nick at Nite shows. They had games like Slingo (still one of my all-time favourite takes on slot machines, 20+ years later). And they had a graphical portal to Gopher/WAIS. The local school district had uploaded text files full of fun activities for us children on their Gopher server.

That computer was also where I first used Telnet to a computer running Solaris. A few friends and I used talk on it to have our own small chat rooms. My aunt ran an IRC channel and we talked on mIRC. We ended up with a webcam and talked with family using NetMeeting. Yahoo Messenger, ICQ, Infoseek… so many things.

Programming and desires

Enough with the ‘net reminiscing, at least for now 🙂

Something else important to mention is that my grandpa also was an Important Person at a facility, and one of the things he did was computer purchasing. He had catalogues from Compaq, IBM, and various other vendors in his house for that reason. I loved flipping through them and looking at all the cool stuff.

Something I always wanted back then was my own server. It seemed so cool. I was especially attracted to the ProLiants and AlphaServers in the Compaq catalogue. Windows NT and Tru64 looked so cool when I looked them up online.

The other thing that was very attractive back then were the Power Macintosh computers. My Mum was a digital artist back then. The Quadra was a nice system but didn’t compare to what I saw the Power Macs could do!

For my birthday in 1998, I received Visual Basic as a gift. It really cemented my desire to be a programmer. This was such an exciting time and part of the VB6 software was a one year subscription to the MSDN Library. From that library I learned about all the different servers one could run, all the different types of NT, the different programming languages of Visual Studio…

And now, in the present

In the past few years, as I am able and as opportunities arise, I have amassed quite a collection of hardware that I want to set up and enjoy:

A Compaq Armada e500 laptop. This is a Pentium III from the year 2000 and is likely the newest system I want to have in my Retro Lab. It runs Adélie right now; I’ll likely remove the hard drive and install another to run period-accurate software. I will likely run Windows NT 4 or 2000.

A beige Power Macintosh G3 with the Bordeaux personality card. I will be inserting a 10 GB disk and installing Mac OS 8.6 on it. This will run all the classic Mac software that I have collected over the years. It will be one of the main focuses of the Lab.

A few AlphaServers. Most are earmarked for Adélie so I can’t really use them in the Lab, but there is a single DS10L that was set aside for my personal use. I’ll likely install NT 4 on this one, but I need to investigate further on the hardware.

A Sun Ultra 60, Netra T1, and Ultra 10. These are all from ’98-’99 and will make great Solaris systems, to relive the glory days and experiment more with what was my first real Unix. I would like to run CDE again and possibly do some Java tinkering with these. It would be very fun to run a Retro Lab network off of the AlphaServer and Netra.

A Power Macintosh 7100/80AV. More fun Mac stuff awaits on this computer, though I’m not sure exactly what I will do with it yet.

A Compaq LTE 5150. This is actually my original laptop from high school, ca. 2004. I’d like restore it to its former glory and use it for Windows 3.1 and early 95 software. It can also run OS/2. The screen probably won’t do well for most games, but I do have the docking bay to connect it to an external monitor…

An SGI Indy. This will need an SCSI2SD adaptor to reach its full potential since the hard disk died many years ago. I would love to dual boot IRIX and a BSD.

A Dell System 316LT. There are a few older DOS games I have that would run much better under a CPU of this speed. It needs some love; I seem to recall it had an issue booting up the last time I had it out. I could also try and run GEOS.

A Compaq Presario 4850. This is, to my knowledge, the oldest original computer I have that still fully functions. We purchased it on my Mum’s birthday, 1998, for her graphic design software. This, along with the Beige G3, will likely be the centrepiece of my Lab. I plan on running either Windows 95 or 98 on it, and also various other OSes of the era: BeOS, OpenStep, maybe early Linux. I know that the Rage Pro functions in high res in Win3.1 and OS/2 from prior hackings. It’s also the first computer I used to tinkered with XFree86 modelines. It has a factory original Hitachi DVD drive.

Don’t forget the accessories!

Oh yes, I have some great period hardware for the tinkering as well:

HP ScanJet 5s SCSI scanner. Drivers for Windows, Macintosh, and IRIX, at least. I believe there is an attachment to scan photo negatives as well, but I can’t remember now.

Aiptek webcam. Yes, the original one from the NetMeetings of old that I talked about in my history section. Should be very easy to bring up under Windows. I am curious about Macintosh support.

HP CD Writer Plus 7200e. This is a parallel port, dual-speed CD writer and rewriter. One of the cool features I found on this back in the day is that if you send multimedia commands and have speakers connected to the external headphone jack, you can power off the computer and still listen to the CD until it finishes! I found this out one day when Win95 crashed while I was listening to Garbage’s Version 2.0.

My MSDN Universal archive. In 2002 I found an MSDN Universal subscription at a flea market for 15 USD. I activated it and have all the CDs, and also sent a special request for them to send me the Archive CDs which included BackOffice 4.5 and a few other goodies.

Unfortunately my back and neck are not up to carrying a CRT. I have a flat panel from 2006, a 17″ ViewSonic, that seems to be very close in specification to what we could have had in 1998 for way too much money 😉 Hey, with everything else being so accurate, a little cheating on the monitor isn’t so bad!

In conclusion

This was a lot longer than I had originally anticipated, but it also covers a lot of ground. Over the coming weeks, I hope to bring up a few of these computers and document the processes. Until then, happy hacking!

Libre software and moral absolutism

I’ve been pondering what to write in my blog now that I no longer lead Adélie Linux. There isn’t a great amount of things to write about around my day job making libre networking software, and there’s even less to write about my “spare time” projects. (Mostly because that spare time is spent playing with my cat, my Mum, or my video games.)

I had originally planned for this blog to cover travel and photography around Oklahoma in addition to tech. For obvious reasons, this isn’t something I’m able to do at this time. There are only so many ways I can photograph the gardens around my flat, and inter-city (let alone inter-state) travel is not exactly possible in 2020.

There are actually a lot of tech subjects I would love to cover, but a lot of them revolve around non-libre software. After spending so many years in the communities I have, there seemed a very real sense of shame in the thought of writing about it. However, the more I’ve thought about it, the more I realise it is not shame I feel.

It is embarrassment. It is a sense of letting my friends and colleagues down by feeling anything other than contempt for running proprietary software.

But I do. I absolutely enjoy using my iPad Air 2. And I have dozens – maybe hundreds – of articles in me about retro computing with classic versions of Windows, Mac OS, and Solaris. Game consoles are another fun hobby of mine that I want to share more widely.

So this is the dilemma I face. To continue to write nothing but articles about Linux would be to hide a part of me that is real. Is that what I want from my life? I don’t think it is.

This is not a rebuke of libre software, nor is this some admission there is nothing left to write about it. The future will always be bright with libre software, and I have plenty of articles left in me with regards to libre software, I’m sure. But I think it’s about time for me to admit to myself, and the world, that I have other technological passions as well. And it’s time to stop being embarrassed about it.

Leaving the Linux distribution community

It is with a heavy heart that I am publicly announcing my immediate retirement from the Linux distribution community. This is not a decision I have arrived at lightly.

This year has been challenging for every living being on planet Earth. For me personally, 2020 has given me a lion’s share of financial challenges. I am financially hurting in ways that I could have never imagined.

Adélie Linux and its community has always been a passion of mine, and that passion is still intact. However, it is not responsible for me to continue to act as Project Lead when I need to focus on things that will make me enough money to survive.

Without the ability to be paid to work on Adélie Linux, there is not enough time in my day to devote to it and give it the attention and care that it so rightfully deserves. It is not fair to the community to have to wait weeks (or longer) for me to review merge requests, fix bugs, respond to issues, and so on.

My heart and soul will always belong to Adélie and the wonderful community we have built together over the past six years.

I will stay around long enough to properly transfer my Lead role to someone else, who can carry Adélie to great things in the future. However, I will not be contributing much in the way of patches or code to upstreams like musl or KDE going forward.

It is my sincere hope that when I am in a better financial situation, I can resume leadership of Adélie Linux, if the community should so desire. I cannot see the future and do not know when, or if, that could happen.

Until then, I wish all of you the very best and look forward to watching Adélie Linux continue to grow, from a distance.

With much respect,


Live from Adélie: Streaming Spotify on musl

Over the July 4th holiday weekend, I was working on a secret project. It was a resounding success and I can now announce to the world: Spotify runs on musl distributions!

This article will describe how I went about accomplishing this feat. If you just want to take Spotify for a test drive on your Adélie workstation or Void desktop, scroll to the “Instructions” heading.


Thanks to these fine dwellers of IRC for helping make sense of the twisty mazes.

  • [[sroracle]]
  • Aerdan
  • cb
  • dalias
  • skarnet

gcompat 0.4.0: how very cash LC_MONETARY of you

The latest release version of gcompat did not get very far:

awilcox on laptop spotify % ./spotify
Segmentation fault (core dumped)

Inspecting the core file was minimally helpful:

Thread 1 "ld-musl-x86_64." received signal SIGSEGV, Segmentation fault.
0x0000000001d6ff60 in ?? ()
(gdb) bt
#0  0x0000000001d6ff60 in ?? ()
#1  0x00007fffffffd738 in ?? ()
#2  0x0000000001e94f13 in ?? ()
#3  0x00007fffffffd6d0 in ?? ()
#4  0x00007fffffffd738 in ?? ()
#5  0x0000000003e9d691 in ?? ()
#6  0x0000000003e9d698 in ?? ()
#7  0x0000000003e9d691 in ?? ()
#8  0x00007fffffffd738 in ?? ()
#9  0x00007fffffffdc40 in ?? ()
#10 0x0000000001ccd0f0 in ?? ()
#11 0x00007fffffffd7a0 in ?? ()
#12 0x0000000000000001 in ?? ()
#13 0x00007fffffffd720 in ?? ()
#14 0x0000000001e92e92 in ?? ()
#15 0x0000000003e9d691 in ?? ()
#16 0x0000000003e9d698 in ?? ()
#17 0x00007fffffffd738 in ?? ()
#18 0x00007fffffffd738 in ?? ()
#19 0x00007fffffffd760 in ?? ()
#20 0x0000000001e9dd51 in ?? ()
#21 0x00007fffffffdc40 in ?? ()
#22 0x0000000003e9b3e0 in ?? ()
#23 0x00007fffffffd7e8 in ?? ()
#24 0x00007fffffffd7b8 in ?? ()
#25 0x00007fffffffd7b8 in ?? ()
#26 0x00007fffffffd828 in ?? ()
#27 0x00007fffffffd810 in ?? ()
#28 0x0000000001e9df09 in ?? ()
#29 0x612f656d6f682f1a in ?? ()
#30 0x0000786f636c6977 in ?? ()
#31 0x0000000000000000 in ?? ()
(gdb) info registers
rax            0x54454e4f4d5f434c  6072345775086453580
rbx            0x53                83
rcx            0x53                83
rdx            0x2                 2
rsi            0x53                83
rdi            0x3e9b1a0           65647008
rbp            0x7fffffffd6f0      0x7fffffffd6f0
rsp            0x7fffffffd690      0x7fffffffd690
r8             0x0                 0
r9             0x0                 0
r10            0x1                 1
r11            0x7fffffffdb9c      140737488346012
r12            0x7fffffffd6b8      140737488344760
r13            0x7fffffffd6b0      140737488344752
r14            0x7fffffffd6a8      140737488344744
r15            0x7fffffffd6c0      140737488344768
rip            0x1d6ff60           0x1d6ff60
eflags         0x10202             [ IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0

What are we trying to do? Looking at symbols present in the Spotify binary, this is actually part of the G++ runtime; specifically, std::ctype::do_tolower:

  1d6ff51:       48 8b 05 18 a8 12 02    mov    0x212a818(%rip),%rax        # 3e9a770 
  1d6ff58:       48 8b 40 70             mov    0x70(%rax),%rax
  1d6ff5c:       48 0f be cb             movsbq %bl,%rcx
=>1d6ff60:       8a 1c 88                mov    (%rax,%rcx,4),%bl
  1d6ff63:       89 d8                   mov    %ebx,%eax
  1d6ff65:       5b                      pop    %rbx
  1d6ff66:       c3                      retq

That rax value looks suspicious, and we can see if we translate it to ASCII that it is the little-endian representation of the string “LC_MONETARY”. We’re trying to reach 0x70 into a structure in %rax for a pointer value, but we’re getting a string instead.

It turns out that when libstdc++ is compiled on a glibc system, it will attempt to access the internal __ctype_* members in the locale_t of the current locale. musl’s locale_t is not ABI-compatible with glibc’s. In fact, it is only 48 bytes in length; 0x70 (or 112 bytes) is past the end of the locale object musl has provided it!

I implemented a stub locale module in gcompat, and… it tried to exec /proc/self/exe, which broke under the gcompat loader. This required me to write a patch interposing the execv* functions to catch this. And suddenly…

The lights that stop me turn to stone

Slight success! We have a Spotify window!

Spotify, but only a white screen

… but a blank white screen only. After some inspecting, I found that one of the many zygotes CEF was forking was segfaulting:

[158358.508029] ThreadPoolForeg[3230]: segfault at 0 ip 0000000000000000 sp 00007fe3203db448 error 14 in spotify[200000+1acd000]
[158365.067313] ThreadPoolForeg[3252]: segfault at 0 ip 0000000000000000 sp 00007f2d69c172e8 error 14 in spotify[200000+1acd000]
[158378.506832] ThreadPoolForeg[3312]: segfault at 0 ip 0000000000000000 sp 00007f52ed7c8448 error 14 in spotify[200000+1acd000]
[158383.654027] ThreadPoolForeg[3339]: segfault at 0 ip 0000000000000000 sp 00007fcb631eb2e8 error 14 in spotify[200000+1acd000]

I replaced from the Spotify DEB package with a matched-version from Spotify’s Open Source builds page. This allowed me to have more debugging symbols, and generating a core dump revealed:

Core was generated by ` --argv0 /usr/share/spotify/spotify --type=utility --field-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000000 in ?? ()
[Current thread is 1 (LWP 12774)]
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f79a8a3d671 in sqlite3MallocSize () at ../../third_party/sqlite/amalgamation/sqlite3.c:26957
#2  mallocWithAlarm () at ../../third_party/sqlite/amalgamation/sqlite3.c:26891
#3  sqlite3Malloc () at ../../third_party/sqlite/amalgamation/sqlite3.c:26913
#4  0x00007f79a8aff232 in sqlite3MallocZero () at ../../third_party/sqlite/amalgamation/sqlite3.c:27118
#5  pthreadMutexAlloc () at ../../third_party/sqlite/amalgamation/sqlite3.c:25755
#6  0x00007f79a8a4e9b2 in sqlite3MutexAlloc () at ../../third_party/sqlite/amalgamation/sqlite3.c:25298
#7  chrome_sqlite3_initialize () at ../../third_party/sqlite/amalgamation/sqlite3.c:24906
#8  0x00007f79a8a350bd in EnsureSqliteInitialized () at ../../sql/
#9  0x00007f79a8a30eb2 in OpenInternal () at ../../sql/
#10 0x00007f79a8a30dfa in Open () at ../../sql/
#11 0x00007f79a8fb8de6 in InitializeDatabase () at ../../net/extras/sqlite/
#12 0x00007f79a8fb9751 in LoadNelPoliciesAndNotifyInBackground () at ../../net/extras/sqlite/
#13 0x00007f79a5abe25b in Invoke<void (leveldb_proto::ProtoDatabaseSelector::*)(base::OnceCallback), scoped_refptr, base::OnceCallback > () at ../../base/bind_internal.h:498
#14 MakeItSo<void (leveldb_proto::ProtoDatabaseSelector::*)(base::OnceCallback), scoped_refptr, base::OnceCallback > ()
    at ../../base/bind_internal.h:598
#15 RunImpl<void (leveldb_proto::ProtoDatabaseSelector::*)(base::OnceCallback), std::__1::tuple<scoped_refptr, base::OnceCallback >, 0, 1> () at ../../base/bind_internal.h:671
#16 RunOnce () at ../../base/bind_internal.h:640
#17 0x00007f79a7776fa0 in Run () at ../../base/callback.h:98
#18 RunTask () at ../../base/task/common/
#19 0x00007f79a7792862 in base::internal::TaskTracker::RunBlockShutdown(base::internal::Task*) () at ../../base/task/thread_pool/
#20 0x00007f79a7792062 in RunTask () at ../../base/task/thread_pool/
#21 0x00007f79a77d42fb in RunTask () at ../../base/task/thread_pool/
#22 0x00007f79a7791a43 in RunAndPopNextTask () at ../../base/task/thread_pool/
#23 0x00007f79a7798386 in RunWorker () at ../../base/task/thread_pool/
#24 0x00007f79a77980f4 in base::internal::WorkerThread::RunPooledWorker() () at ../../base/task/thread_pool/
#25 0x00007f79a77d4a05 in ThreadFunc () at ../../base/threading/
#26 0x00007f79ac9fe2dd in ?? ()
#27 0x00007f79aca799e8 in ?? ()
#28 0x00007f7998247ce0 in ?? ()
#29 0x0000000000000000 in ?? ()
(gdb) frame 1
#1  0x00007f79a8a3d671 in sqlite3MallocSize () at ../../third_party/sqlite/amalgamation/sqlite3.c:26957
26957     return sqlite3GlobalConfig.m.xSize(p);

Inspecting the SQLite3 code, I realised that it was somehow getting a nullptr for the malloc_usable_size pointer. Further inspection revealed that this was not exactly the case:

(gdb) disassemble 0x7f79a77d5520
Dump of assembler code for function malloc_usable_size():
   0x00007f79a77d5520 :     push   %rbp
   0x00007f79a77d5521 :     mov    %rsp,%rbp
   0x00007f79a77d5524 :     mov    %rdi,%rsi
   0x00007f79a77d5527 :     mov    0x484a76a(%rip),%rdi        # 0x7f79ac01fc98 
   0x00007f79a77d552e :    mov    0x28(%rdi),%rax
   0x00007f79a77d5532 :    xor    %edx,%edx
   0x00007f79a77d5534 :    pop    %rbp
   0x00007f79a77d5535 :    jmpq   *%rax
End of assembler dump.

Looking at how the Chromium allocator works internally, the issue is that RTLD_NEXT won’t work on libraries loaded before libcef. And looking at the output of ldd spotify revealed both libm and libdl before libcef; musl always redirects these to libc for glibc ABI compatibility.

Using PatchELF to remove these two DT_NEEDEDs from the binary yielded a surprising result…

Music makes the people come together

Spotify on Adélie Linux
Spotify, playing “Rhinestone Eyes” by Gorillaz, on my Adélie laptop

It works! All the features I tested work: Spotify Connect, which means I can control the laptop’s playback using the iOS and Apple Watch apps; radio playback; Bluetooth speaker support.


You will need to download the official Spotify 64-bit DEB. I have not tested this on a 32-bit system yet, but I see no reason it won’t work. Once you have the DEB, extract the data.tar.xz file somewhere. Use PatchELF on the Spotify binary as so:

$ patchelf --remove-needed usr/share/spotify/spotify
$ patchelf --remove-needed usr/share/spotify/spotify

Move the extracted usr/share/spotify directory to your system’s /usr/share directory. For better integration, I moved the /usr/share/spotify/spotify.desktop file to /usr/share/applications. Then move the usr/bin/spotify link to /usr/bin.

Ensure that you have the latest gcompat installed. As I write this, only Adélie has the newest version in the current repo. I’ll be submitting merge requests to the distros I know that ship gcompat this week to ensure everyone has a chance to play around with the new bits.

Have fun!

Do you like running Spotify on musl? Or do you just like reading about fun hacks? Consider donating to Adélie to keep the fun going!