Tag: Apple

Reckless Software Development Must End

On the 6th of November, 2019, I made a comment on Twitter:

Okay, so today’s news isn’t as dramatic as Uber killing a homeless woman by not programming in the fact that pedestrians might not use crosswalks, but it is based in the same mode of thought.

Today’s news is that the US state of Iowa has had issues with their election processes (processes that are a bit too complex for me to provide you an overview in this blog). The problem boils down to reckless abandon of software engineering principles.

As reported in the New York Times and The Verge, in addition to many other outlets, there were a number of failings in the development and deployment of this software package that would have been trivial to prevent.

My personal belief is that the following issues significantly contributed to the failure we have seen.

No test plan

There was no well-defined plan of testing.

The test plan should have covered testing of the back-end (server) portion of the software, including synthetic load testing. My test plan would have included a swarm of all 1600+ precincts reporting all possible data at the same time, using a pool of a few inexpensive systems running multi-connection clients.

The test plan should have also included testing of the deployment of the front-end (user facing) portion of the software. They should have asked at least a few of the precinct staffers to attempt to complete installation of the software.

Ideally, a member of the development team would be present for this, to note where users encounter hesitation or issues. However, we are far from an ideal world. My test plan would have included a simple Skype or FaceTime session with the poll workers, if face-to-face communication would have been prohibitive.

These sessions with real-world users can be used to further refine the installation process, and can inform what should be written in documentation to simplify and streamline the experience for the general user population. Then, users should be allowed to input mock test data into the software. This will allow the development team to see any issues with the input routines, and function as an additional real-world test for the back-end portion.

By “installation”, I mean the set up required after the software is installed. For instance, logging in with the unique PIN that reportedly controlled authentication. I am not including the installation of the app software onto the device, which should not have been an issue at all — and which is covered in the following section.

Lack of release engineering

Software must be released to be used.

It appears that the developers of this software either did not have the software finished before the Iowa caucus began (requiring them to on-board every user as a beta tester), or they did not intend to have a proper ‘release’ of the software at any time (meaning every user was intended to be a beta tester). I could write a full article on the sad state of software release engineering, but I digress.

The software was distributed to users via a testing system, used for providing pre-release or “beta” versions to testers. This is an essential system to use when you have a test plan like what I described above. This is, however, a bad idea to use for releasing software for production.

On Apple’s platform, distributing final releases via TestFlight or TestFairy can result in your organisation being permanently banned from accessing any Apple developer material. Not counting the legal (contract law) issues surrounding such a release, on Android this requires your users to enable what is called “side-loading”, or installing software from untrusted third-party repositories.

All of the Iowa caucus precinct workers using the Android OS now have mobile devices configured in a severely vulnerable way, and they have had sideloading normalised as something that could be legitimate. The importance of this cannot be understated. This is a large security risk, and I am already wondering in the back of my mind how this will affect these same workers if they are involved with the general election in November. The company responsible for telling them to configure their mobile devices in this manner may, and in my opinion should, be liable for any data loss or exploitation that happens to these people.

My release plan document would have involved clearly defined milestones, with allowances for what features would be okay to postpone for later releases. This could include post-Iowa caucus releases, if necessary — the Nevada Democratic Party intended to use this software for their 22nd February caucus. Release planning should include both planned dates and required dates. For example:

  • Alpha release for internal testing. Plan: 6 December. Must: 13 December.
  • Beta release, sent for wider external testing. Plan: 3 January. Must: 10 January.
  • Final release, sent to Apple and Google app stores. Plan: 13 January. Must: 20 January.
  • Iowa Caucus: 3 February (hard).

Such a release plan would have given the respective app stores at least two weeks to approve the app for distribution.

Alternatively, if the goal was to avoid deployment to the general app stores of the mobile platforms, they could have used “business-internal” deployment solutions. Apple offers the Apple Business Manager; Google offers Managed Google Play. Both of these services are included with their respective developer subscriptions, so there is no additional cost for the development organisation.

Lack of security processes

Authentication control is important in all software, but especially so in election software. This team demonstrated to me a lack of understanding of proper security processes by providing the PIN on the same sheet of paper that would be used on the night of the election for vote tallying.

I would have had the PIN sent to the precinct workers via either email, or using a separate sheet which they could have in their wallet. Ideally, initial log in and authentication would have taken place on the device before the release, with the credentials stored in the secure portion of device storage (Secure Enclave on iPhone, TrustZone on Android). However, even if this is not possible, it was still possible to provide the PIN to users in a more secure manner.

Apparent lack of clearly defined specification

I have a sneaking suspicion that the combination of these failings mirror the many other development organisations who refuse to apply the discipline of engineering to their software projects. They are encouraged by bad stewards of engineering to “Move Fast and Break Things”. They are encouraged by snake-oil peddlers of “process improvement” that formal specification and testing are unnecessary burdens. And this must change.

I’m not alone in this call. Even the Venture Capitalist section of Harvard Business Review admits that this development culture is irresponsible and outdated. Software developers and project managers must be willing to #Disrupt the current industry norm and be willing to Move Moderately and Fix Things.

FreeBSD on Apple MacBook Pro 8,2: Epilogue.

Why I left FreeBSD.

It is with a fairly heavy heart that I write I am no longer running FreeBSD on my MacBook Pro.

What happened to improving?

Part of the problem is that I finally received gainful employment in March, and that work is almost impossible to do on FreeBSD. A lot of it involves Chrome (which I still have been unable to run on FreeBSD), Qt5-based applications (which crash due to known bugs in libv8 that Google do not care to resolve), and some Python libraries that have truly terrible performance on FreeBSD.

Why not run Linux in a VM for work?

Sure, I could have, if VirtualBox ever worked…

Weren’t you excited to fix up FreeBSD?

I was. I still am, but something just feels different. For over a decade, FreeBSD has for me been the go-to operating system for any use case: servers, embedded projects, desktop systems, and more. But the current heading of development seems to strongly suggest this is no longer encouraged or desired.

When I first started out with Gentoo nine years ago, they were pretty much bent on making it for newer hardware only. Back then, Pentium computers were like the Pentium 4s of now – something you give your grandma or little sister for web browsing, but nothing too serious. And Gentoo developers did not really care if they broke compatibility with these older systems. I can understand that, given that compiling the entire system by hand is something that is pretty taxing for older hardware.

The nice thing about FreeBSD was their community never looked down on you for using these older machines, and realised they still have use. My first interactions with #FreeBSDHelp on EFnet were in 2006 and related to getting SLIP support working in sysinstall so I could remotely install FreeBSD 6 on my Pentium 90 laptop. They were happy to help.

The roles have largely reversed now. Running into issues with older hardware get me looks of disdain and “great, upgrade your hardware and try again” from the FreeBSD community. Meanwhile, the Gentoo team was happy to help me with an issue regarding my retro Intel486 box, in 2015. This computer has no business still functioning, and they were still willing to help me configure a kernel that would boot on it with its anaemic 20 MB RAM.

The other thing I have noticed is that even now, months later, none of my Ports bugs have been handled. In the same amount of time, I have filed three bugs against Portage packages… and all of them were closed within one week of being opened. I feel like my contributions matter to the Gentoo Linux team.

What have you learned?

FreeBSD is more fun to hack on than Gentoo. FreeBSD is harder to get things done on than Gentoo.

FreeBSD is lighter on resources than Gentoo. FreeBSD is heavier on bug backlog than Gentoo.

FreeBSD on Apple MacBook Pro 8,2: Now with more features

Making FreeBSD perform well on a MacBook Pro (Late 2011).

As an update to my last post, I have been able to make more of the features of my 13″ Late 2011 MacBook Pro work on FreeBSD 11.0-CURRENT.

Sensor Readings.

Simple: kldload coretemp. To have on boot, add coretemp_load="YES" in /boot/loader.conf.

Cool Temperatures and Power Saving.

You will need to ensure i915kms is loaded and add drm.i915.enable_rc6=7 to your /boot/loader.conf. This was pointed out by the FreeBSD wiki which I could not find, so thank you Elly for helping me find these docs.

Additionally, you can add hw.pci.do_power_nodriver=3 to power down unused PCI cards. They re-enable when you kldload the modules, so this will help if you do not use the SD card reader or FireWire port very much.

Touchpad Gestures.

The atp(4) driver is flaky, but does work. Just add atp_load="YES" to /boot/loader.conf. Two fingers = right-click, three fingers = middle-click. Three finger tap is somehow hilariously much more reliable than two fingers, but with a little persistence they work. Two-finger scroll is a bit wonky and scrolls at unpredictable speeds, but I have a feeling this is a sysctl that may be fixable.

Bluetooth.

This actually does work right out of the box. I can pair with my phone and use the nifty obexapp utility to transfer files to my Android phone, an HTC One (M8). Unfortunately I was unable to transfer any files or pictures from the phone; obexapp consistently reported “0 bytes streamed in 2 seconds”. I think it may be a permissions error as I able to upload files to my phone’s Download/ directory correctly but no other directory.

To establish a Bluetooth connection (with an Android phone at least), you have to do the following:

  1. Add your phone to /etc/bluetooth/hcsecd.conf. The examples are quite good. Remember your PIN.
  2. Enable hcsecd in /etc/rc.conf and set flags to ubt0.
  3. Enable sdpd in /etc/rc.conf.
    Your /etc/rc.conf should now have at least the following lines:

    hcsecd_enable="YES"
hcsecd_flags="ubt0"
sdpd_enable="YES"
  4. Start both services… 
    /etc/rc.d/hcsecd start
/etc/rc.d/sdpd start
  5. Now run obexapp -a phone:bluetooth:mac -C FTRN. Your phone will ask you for the PIN you specified in /etc/bluetooth/hcsecd.conf. Enter it, and you can see files on your FreeBSD computer!

Some More Bumps.

With all of these things fixed, I have had two more snags. One was that my tilde key (the ~ next to the 1 key) on my MacBook’s internal keyboard was not working. This was fixed by adding the following line to ~/.xmodmaprc and running xmodmap ~/.xmodmaprc:

keycode 94 = grave asciitilde

My other new bump is that when I plug in headphones, the sound still comes out of the external speakers (and there is no sound in the headphones). This is probably due to some weirdness in the Apple HDA and I will have to take a closer look at this.

In conclusion, I am still quite happy with this change and am looking forward to fixing the rest of these issues.

 

FreeBSD on Apple MacBook Pro (13", Late 2011): Initial thoughts

Some background on why I left glibc Linux distros and went to FreeBSD. Includes tips on how to install FreeBSD on a MacBook Pro.

Background.

I rely on my laptop, an Apple MacBook Pro 8,2 (13″, Late 2011), for most of my work. For the past year or so I have been using Gentoo Linux on it. Gentoo is probably the best distribution of Linux out there today, but that isn’t really saying a lot about it. Underneath the beautiful and easy-to-use Portage system lies the same glibc, the same turmoil over a switch to a less-than-ideal init system, and the same kernel-level bugs that bring my productivity down.

I had considered sticking it out, and possibly even picking up maintaining OpenRC if it is abandoned as some people seem to want. However, I have been having many glibc-related bugs lately:

  • Almost all GTK apps keep locking up in waitpid.
  • gdb locks up when I try to attach to processes – even after checking /proc/sys/kernel/yama/ptrace_scope and the like (no error, just hang).
  • Starting VirtualBox VMs cause the entire machine to lock up for up to two minutes before returning to normal. This isn’t from starting the VM either; it occurs before the VM can even begin.

I don’t take system-level changes lightly, but it was time to try something else. Trying to rebuild everything against something like musl seemed like an interesting idea, but it still wouldn’t solve the eventuality of systemd. Additionally, there are still other compatibility issues using musl. And if I was having to rebuild everything anyway, I might as well try something completely different…

Enter FreeBSD.

I have been playing around with FreeBSD on my UltraSPARC system, as I wrote earlier. It is pretty snappy on there, even though it is a very old and slow system. Therefore, I decided to research about FreeBSD on MacBooks. There are a few requirements I have with a primary OS:

  • It absolutely must boot natively via EFI. I do not have the time to wait 90+ seconds for BIOS boot, and some of the hardware in the MacBook won’t even perform correctly under that mode; the DVD drive and the Bluetooth are very fickle, at least in my experience.
  • It must be able to run a performant virtualisation solution, like VirtualBox, which is my go-to system emulator. I use virtualisation heavily, so this is a requirement.
  • It must work with the hardware components I use on my computer. This is primarily Gigabit Ethernet, FireWire, the SD card reader, and the Bluetooth radio.

I had enough disk space free on my Mac OS X partition to resize it down and give FreeBSD a “comfortable” 64 GB. Since this is just for testing, I left my Gentoo install and its LVM PV alone. I resized the Mac OS X partition and moved down the LVM PV to put FreeBSD in the middle (otherwise the space would be oddly divided) using GParted running on the Gentoo install. I was horrified when it, too, was locked in waitpid. I waited over three hours before it finally became responsive again, and hilariously enough, it reported that the operations completed in just 49 minutes.

I created a 300MB HFS+ partition used for the EFI loader, since Apple’s firmware really likes having EFI code on HFS+. This also allows it to show up native in Chooser. You can also use FAT16 or FAT32, but I prefer to use HFS+ for this since it is just a single file. The rest of the 64 GB was given to a single FreeBSD UFS partition.

Next, I did something that I absolutely do not recommend for any reason unless you are willing to accept permanent data loss: I ran a Qemu instance with /dev/sda as the virtual hard disk. This allowed me to manipulate the disk directly from Qemu, which was convenient since the FreeBSD UEFI USB install image does not yet support Apple’s firmware. However, if I typed even one letter or number wrong, this could have erased my entire drive so please do not do this! I fetched the sets manually from FreeBSD’s FTP archive and extracted them to the new FreeBSD UFS partition on my disk. I shut down the Qemu instance after I configured /etc/rc.conf to my liking.

Finally, back in Gentoo, I copied the /boot/boot1.efi file from the FreeBSD UFS partition to the HFS+ partition, using the name boot.efi. The directory tree on the HFS+ partition, to boot natively and show up in Chooser (and Startup Disk in Mac OS X), is as follows:

  • /mach_kernel – 0 byte file, must be present to show in Startup Disk.
  • /EFI – copy this directory from the ESP. It contains the encrypted firmware used to start the computer’s components.
  • /System/Library/CoreServices/boot.efi – this is the FreeBSD /boot/boot1.efi file.
  • /System/Library/CoreServices/SystemVersion.plist – this is an XML file that you can put the details of your FreeBSD install into to show what version it is and such. This file isn’t strictly necessary but adds a very nice touch of professionalism to the install.

Booting the FreeBSD: EFI on the First Try!

I snuck in a /.VolumeIcon.icns file with the FreeBSD Beastie head logo, then rebooted my system. I waved goodbye to Gentoo, but wasn’t sure if my FreeBSD installation was fully successful. Lo and behold!

I pressed RETURN and hoped for the best… and the best is what I received!

I was overjoyed to see how fast it boots; under 20 seconds from Chooser to a login prompt. I spent the rest of the night installing binary packages using pkg for software which I could tolerate the default options, but actually installed most things from the Ports system. Xorg support was native and flawless, including DRM for acceleration using my laptop’s Intel HD Graphics 3000. The next morning I had the entire KDE 4 base system built and working in just two hours! KWin’s compositing looks just as good as it did in Linux, but now uses much less CPU and memory – the entire thing boots up (using KDM4 to start my KDE session) using just 290 MB RAM, and will actually idle at 0.0% CPU (Linux would idle around 1.1-1.5%).

Some Issues.

While FreeBSD appears to be a panacea for everything I had wrong in Gentoo Linux, all is not immediately well. Of course, as with any new system, there are some more things I need to fix and figure out:

WiFi/Bluetooth
These will need some configuring and driver twiddling, but I have high hopes based on what I read on the very useful FreeBSD forums and mailing lists.
Ext2FS crashes
I backed up my system before installing FreeBSD, of course, but my backup disk (along with my primary disk which was unharmed) is formatted as ext4 (Linux FS). FreeBSD has a module to be able to read it, but when I try and access almost any file, it has a kernel panic. I am currently looking in to what is causing this and how to fix it.
Sensor readings
Currently, my laptop thinks it is -273.1 °C. This is quite obviously impossible, so I will need to investigate how to fix KDE’s system monitor widget 🙂
Touchpad
While the FreeBSD kernel has a great touchpad driver merged in to 11.0-CURRENT, and it shows up in my kernel log as I boot up, Xorg only sees my mouse as having one button with no gesture support either. This means no scrolling on the side or with two fingers, no right clicking, no middle click for pasteboards, and so on. This will also need some fixing.

Overall Thoughts.

FreeBSD is very performant on this hardware, and seems to be doing a good job driving it. There are some quirks to it, but this is a new system and the kernel I am running is a beta anyway, so it of course will not be perfect. I am happy with this switch though and probably will not go back to Linux any time soon unless they can sort out their issues. I hope they can, because the Linux kernel itself is actually decent, and has a lot of commercial support which is good for open-source going forward. However, FreeBSD’s performance is amazing and gives me renewed hope for the quality of systems that can be made in open-source communities.

Now playing: Devil in a New Dress – Kanye West (Double entendre… FreeBSD’s logo is Beastie, and my laptop is a devil of a computer wearing a new dress with FreeBSD.)