Tag: politics

Reckless Software Development Must End

On the 6th of November, 2019, I made a comment on Twitter:

Okay, so today’s news isn’t as dramatic as Uber killing a homeless woman by not programming in the fact that pedestrians might not use crosswalks, but it is based in the same mode of thought.

Today’s news is that the US state of Iowa has had issues with their election processes (processes that are a bit too complex for me to provide you an overview in this blog). The problem boils down to reckless abandon of software engineering principles.

As reported in the New York Times and The Verge, in addition to many other outlets, there were a number of failings in the development and deployment of this software package that would have been trivial to prevent.

My personal belief is that the following issues significantly contributed to the failure we have seen.

No test plan

There was no well-defined plan of testing.

The test plan should have covered testing of the back-end (server) portion of the software, including synthetic load testing. My test plan would have included a swarm of all 1600+ precincts reporting all possible data at the same time, using a pool of a few inexpensive systems running multi-connection clients.

The test plan should have also included testing of the deployment of the front-end (user facing) portion of the software. They should have asked at least a few of the precinct staffers to attempt to complete installation of the software.

Ideally, a member of the development team would be present for this, to note where users encounter hesitation or issues. However, we are far from an ideal world. My test plan would have included a simple Skype or FaceTime session with the poll workers, if face-to-face communication would have been prohibitive.

These sessions with real-world users can be used to further refine the installation process, and can inform what should be written in documentation to simplify and streamline the experience for the general user population. Then, users should be allowed to input mock test data into the software. This will allow the development team to see any issues with the input routines, and function as an additional real-world test for the back-end portion.

By “installation”, I mean the set up required after the software is installed. For instance, logging in with the unique PIN that reportedly controlled authentication. I am not including the installation of the app software onto the device, which should not have been an issue at all — and which is covered in the following section.

Lack of release engineering

Software must be released to be used.

It appears that the developers of this software either did not have the software finished before the Iowa caucus began (requiring them to on-board every user as a beta tester), or they did not intend to have a proper ‘release’ of the software at any time (meaning every user was intended to be a beta tester). I could write a full article on the sad state of software release engineering, but I digress.

The software was distributed to users via a testing system, used for providing pre-release or “beta” versions to testers. This is an essential system to use when you have a test plan like what I described above. This is, however, a bad idea to use for releasing software for production.

On Apple’s platform, distributing final releases via TestFlight or TestFairy can result in your organisation being permanently banned from accessing any Apple developer material. Not counting the legal (contract law) issues surrounding such a release, on Android this requires your users to enable what is called “side-loading”, or installing software from untrusted third-party repositories.

All of the Iowa caucus precinct workers using the Android OS now have mobile devices configured in a severely vulnerable way, and they have had sideloading normalised as something that could be legitimate. The importance of this cannot be understated. This is a large security risk, and I am already wondering in the back of my mind how this will affect these same workers if they are involved with the general election in November. The company responsible for telling them to configure their mobile devices in this manner may, and in my opinion should, be liable for any data loss or exploitation that happens to these people.

My release plan document would have involved clearly defined milestones, with allowances for what features would be okay to postpone for later releases. This could include post-Iowa caucus releases, if necessary — the Nevada Democratic Party intended to use this software for their 22nd February caucus. Release planning should include both planned dates and required dates. For example:

  • Alpha release for internal testing. Plan: 6 December. Must: 13 December.
  • Beta release, sent for wider external testing. Plan: 3 January. Must: 10 January.
  • Final release, sent to Apple and Google app stores. Plan: 13 January. Must: 20 January.
  • Iowa Caucus: 3 February (hard).

Such a release plan would have given the respective app stores at least two weeks to approve the app for distribution.

Alternatively, if the goal was to avoid deployment to the general app stores of the mobile platforms, they could have used “business-internal” deployment solutions. Apple offers the Apple Business Manager; Google offers Managed Google Play. Both of these services are included with their respective developer subscriptions, so there is no additional cost for the development organisation.

Lack of security processes

Authentication control is important in all software, but especially so in election software. This team demonstrated to me a lack of understanding of proper security processes by providing the PIN on the same sheet of paper that would be used on the night of the election for vote tallying.

I would have had the PIN sent to the precinct workers via either email, or using a separate sheet which they could have in their wallet. Ideally, initial log in and authentication would have taken place on the device before the release, with the credentials stored in the secure portion of device storage (Secure Enclave on iPhone, TrustZone on Android). However, even if this is not possible, it was still possible to provide the PIN to users in a more secure manner.

Apparent lack of clearly defined specification

I have a sneaking suspicion that the combination of these failings mirror the many other development organisations who refuse to apply the discipline of engineering to their software projects. They are encouraged by bad stewards of engineering to “Move Fast and Break Things”. They are encouraged by snake-oil peddlers of “process improvement” that formal specification and testing are unnecessary burdens. And this must change.

I’m not alone in this call. Even the Venture Capitalist section of Harvard Business Review admits that this development culture is irresponsible and outdated. Software developers and project managers must be willing to #Disrupt the current industry norm and be willing to Move Moderately and Fix Things.

Libre software funding and market abuse

I’ve just read a troubling article from the developer of Aether.

What troubles me is not so much the differences we have, which likely stems from being in vastly different segments of libre software (he’s doing social media, and I’m in low-level systems). What troubles me is that he claims that it is an economic imperative to work at FAANG or a Silicon Valley startup for a number of years before working on libre software full time, and all of this on a false pretense.

Encouraging someone to have a long-term savings and funding plan is a good idea, perhaps even a great idea. It falls apart when he states that working for startups or FAANG are the only or best way you can earn that money — and then claiming that you could make 250,000 USD per month working at them[1]. This is flawed mathematics at best, and actively malicious to society at worst.

Most people are going to have to work at a company before founding their own, unless they have funding from external sources (be it angel investors, VC, inheritance, family and friends, etc). This is not what I take issue with. This issue I have is this false dichotomy that you can only make good money by working at FAANG or an abusive startup. As someone who actually has worked at two different startups in their life, I take personal issue with the way startup culture exploits its workers, investors, and society at large. This doesn’t even go in to how startup culture can also be bad for business.

This abuse is ingrained in to most, if not all, of the industry of Big Tech, ala FAANG. You might be able to wrestle some division of Apple, or the security research division of Netflix, out of this hole and point to them as an example of where I’m wrong. Oh, dear reader — even if you have the privilege of working in an area of the company that isn’t abusing its workers, you’re still complicit in that abuse by furthering the company’s mission and control over some part of the industry at best, and indirectly engaging in it yourself at worst.

It’s time for the computer industry to rise up and work at companies that respect their workers, and society. Quit FAANG like a bad habit, and find a company to work for that doesn’t trade in the abuse of power and users as its main product. And where those don’t yet exist, it’s time to found some. At the end of the day, we are all defined by the actions we take — which side of history do you want to be on?

[1]: And I quote, “If you can make $10,000 a month from donations doing open source work, I can guarantee you that your salary in any large tech company (or even startup) would be much more — to the tune of 10x to 25x.” The firm Indeed claim, at time of writing, that the highest paid research engineers at Google make about 246,000 USD per year; other companies pay even less. That’s 20,500 USD per month, or just about twice the amount he claims you might be able to make on donations doing ‘open source work’. And this doesn’t require you to further Google’s surveillance state.

Mozilla finally disavows Discord

mhoye’s new blog post on the future of Mozilla community chat came out last week. He notes about Discord that “their active hostility towards interoperability and alternative clients has disqualified them as a community platform.”

I am very thankful that the Mozilla brass have realised this, as I pointed out in an earlier installment. Kudos to them that three of their four options are fully libre — I sincerely hope they choose one of those three, and keep the Mozilla community libre and open.

The problem with “patches welcome” culture

I’m going to tell you a secret.

Most computer users cannot write computer code. (Shock!) This is not anything new, and I would dare say this is not even necessarily a problem that needs to be corrected. In a similar vein, a great deal of first-world citizens use cars daily, but I doubt many drivers would be able to fully rebuild an engine, or even describe the difference between EFI and carburetors.

This is the fundamental flaw behind the “patches welcome” culture, and why some libre open source projects have less-than-ideal user experiences, and some even have communities that most would describe as “elitist”.

While drafting this article, a few people told me that they agreed with the substance but did not like that I was using “patches welcome” to describe the culture. While it is correct that most libre software projects should be welcoming of patches, that is not what this article is about. When these people say “patches welcome”, it is a deflection; they don’t want to put forth the effort to properly maintain their software.

Let’s consider an example of this. In a welcoming environment that fosters participation and communication, a request for a feature from a user typically goes somewhat like this:

User: I would really like to be able to select an entire sentence using a key combination so that I can make the sentence bold or underline without dragging.
Developer: Okay. We’ll add that to the list of features that have been requested. Thank you for telling us!

Sometimes these features take time to add; maybe some will never see the light of day. Nevertheless, the user still informed the developer of the software what they needed, which allows the developers to make better choices about how they approach building the software, and what features to prioritise.

Now, in a project with “patches welcome” culture, the users are ignored or even chastised. I’ve actually seen discussions very similar to the following take place:

User: I would really like to be able to select an entire sentence using a key combination so that I can make the sentence bold or underline without dragging.
Developer: patches welcome
User: I don’t know how to write code.
Developer: Then you shouldn’t ask us for help.

This behaviour undermines what libre software is supposed to stand for. It gives the user reason to go back to using proprietary software, where they can call up Microsoft or Apple and tell them what they want or ask for help when they need it. Even if the proprietary software corporations never add their suggestion, they still feel more connected and respected than they do by this example libre project. This behaviour gives the user no reason to use the software that respects them and their freedoms. Free Software is meaningless if it has no users to use it.

Some may argue that people should be empowered to learn to program, and there definitely is a case to be made for that. However, you really need to consider all the reasons people *would not* want to learn to program:

  • They would rather spend that time with family, friends, or their hobbies.
  • They have learning or mental disabilities that make algorithmic reasoning, logic, or concentration required to program difficult.
  • They have physical disabilities that make programming difficult.
  • They simply aren’t interested.

That last item is especially important. Do you want someone who has no interest in programming – no interest in security, or correctness, or doing things the Right Way (or even the way you want them done) – to commit code to your repository? Are patches really welcome, or are you just deflecting the requests of your users so you don’t have to maintain the software you’ve written?

All people deserve Free Software. Nobody deserves to be denigrated, shamed, or ignored because of their inability or lack of desire to program a computer. My personal suggestion to those who do not want to accept feature requests unless patches are attached is to not publicly release your software. If you do, add a notice stating that you do not wish to be contacted about your software by users, so that they may make an informed decision about the software that they use.