I’ve just spent the better part of three hours arguing on IRC about Let’s Encrypt clients. After speaking with two others, I realised that nobody who I spoke with before knew their facts were facts.
Different people all told me various incorrect information, such as:
- No ACME client supports doing a manual DNS TXT record for verification bootstrapping until you have an httpd up. (acme.sh, dehydrated, and certbot all support this.)
- LE needs IPv4 for the HTTP challenge. (It worked fine for me with an IPv6-only host. I’m not sure which it would prefer if it had the choice between v6 and v4, or if it’d use Happy Eyeballs and connect to whichever responded first.)
- It isn’t possible to step through the process manually as a debugging aid; you have to rely on your ACME client’s debugging facilities. (https://gethttpsforfree.com/ helped a tonne.)
- You have to be listening for the HTTP challenge on port 80. (The TLS-ALPN-01 challenge type exists which will only ever use port 443 for the challenge.)
- Critique of how I isolated each service on a separate VM so that they would be more secure, saying it was “over-convoluted”.
All of these people spoke with an air of authority. They sounded like they genuinely knew what they were talking about, and were trying to inform me of the limitations of ACME clients / Let’s Encrypt. Nobody actually knew the answer, but they thought they were right because it fit their experience.
When I speak to people about technology, whether in real life, on IRC, or on a mailing list, I always try to make the limitations of my knowledge clear. Many is the time I have said “I’m not sure if you can do that”, or “I don’t know if X supports Y“. And sure, on occasion I will say “I have never done Y and last I knew X couldn’t do that”. Note, however, that all of these are presented as statements from my hive of knowledge, and not presented as plain facts. The art of communication seems to be lost on far too many in the technology field.
There is no shame in not knowing the answer to something. It is certainly more helpful to say “I’m not sure you can do that”, instead of “you can’t do that”. I almost gave up on Let’s Encrypt and wrote another article on how useless it is, because I was told by people who used Let’s Encrypt that it had all of these limitations that made it seem useless, arbitrary, and ridiculous to me. (Thanks to Rich Felker of musl, and Freeyorp, for setting the record straight.)
Maybe we need a new term for this. “Organic FUD”, since it comes from the community itself? At any rate, I hope that in the future, more people note the limitations of their knowledge up-front rather than sounding authoritative about a subject they know little about.
The Cat Fox Life 
To be fair, the entire centralized certificate authority structure is overwrought and convoluted to begin with. The vast array of possible ways to verify with LE, the completely unstandardized DNS APIs, the bloated reference client, and the 90-day check-in dependency don’t contribute to any real security. It’s busywork for sysadmins and another thing to babysit. I have never enjoyed the process of acquiring or renewing my SSL certs, but I use it because it’s the only encryption available in widespread use over HTTP and some encryption is better than no encryption. Still, I kinda regret all the time I spent on it.
Writing as the prior maintainer of Certbot for your distro, it now has pre-,post-, and deploy- hooks that can be used to try to automate it, but it’s fragile in my experience. For my personal use, I ended up writing a script that was basically a big-ass Certbot invocation, tied together with secondary scripts for auth, cleanup, and deployment. Then you hook that up to cron and pray it works ~70-90 days later. Sometimes it works, sometimes it doesn’t. I tried editing the INI-style file to change the next renewal, but that backfired, so it defies conventional configuration ideas. Each time has been a different error, for me. I’m (un?)lucky enough to have a registrar that does its crap via an API, so I had to wait for my registrar to issue me a key to update my own DNS. In short, there’s a lot that can go wrong in the certificate issuing process, and the docs are not very clear about what the major failure states look like and their solutions. Last time I checked, they still don’t automate the combined.pem (cert+privkey) file needed by lighttpd to serve over HTTPS, so I had to add that to my scripting as well.
Additionally, to properly backup Certbot certs, you need all the historic ones or it’ll complain at you when you try to renew, because new certs want the old certs. Yet another place where renewing can go wrong.
Aside from that, Certbot has also experienced some feature creep. It used to be incapable of wildcard certs altogether. It used to not be able to verify over DNS. As the hurdle for Website maintenance raises, the cracks in peoples’ knowledge is going to show through. That’s what happens when knowledge becomes out of date, and there are only 24 hours in a day. 16 of them are usually gone for people who get enough sleep and work full-time, leaving a scant 8 (in practice it’s closer to 4 for most) hours to pursue life. Should these people commit hours of their time to learn every nook and cranny of their SSL renewer, to be told by some other stranger that their answer (which worked for them) was FUD or they should litter their language with qualifier phrases? Sorry, that’s too much social and emotional labor.
IMO there’s no good reason to make SSL so complicated. We need better trust models that don’t depend on central authorities who turn security into a business instead of a practice or mindset. Tackle the problem at the source: complexity.
Not to take away from your core point: people should be more forthcoming in their limits of knowledge. To that, I ask: why do you think people hide this?
I answer: because they do not want to be attacked, ridiculed, mocked, or be seen as unhelpful or unsure of their answer. There’s a lot going on socially when discussing these sorts of things, and personal experience is the ultimate bias modifier. We are all limited by perception, so telling someone that their statement was representative of their experience rather than fact is telling them their experience did not happen. What is fact if not a consensus of experiences?
I think communities that mock ignorance and embarrass earnest helpers contribute to what you’re pointing out. Your blog post wants people to admit when they don’t know something, but nobody’s saying “don’t be a dick to people who only know part of the answer”.
If your humility is not respected, then you’re just a doormat. People don’t like being doormats.
LikeLike
You make some really good points here.
> the combined.pem (cert+privkey) file needed by lighttpd to serve over HTTPS
Thankfully, 1.4.53 finally added the ability to have separate cert and key files: https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL
Much too late, I’m aware.
> Your blog post wants people to admit when they don’t know something, but nobody’s saying “don’t be a dick to people who only know part of the answer”.
That’s a really good point that I hadn’t considered. I have been very lucky that I have been able to leave most of the communities I was in who behaved that way, but not everyone is so lucky. Sounds like a topic for a future article to me.
LikeLike
Thanks for the lighttpd note. I’ll check it out the next time I’m reviewing my config.
If you write about the converse of this post, I’ll read it too. 🙂
LikeLike