> For 33 hours in late August, a rival network hijacked a block of Hetzner-hosted infrastructure serving Softaculous and Virtualizor, using it to issue a valid TLS certificate and push a malicious software update. The stolen route was RPKI-invalid the entire time, on file, checkable by anyone. It still won on most of the internet.
> At 20:57 UTC on 28 August, a network nobody at Hetzner or Softaculous had dealt with before started announcing 162.55.80.0/24 to the internet. It was a small, unremarkable-looking slice of Hetzner’s own address space — the block hosting Softaculous’s client area, its billing pages, and the update endpoint that every Virtualizor-licensed control panel on the internet periodically calls home to. For the next 33 hours, with an 11-hour lull in the middle, a meaningful share of global traffic bound for that block went to the hijacker instead. Some of it was Let’s Encrypt’s own domain-control validation, which is how the attacker walked away with a certificate for virtualizor.com that no browser would ever flag.
* https://cloudpunk.blog/post/bgp-hijack-rpki-gap/
So it's all very well to have multiple network locations to prevent hijacking on the path (i.e., you have to attack multiple paths), but if the network as a whole thinks the destination is someplace else then all those paths will lead to the same (false) location.
Perhaps the CAB should consider RPKI validation as a "MUST" rather than piling on multiple sources of validation who still have the same single point of failure (the spoofing of the destination).
I do prefer the current model (the trade-offs are worth it, IMHO), but it wasn't always this way.
> An Extended Validation (EV) Certificate is a certificate conforming to X.509 that proves the legal entity of the owner and is signed by a certificate authority key that can issue EV certificates. EV certificates can be used in the same manner as any other X.509 certificates, including securing web communications with HTTPS and signing software and documents. Unlike domain-validated certificates and organization-validation certificates, EV certificates can be issued only by a subset of certificate authorities (CAs) and require verification of the requesting entity's legal identity before certificate issuance.
* https://en.wikipedia.org/wiki/Extended_Validation_Certificat...
Within a few years at most the Race to the Bottom is well underway - if I spent $10 on my "streamlined" process to offer 1000 certificates for $10 each I make more money than a rival spending $1000 to offer 100 certificates for $100 each. As a result "Domain Verification" is normal by the turn of the century. If you weren't here for the Dotcom boom, you have never seen any of this. If you bought books on this stuff on Amazon, or worse you've never read any books about it because it was all online, you arrived after DV was everywhere.
A peer mentions EV, that and OV are actually newcomers, the CAs realised that customers don't want to pay $100 for a "better" certificate when customers can't tell the difference so there began a meeting, the CA/Browser Forum sometimes just referred to as "cab" or "cabf". This meeting had two initial outputs. One of them endures to this day and only got stronger:
1. Certificate Authorities must meet at least certain Baseline Requirements (the "CA/BF BRs") for all certificates. You can't issue certificates which do not meet these requirements, putting a floor on how terrible your verifications can be.
2. Browser Vendors will add shiny UI for a new "Enhanced" verification certificate so that customers can see it's better and that's a new sales opportunity for CAs.
In practice after some years (2) stopped making sense. A shiny UI works on a desktop PC but there's no room on an early smartphone, so you just don't add this. Soon enough you make Desktop browsers look the same as your smartphone product and all the time your UX experts are telling you users do not undertand this UI (no surprise it was just a sales gimmick) and Safety experts are telling you that it's a net negative for user safety. So, bye bye EV UI in browsers.
But the other side of the bargain went very well. The BRs were a ratchet. In 2012 a ten year certificate issued based on some sales guy who met the director of a company at lunch was fine. By 2016 even five years is clearly too long and we should check some actual official records or have some kind of formal process. In 2012 it's fine to issue for mysql.test.internal because that's not a public name right? By 2016 uh, no, that's why it's not OK to issue that certificate, nobody could possibly own that name, do not issue bogus certificates, this is Serious Business. The BRs are how, gradually, we got where we are today.
It's like those arcane verification checks used by US-based services that ask for utility bills, as if they had any way of verifying an invoice issued by a small utility provider in rural Finland.
I personally believe the real fix would be to incentivise the use of CAA records, ideally including the accounturi parameter.
[1]: https://ipinfo.io/blog/rpki-protects-70-internet-routes
And ASPA validation[1]
[1] https://blog.apnic.net/2026/07/09/aspa-at-apnic-strengthenin...
I'm not sure how CAs deal with situations when a domain has several authoritative DNS servers, but not all of them respond with expected reply. Would it would be a good idea for CAs to ask *all* authoritative DNS servers and require a coherent answer from them?
For the case of the dns-01 challenge type, sure. But if you're using http-01 (or the attacker asks for it), then all those DNS servers will answer back with the same IP address for the A(AAA) query and you're back to the hijacking scenario.
Good point, I forgot about that! But then CAA record might help, I guess, if you specify your accounturi?
* https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-ac...
I meant to say that CAA record can also specify account, or at least Let’s Encrypt docs mention it[1]. So unless attacker has access to victim's account - they can't get a cert (unless they can hijack dns, too).
[1]: https://letsencrypt.org/docs/caa/#the-accounturi-parameter
Moreover, with dns-01, DNS provider can issue fake certificates - unless you self-host DNS - in which case, only your DNS registrar can... and a government.
Which is what DNSSEC is supposed to help with: the registrar/government would have to replace your DS/KSK with their own. Without DNSSEC queries/replies could potentially be altered over the wire (or you're in a race condition as to whether the real or attacker's reply get back first).
Maybe in practice this is good enough, but the security nerd inside me rebels at how sloppy the whole thing feels.
When we think about "sound engineering" this is the level we need to start at, not "what features could we bolt on to core Internet infrastructure to define the problem away".
The people most avidly pushing DNSSEC for this problem aren't stupid. They're just smuggling in an additional premise: that the WebPKI itself is bad, and should be replaced with a DNS PKI. Seen through that lens, secure proof of domain ownership would be deeply problematic, as it would leave the DNS PKI with literally no reason to exist.
Except for all the legacy systems and embedded devices I have to deal with that have no certificate automation functionality and are still within their budgeted lifetimes. Those dumb things still require manual enrollment (or, at the very least, ugly one-off scripting to hand-puppet their stupidity) and the decreasing certificate lifetimes makes the problem that much more excruciating.
LetsEncrypt is what finally solved the problem with the previous systems were perfect was the enemy of the good. There used to be a lot of focus on trying to verify that the person getting the cert was who they said they are, but LetsEncrypt came and sad "that part isn't important, protecting the data is what really matters" and basically solved the real problem wholesale. Very few people inspect the certificates on every website they visit, so those protections were mostly useless and created a roadblock to widespread adoption of HTTPS, scoping the problem down to just making sure nobody can inspect or modify the data in transit made automation not only possible but easy.
> That summer, Edward Snowden came forward with chilling revelations about NSA mass surveillance, and the need for ubiquitous encryption became clearer than ever.
It’s not a claim that it was the instigator, but it seems like it was heavily on their minds.
Google made HTTPS default-on in 2010, but there was a toggle so that you could disable the forced use:
* https://workspaceupdates.googleblog.com/2010/01/default-http...
In 2014 the toggle was removed and all connections were forced to HTTPS.
The Snowden revelations occurred in 2013:
Related: https://www.jefftk.com/p/history-of-https-usage
[1] https://googleblog.blogspot.com/2011/10/making-search-more-s...
We really need a CA that does for mail what Let's encrypt has done for HTTP:
The CA needs to provider proper S/MIME certificates:
- free like Let's Encrypt
- duration of 1 year or longer and
- users generate their own keys
Those issues are partially addressed with "identity based encryption," but IBE always has a "backdoor" because there is a party that issues the users' private keys (and hence can always read a users' messages). On the other hand IBE does allow you to "just send" an email to someone because the public keys are just the "identities" (e.g. email addresses) combined with a single master public key. Key management is not as much of a problem because of how keys are generated. IBE does have some users, but as you said it is mostly business/organizational use.
If you could actually solve the deeper problem of the sender being able to find the correct certificate to use for the recipient(s), then the lifetime could and should be as short as possible, as in days not years. The recipient can decrypt the message forever, as long as they continue to possess the private key, even if the associated certificate expires. Short-lived certificates are the only effective mechanism for dealing with key theft and mis-issuance that works at scale.
Let's Encrypt is bringing the duration of certs down not up. I believe their goal is 30 minutes validity.
This is mandated by the CA/Browser Forum, which, as the name implies, is related to browsers. It says nothing about e-mail clients. In its Extended Key Usage filed (RFC 5280 § 4.2.1.12) a certificate could not have server or client authentication, but have email protection as an allowed use.
Edit: there is an RFC for "Extensions to Automatic Certificate Management Environment for End-User S/MIME Certificates"
The most aggressive plans (not Let's Encrypt's specifically) target 7 day validities.
If the cert lifetime is 5 years, Bill installs "just the first one" manually, solemnly swearing they will automate next time, and in five years nobody even knows who Bill was, Bill left three years ago. How do we replace the certificate?
If the cert lifetime is 2 years. Bill is still there or maybe somebody on the team still knows Bill's email address, but most likely the decision is taken to "just do it manually once because it's urgent" and we just recurse. This would have been the "obvious" choice when ISRG lit up Let's Encrypt.
But because the cert lifetime was 90 days, Bill gets told it's expiring at about the same time he's about to shove that "Actually automate certificates" work onto the back burner. Oh yeah, right, we should actually do that.
Now, it may be that at a startup that "Automation" is initially "Put it on Natalie the intern's TODO list every Monday" but by week six Natalie has learned about cron, or Windows event scheduling, or a dozen other ways to make stuff you always do regularly happen without actual work. Today I expect her LLM will tell her how to do it.
And Natalie is going to discover that some "permanent" solutions are more permanent than others, that you want monitoring and so on. But because Let's Encrypt is so popular unless you're doing something pretty esoteric you won't even be the first person to ask about it this month.
Addding new methods of validation does not help nearly so much as just supporting existing methods for more than a few years would. The race to the bottom will end up with an incredibly fragile system that breaks the moment automation does. Combine this with the trend towards HTTP-only and any given website will only have a lifetime of a couple years without direct human intervention. This is a drastic change from the status quo of websites simply lasting forever and being accessible till their webservers stop.
It is 100% a corporate-for-profit-only mindset and architecture. LE is not designing or accomidating human person goals and needs.
It's practically zero overhead, low maintenance (most updates are not breaking) and you get the latest version of everything. If you're worried about supply chain attacks set a minimum release age.
For example, if you were validating example.com, you'd need to set up some response to the ACME challenge somewhere, either through direct DNS-based validation (DNS-01, DNS-PERSIST-01) or through a server running at the resolved A/AAAA IP address for example.com (HTTP-01, TLS-ALPN-01). In all cases, when a CA goes to validate your challenge response, it relies both upon DNS and upon the network integrity between the CA's datacenters and your DNS and/or app servers. Both of these can be subverted, eg via BGP attack, to allow an attacker to replace a DNS response, re-route the destination IP address to a different server, etc, at which point they can inject their own challenge response, and therefore issue their own (unauthorized) certificates.
Multi-perspective validation means that the CA validates your challenge response from multiple datacenters on different networks instead of just one, such that a hypothetical network attack would have to target multiple independent networks in order to successfully issue their unauthorized certificate.
More info:
https://letsencrypt.org/2020/02/19/multi-perspective-validat...
https://community.letsencrypt.org/t/multi-perspective-valida...
But remote perspectives are still worth doing, Let’s Encrypt should not be the “less diverse” than the infrastructure of their typical certificate requestor.
However, if you can MitM the uplinks at all of Let's Encrypt's perspectives, you can basically break the entire CA for everyone. Adding more perspectives reduces the risk of that.
> Domain Validated (DV) certificates are the least-identity-validated SSL certificates and can be obtained quickly and easily—even by a malicious bot. These certificates are low-cost certificates that only require validation that a company or person can demonstrate control over a web domain for which they want to secure a certificate.
This is arguably a flaw in allowing DV certs at all, but given that it's there, LE doing this raises the bar for spoofing about as far as it can be raised.
That's a policy, not an inherent consequence of DV certs.
If every verification method other than DNS was shut off, it would still be DV, but with care it could be immune to MitM. I'm not advocating for that, but it's an option.
I think few independent site owners would go through the more costly and complicated EV process.
I say “conceptually” since I know: 1. Actual attacks found with CT have used compromised TLDs, which kills the whole thing anyway, and 2. I haven’t really seen where the hosting company’s infrastructure (BGP ads and internal routing) is what’s compromised. Well certainly in the latter case they can probably also get in to the hosting machines directly.
The Use case - Lot of SAAS based products allow/require the administrators to prove the ownership of a domain via ACME methods. The administrators then are allowed to manage the profiles for users that have email address with same domains in the SAAS product.
Ironically the testing setup itself used the network hack for the successful ACME challenge