Is this real? This seems so ridiculous it makes me wonder for how long this CA has been in circulation in systems and if it has been exploited by someone previously discovering it in that time (state level actor or not). My old phone running Android 6 does not have any certificate from EMudhra, at the very least.
There are plenty of ways you would never be able to tell if a fake cert had been generated and used via this attack vector (let's ignore validation attacks):
1. CT does not do anything if a state actor demands a CA in its borders generate a cert without publishing to the CT logs (or a three-letter-agency doing this illicitly). They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees.
2. Browsers won't check for SCTs in certs if they come from a private CA. you can MITM any domain through ZScaler all day long and nobody will notice
3. Wildcard certs make it easy to make/renew a cert that looks normal but hides requests on new/different hosts
“They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees”
I don’t think you can get the SCT from a log without the log publishing it? That would defeat the purpose.
“Certificates that support certificate transparency must include one or more signed certificate timestamps (SCTs), which is a promise from a log operator to include the certificate in their log within a maximum merge delay (MMD).[4][3] At some point within the maximum merge delay, the log operator adds the certificate to their log.”
I feel somewhat confident in speculating that a bunch of people at large companies will, in fact, be auditing the logs following this. That’s their main value; they’re not really an online/active prophylactic.
Nope. You can find this from the Chrome Root Program.
> Because no TLS certificates chaining to these six intermediate CAs were logged to CT prior to our CRLSet block on October 17, no fraudulent certificates issued by these CAs could have been trusted by default in Chrome or other CT-enforcing browsers.
I'm not exactly sure which of the roots from eMudra were compromised, but the "emSign ECC Root CA - G3" which is mentioned in the OP site as one you should blacklist, is present on my phone which is running Android 12 with the last security update from three years ago. Having a better timeframe for when the impacted certificates appeared really would have been nice.
None of the roots were compromised. (Some of) Their own sub-CAs for TLS certs, which are also cross-signed by Sectigo (since their own roots are rather recent), are affected.
How you have that root cert on your phone with a 3 year old CA set I have no idea, presumably something updated your CAs in the meantime? (Also, aren't you incredibly exposed to the general onslaught of vulnerabilities from the past 3 years?)
To be clear, emSign is another name for EMudhra, and Comodo is now known as Sectigo. (You can't just look at the organization name in a certificate subject; you have to consult the CCADB: https://www.agwa.name/blog/post/the_certificate_issuer_field...)
Inclusion in the Chrome Root Store requires that a CA Owner continuously demonstrate the technical competence and operational rigor necessary to protect Chrome users. Due to the seriousness of this incident, the Chrome Root Program is conducting a thorough evaluation regarding the status of eMudhra’s remaining CA certificate to ensure these standards continue to be met. We will share further updates as our review progresses.
Third party audits from a qualified auditor are required, although the CA chooses and pays the auditor, and the auditors are basically just accountants. So it's... not a perfect system.
> Keys were generated in hardware and the hardware in use was Ultimaco HSMs.
The plot thickens.
Broken HSM? Misconfiguration? Confusion? Straight up lie?
> Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault.
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
Reading further, it seems like they say they first generated some keys on-HSM, but then used 'test keys' (that were used to test wrap/unwrap of key material) instead by accident. Fishy.
They used the "correct" key for signing the CRLs (which of course was then an invalid signature):
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
And THAT is where I'm drawing the line between "honest mistake" and "gross negligence". For that to not raise giant red flags is an unacceptable failure in process and work culture.
> The other two certificates are for codesigning and trusted by Microsoft
That seems like a WAY bigger problem than TLS, but AFAIK Windows still validates code signing certs against CRLs before trusting them (can't remember if it soft-fails).
Why is that a way bigger problem? Windows will happily run unsigned binaries. Code signing just decreases the friction for users slightly, but is in no way required. TLS is definitely a way bigger problem.
> "The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys."
Ah yes.
"The system says something is wrong, but it's also 4.30PM. Let's just re-sign the thing and go home."
Definitely not. Their CPS requires that these keys be generated by an HSM and no HSM would ever generate these, so there's little reason to believe they're following any of their other controls.
My big fear about HSM is that they are inscrutable by design. There is no way to verify they are working as advertised. Plus side, all sorts of fancy hardware to protect your keys. Down side, all sorts of fancy hardware that obscures flaws in the implementation.
The most likely explanation is that they generated the weak key externally, imported it into the HSM, generated the CSR from it, and finally had Root CA sign it.
A FIPS HSM does not forbid one to import keys into it.
My vague and inexperienced read on their description boils down to "we were doing something weird and non-standard as part of our key ceremony because we intended these keys to be exported, and because of that process weirdness we blundered and gave the HSM faulty instructions."
I am shocked that one of the private keys was literally “4”. Anyone could have broken into this. I really hope there is an industry-wide effort to stop things like this in the future.... but I doubt it.
There is a variety of projects and researchers scanning for weak keys; possibly that's how this was found (haven't seen anything on that). These certs existed for a year and a half; that's bad but could be worse still.
One of the issues with sub-CAs (well, X509 certs in general) is that they can be ghosts in that there's no way you'll ever get it as a response on a TLS connection, so automated scanners on that won't find it.
CT should cover this (i.e. expose the cert so you can check for weak keys), but I have no idea if/how CT works for sub-CAs. [ed.: see below, doesn't seem to be in CT]
[ed.: these are technically sub-CAs, but rather cross-signed for bootstrapping a new root CA. Thanks @agwa for pointing that out.]
For their own stuff, Mozilla has talked in the past about specifically just only trusting intermediates they've seen. Their modern root agreement says the CA will disclose all relevant intermediates, so arguably they have a complete list and further if that CA "forgot" about a few of them whether that's because your firm is too disorganised ("Oh, right, the one on Barry's laptop, forgot that one") or an actual attempt to sidestep this rule - then we certainly don't want to trust these undisclosed intermediates, do we ? They're either purposefully shady, or they're forgotten about and likely never have been audited.
Nothing for "CN=EM DV TLS CA - G3B,O=eMudhra BV" (not sure which sha256 this uses, 2nd is the spki sha256)
I should be clear here my understanding is/was these are sub-CAs. Apparently they're (trying to be?) roots themselves? The point of logging a root cert in CT is somewhat moot… though tbh I would argue anything issued by a CA should be in CT logs, even cross-signing another CA that is becoming a root itself.
> One of the issues with sub-CAs (well, X509 certs in general) is that they
> can be ghosts in that there's no way you'll ever get it as a response
> on a TLS connection, so automated scanners on that won't find it.
Could you expand on this? I don't have a full grasp of what you're saying (I'm not super knowledgeable in CA stuff).
My understanding is that if something, like a browser, connects to a SSL site, they will either have or can obtain, one way or another, a complete chain all the way up to the root, so there can be no "ghost" certificate. And I would imagine a scanner could do all those same steps, and thus access all the same certs.
Is this a deficiency in scanner software? You seem to indicate it is a fundamental weakness in X509; I'm just trying to understand what.
Could you maybe describe a specific example of what you're describing?
If you're doing a MITM attack, you can selectively intercept certain clients, and only serve the compromised certificate in those cases only. Automated scanners won't pick that up. None of this actually works though, because browsers started mandating certificate transparency.
In the most extreme case, you can just issue a certificate and not use it anywhere. If it leaks, it's still a security issue, because it's valid.
More realistically, especially with sub-CAs, they might only be used on e.g. corporate networks. TLS scanners may not be able to reach any of the hosts that uses such a cert. As the sibling post points out, clients checking for the cert in Certificate Transparency logs counters that (adding a requirement that the cert be publicly known, which TLS/X509 by itself very much does not have) — but of course this only helps if your client does CT…
Ah okay, thanks. So to be concrete, something like automation running 'curl' on an internal network might see the full chain, including the weak cert, and be vulnerable, but scanners on the wider internet won't see it. I was confused by "...there's no way you'll ever get it as a response on a TLS connection..." — I guess you just meant a _public_ TLS connection. Or it could be otherwise installed on corporate machines' local cert bundles.
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
They don't blame the HSM.
Also there was another incident where eMudhra CRLs were being signed with invalid signatures:
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
> these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one)
This bit sets my hair on fire. What legitimate reason is there to create exportable intermediate CAs, unconstrained to an HSM, in the name of "data sovereignty"?
This is a really, really dumb way of doing online spying (presumably, you mean by running a program of deliberate misissuance) since every cert they issue is recorded in a public cryptographic ledger.
I'm not saying this will never happen, just that rolling out a dramatic infrastructure change for "data sovereignty" just to pull off a SIGINT heist that is almost certain to get your CA distrusted seems like a poor risk/reward calculation. I fully expect IC agencies to get some CAs killed and be fine with that.
Additionally, from the Mozilla researcher verifying the compromise:
> I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates."
The impact of this incident is very unevenly spread.
Over 10 years ago, Mozilla shipped OneCRL [1]. We remediated this incident in a couple of hours for Firefox users who were protected automatically without the need for updates. Chrome, Apple and Microsoft have similar tooling. Since then, we've built more advanced tooling to handle revocation at scale for leaf certs (CRLite [2]).
However, for the rest of the ecosystem, there has been little to no investment in revocation and for those clients, this is a massive problem. Hopefully, we'll see folks invest now. Efforts like upki [3] look promising.
Firefox Mobile 158.0 on Android 17 (Pixel 7) connects to all the demo sites with zero warnings. Any idea why the "automatic protection without updates" doesn't seem to be working with that combination?
Pixel 10 with 158.0.1 still connects to the emSign ones, even though the patch notes for 158.0.1 mention that it updates the CRL for those without 'Remote Improvements' enabled?
edit: Disabling all the eMudhra CAs in Android settings does nothing, cool?
edit: I guess there's no way to manually ask for CRL update, cool??
edit: Gave up and reinstalled Firefox. Seems to be fine now.
I'm hoping some organizations will step forward to fund work on this.
(Note that curl on Ubuntu 26.10 now has CRLite support baked via upki -- unfortunately that doesn't help with this because AIUI CRLite is only for end entities, not for intermediate certificates.)
It's very inconsistent for me even on just Firefox. On multiple systems running different instances of Firefox, checking with https://emudhra.we-have-a-problem.com/:
One blocks everything; one blocks 6 out of 12; one blocks the other 6 out of 12; one blocks nothing.
You mean the response was uneven. The impact was a complete and total potential compromise of literally everyone. Every domain. Every client using one of the public root programs. Every user of those clients.
I mean, it's an emergency, but the acute severity is debatable; many times a month the major browsers quietly deal with vulnerabilities that yield attackers direct control of victim computers; here, attackers get the ability to spoof websites if they're in a position to hijack connections.
The middlebox situation, with non-updatable roots, is very annoying. But middleboxes also have unfixable RCEs, so...
It's totally fair to evaluate different projects by how well positioned they are to update their WebPKI trust anchors. I'm just saying, keep perspective.
So it other words, it allowed attackers to do the one thing that Web PKI is supposed to prevent.
Being able to quickly update trust roots is at best a hack to paper over the issue when there is no good reason why every podunk CA should have authority to generate certificates for every domain in the first place. It's an inexcusable design flaw. Or a feature that gives power to root stores at the expense of everyones security, from the perspective of the root store operators.
Keep in mind that a _lot_ of software out there uses TLS as its only trust anchor for updates, commands or similar.
This can and will be weaponized for take-overs of systems. They will be of the consumer kind, but they will also be of the industry kind. Let's pray that they will not be of the state or military kind.
You mean you and Chrome stopped supporting normal CRLs and OCSP, replacing (for normal users) these distributed-trust technologies with central lists controlled by browsers
The screening process has been more involved in that since around the late 2000's. I wouldn't call 20 years "very recently," especially in the context of the history of the web.
Which is to say, sub-CAs in itself are a giant loophole that needs plugging. Especially since AFAIK nobody is bothering with name constraints (RFC5280 section 4.2.1.10).
They hold intermediate certificates issued by TWO different root CAs, neither of which they operate.
So two separate entities didn't bother to read their certificate policy or inspect their operations. Have we not learned anything from past mistakes?
Roots should be responsible for all their sub-CAs, full stop. If they don't trust them enough to accept that responsibility, then the system is broken.
eMudhra is also a root CA operator. One root CA operator cross-signing another root CA operator is a good thing actually, and it's what allowed Let's Encrypt to work before their root certificate was widely trusted.
Issuing sub CAs to non-root operators has historically been a big loophole though it's not as bad these days since multiple root programs now require approval before CAs can do it.
Didn't Let's Encrypt have to jump through all sorts of hoops to prove they were competent and trustworthy before that happened?
Now it's coming out these guys used "4" as a private key. That's a crazy amount of incompetence and malfeasance. Did they vibe-code a CA? HOW DOES THIS HAPPEN? I am dumbfounded.
You couldn't even do this if you installed Microsoft Small Business Server 2008 today, unpatched, with the CA on default settings.
The reason nobody bothers with NCs is because they don't work very well in practice. They also have their own baroque matching rules that are not intuitive if you're from the land of UNIX globs.
(In fairness this is a self-fulfilling prophecy: a lot of validators handle NCs poorly or not at all, so there's no impetus to rely on them. But that's not something we can really save in the Web PKI's current state.)
> But that's not something we can really save in the Web PKI's current state.
I agree with your assessment but it's not an excuse to give up without trying. I hope this is just because I'm not looking in the right places, but I don't see people trying.
What really makes me salty is that there was enough energy to mess with the rules around EKU, the primary effect of which is that non-browser software is now starting to just ignore the EKU bits. Even if it would've amounted to nothing, I'd really rather that effort had gone into NCs.
I think a lot of people have tried! The reason EKUs stuck where NCs haven’t is because EKU rules are simple (in relative terms) and OpenSSL didn’t make as much of a mess out of them.
I can't help but read your post as "OpenSSL didn't make a mess of EKUs so the CA/B had to fill that role instead" :/
(I really am salty about the EKUs. In a few years the EKU field will just be useless, due to operational annoyance driving devs to ignore it. Except in browsers of course.)
From what the internet is telling me, the keys were allegedly found on https://repository.emsign.com/certs/<redacted>.crt
Which, if true, is definitely a lot of fun.
Hope you people are ready for lots and lots of suddenly hackable IoT.
__
Edit: I might also be misinterpreting what I am seeing and it's actually weak logic that created the keys, in the sense that one could guess the private key?
Someone actually qualified please give credible answers.
I'm just pretending.
I think you’re right. And since the intermediate certs expire around 2039 and the private keys are known, we have plenty of smooth MitM fun against IOT in front of us.
> After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys.
Yeah that’s just oof. Not hard to imagine the kind of code error that could cause this, but hard to imagine that any serious vendor of cryptography hardware or software could make a mistake like this.
Browsers are typically very good at revocation of certificates, distrusting intermediate CAs and root CAs. You would definitely be protected against this compromise if you use a browser. Command line tools less so. Try this curl one liner
> I think it's worth noting that this compromise is much worse than the private keys simply being stolen by a researcher. They are inherently weak single-byte keys that anyone can factor, and the issue cannot be contained.
Why are the intermediates, and not the root cert, being revoked?
The intermediates seem to be difficult in that the root stores don't seem to have a mechanism for it (… kinda seems like an oversight, at this 20:20 vision point?). If eight of the CA's intermediates are hosed … how many are still "good", and are they good good, or are we just fooling ourselves?
The Go and Rustls upgrades specifically block the intermediate certs because neither of these use the system revocation lists, I believe. Once you upgrade to a trust store without the root cert for the CA this will also solve the issue. Not sure about other ecosystems.
Linux - the platform which offers no ability to revoke an intermediate certificate, thus necessitating every certificate library to implement its own blocking?
Windows is already blocking the compromised CAs as of today with no need for users to upgrade anything.
Every certificate library has to implement its own blocking no matter which platform you run on. Windows provides an API to do this at the system level, which means nothing if a given application vendors their own TLS library.
It's uncommon on Windows for applications to use a third party certificate verification library because Windows provides an API that does it. On Linux, you have no choice because all the platform provides is a bundle of PEM certificates.
> uncommon on Windows for applications to use a third party certificate verification library
I bet you have at least one copy of OpenSSL on your Windows system, probably multiple. Using the Windows native APIs is work. OpenSSL just works everywhere.
(Also, "the platform" doesn't technically even provide the cert bundle, depending on exact distro & packages.)
The Mozilla rep who created that thread (not the reporter, to be clear), created https://emudhra.we-have-a-problem.com/, so I suspect that this page is better for general public disclosure.
With that being said, that Bugzilla is the best thread to understand "how the fuck did this happen?"
Seems like the root cause here was human error during EMudhra's key generation ceremony process.
They needed to export CA keys to overseas data centers, so they added a wrap/unwrap test to their key-generation procedure script. Somehow that generated a second set of weak (test?) keys from the CryptoServer Se-Series Gen2 HSM.
Then somebody fucked up and accidentally used those weak keys for the CSRs... then their root and Sectigo signed it. And it shipped.
That's why the private keys are literally one byte.
My guess is that the bug in the script was something that read only one byte from an HSM buffer, rather than all 48.
In that case it was ssh sessions being intercepted.
But my question is if a tor exit node or a vpn provider or your internet provider has a private key to any CA then can they mitm any normal web traffic even though it will show up as https without warnings in your browser?
Just wrote up a quickie about how Nix + LLMs make patching everything super simple via overlays. Simplest security response I've ever done in my life. http://ghuntley.com/patch
If you're responsible for PHP client software, and waiting on composer/ca-bundle to release a new update, be aware that https://github.com/paragonie/certainty has been updated with the 2026-10-20 bundle already.
> An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.
And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
It's been obvious for a while now that root-of-trust schemes create juicy targets, and that juicy targets attract all sorts of bad behavior that wouldn't be economical if we used a web-of-trust.
Building a web-of-trust scheme that works might not be easy, but also it's not like we've been trying exceptionally hard lately either.
Web-of-trust doesn't work. Even before the PGP web-of-trust broke down several years ago, analyses of how the web-of-trust actually ended up working out showed that it basically boils back down to CAs: people rely on a few highly central keys to do most of the signing rather than signing keys themselves, and these keys effectively function like CAs.
The broader issue is that users don't want to (and probably don't have the ability to) actually make any trust decisions; they much prefer to delegate all of the decisions on trustworthiness to somebody else. The PKI root CA system does have its flaws, but on the whole, I suspect that people like Mozilla are better stewards as to whether or not the person vouching for somebody else is trustworthy than the average or even the above-average user.
Web of trust has been working between humans for over a million years. If we've failed to make computers do it, it's probably just for lack of trying.
To your second point, that might be true in a world where the consequences of making a bad trust decision was some kind of hassle related to changing passwords or reporting your cards stolen. But we seem to be moving towards a world where it lands you in a detention facility or some other dire consequence. I think that sooner or later, users are going to realize that they want that control and will be willing to do the corresponding trust hygiene work. Besides, isn't making that kind of work easy the point of technology? We just need to do better re: making it usable rather than investing in systems we know to be unsafe.
DANE replaces this system with one that has the same basic cryptographic properties, but in which you can't eject bad actors --- or even monitor them with a mandatory transparency log. It's strictly worse than this.
Maybe if people implemented name constraints and the CA/B forced CAs to use them. But not as it is; unconstrained sub-CAs are a scourge on the Web PKI.
I like constraints too, but listen carefully to EKR and 'agwa on this, because a lot of things that seem like straightforward fixes are much more complicated in reality. The most important thing I want from the WebPKI is expensive entry and the death penalty for fuckups, and I think we basically have that now.
It has the property of aligning cryptographic authentication with the domain name hierarchy. Yes, you trust the hierarchy. You trust LE too, so what? We need to move away from any authority being able to authenticate anyone, and DNS naturally aligns with the goal.
You've been vocal for a long time against DNSSEC deployment too for no discernable reason.
Why are you against every single thing that tries to make DNS useful for securing anything?
Do the words "no discernible reason" just mean something very different to you than they do to me? Because I've written somewhere in the vicinity of 100,000 words about the problems of DNSSEC over the last 20 years. My written DNSSEC critique predates HN itself.
This particular issue has nothing to do with "aligning authentication with the name hierarchy". The exact same thing that happened to EMudhra could happen to any TLD operator. The difference is that we have essentially no recourse when that happens, and far less ability to detect malfeasance.
People have debated DANE endlessly here, so I'm not sure how much more I have to add at this point. However, browsers, at least, do swiftly handle revocation for CAs, as indicated in this comment: https://news.ycombinator.com/item?id=50167244. They also handle EE revocation to varying extents, with Firefox having complete coverage and other browsers having partial coverage.
This is so blatently terrible, its hard to imagine they won't be permenantly removed from the CA ecosystem. I'm pretty sure other CA's have been removed for less.
Previously such incidents have lead to the CA business cessation by the involved entities (e.g. Diginotar case). I dont see how it would be different in this case.
they will have to file an incident report, which will be heavily scrutinized, and can possibly (edit: i am leaning towards likely, as more information comes to light) result in their ejection from most/all root stores.
trying to understand where this service is mainly used. judging by a quick google search ...
is this related to Government of India related websites and their services/websites like tax filing etc? is the impact expected to be concentrated around indian citizens availing these services?
It doesn't have to, you can set X509 name constraints which say "the CA may only issue/may not issue certificates for these Subjects," or you can set CAA in DNS, for example.
(Of course, the former isn't really intended to constrain trusted Root CAs, moreso internal PKIs. But it is an option people should know about!)
Wow, Thanks for digging up/working out that support list! Actually a glint of hope.
…but, yeah, the issue is on layer 9. Even just nailing sub-CAs to ccTLDs breaks the moment a marketing person registers a .lol domain.
Then again, the CA/B is essentially a layer 9 entity. I just refuse to believe this is an unsolvable problem (though it might need NCv2 with a whole bunch more shit to cover more human creativity in policymaking.)
Yes and NO. These are intermediate CA's and for them to be recognized by your browser or system, they need to be signed by another CA or root certificate that your system already trusts.
SO, not all CA's are signed by root certificates already on your machine
You can pin, but almost nobody does (and half the world that does pin is probably pinned to Let's Encrypt).
Those who manage the root domains should have been able to pin who they trust, and then if you WANTED to you could add someone beyond what .com or .egg or .muffin lists.
https://kagi.com/search?q=dns+CAA CAA is one way, but you can also pin the actual certificate or certificate chain (which has its downsides, mind you).
Web PKI is cursed. Why should anyone have to ship a new version of Go, Electron, webpki-roots, etc. (possibly over a now-compromised TLS connection!) just to revoke a certificate? Why are so many packages independently shipping high-sensitivity trust-bootstrap credentials? And why can any CA still vouch for any domain instead of each being limited to a specific ccTLD or other non-total shard of subject names?
Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.
Web PKI doesn't mandate that every client ship with its own root certs. The clients you listed can be configured to use a centrally managed trust store, if that's what you want. The fact that they don't by default is a deliberate tradeoff that the clients made, not some standard's fault.
This isn't, like, one misbehaving client. It's the way the ecosystem has evolved. GP didn't specify whether they were criticizing the standard or the ecosystem more broadly, but even if they were criticizing the standard, it's pretty fair to evaluate a standard at least partly based on how it gets implemented in practice. There's a reason cryptography tools for the last decade or more have been focused on being misuse-resistant.
There is no one ecosystem. The world of web clients is highly diverse. Expecting every client implementation out there to converge onto one particular trust store is, first of all, magical thinking, but second of all, it reintroduces the single point of failure that web PKI is meant to avoid. The OS trust store is just one of many possible trust stores, there's nothing ordained about it. The problem is not with some standard, it's with a mistaken pre-conceived notion that there must only be one.
Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.
I'm not making an argument for a single centrally-managed trust store. The thing that seems most obviously bad to me here, as a non-crypto-expert, is that any CA can issue a certificate for any domain. This is probably made worse because there are hundreds of CAs, and AFAICT functionally every trust store trusts all of the same ones.
I don't know how we fix this, but it does seem like a big problem.
Yes. I am criticizing the de facto web PKI, not the somewhat better design intent. The design intent is still bad though: we should never have given CAs universal issuance privileges.
The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.
For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
> The design intent is still bad though: we should never have given CAs universal issuance privileges.
This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.
I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."
DNSSEC key rollovers don't change who has custody of the keys. They're not a magic spell that fixes stuff like this. It is closer to the truth that DNSSEC is limited to recovery from incidents like this to doing rollovers, rather than immediate revocations. Rollover is not something that makes DANE more attractive here.
The private key for all 8 "compromised" CAs is a single byte: https://mastodon.catgirl.cloud/@49016/117474217042191790
One of the private keys is literally "4".
I'm putting "compromised" in quotes here, because I can't call it that without cringing and cursing.
[cf. also: https://news.ycombinator.com/item?id=50169109]
(But in any case, CT ensures that it can’t be “quiet.”)
The lack of CT entries only proves that (most likely) browsers were not MITM’d.
1. CT does not do anything if a state actor demands a CA in its borders generate a cert without publishing to the CT logs (or a three-letter-agency doing this illicitly). They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees.
2. Browsers won't check for SCTs in certs if they come from a private CA. you can MITM any domain through ZScaler all day long and nobody will notice
3. Wildcard certs make it easy to make/renew a cert that looks normal but hides requests on new/different hosts
4. Non-Browsers do not validate SCTs
I don’t think you can get the SCT from a log without the log publishing it? That would defeat the purpose.
“Certificates that support certificate transparency must include one or more signed certificate timestamps (SCTs), which is a promise from a log operator to include the certificate in their log within a maximum merge delay (MMD).[4][3] At some point within the maximum merge delay, the log operator adds the certificate to their log.”
> Because no TLS certificates chaining to these six intermediate CAs were logged to CT prior to our CRLSet block on October 17, no fraudulent certificates issued by these CAs could have been trusted by default in Chrome or other CT-enforcing browsers.
How you have that root cert on your phone with a 3 year old CA set I have no idea, presumably something updated your CAs in the meantime? (Also, aren't you incredibly exposed to the general onslaught of vulnerabilities from the past 3 years?)
https://groups.google.com/a/mozilla.org/group/dev-security-p...
Inclusion in the Chrome Root Store requires that a CA Owner continuously demonstrate the technical competence and operational rigor necessary to protect Chrome users. Due to the seriousness of this incident, the Chrome Root Program is conducting a thorough evaluation regarding the status of eMudhra’s remaining CA certificate to ensure these standards continue to be met. We will share further updates as our review progresses.
from a post on https://groups.google.com/a/mozilla.org/g/dev-security-polic...
Given Wayne's hints in the timeline that this issue was discovered by an agent, it might be one of the first big corporate death sentences due to AI?
I can't imagine one part of a business being on the up and up, and the guys down the hall running completely rogue.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
The plot thickens.
Broken HSM? Misconfiguration? Confusion? Straight up lie?
> Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault.
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
…
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
And THAT is where I'm drawing the line between "honest mistake" and "gross negligence". For that to not raise giant red flags is an unacceptable failure in process and work culture.
That seems like a WAY bigger problem than TLS, but AFAIK Windows still validates code signing certs against CRLs before trusting them (can't remember if it soft-fails).
This is absolutely false, it's a simple policy change to enforce code signing, any organization that is serious about security can do it.
I'll give you the benefit of the doubt and assume you're another Linux guy who doesn't understand Windows security.
The concern of course is with malware being signed, which is far more likely to be abused than some TLS MITM fantasy.
Ah yes.
"The system says something is wrong, but it's also 4.30PM. Let's just re-sign the thing and go home."
Good luck recovering from this.
https://i.sstatic.net/Y3zE3.gif
You trust the magic black box far more than I do.
My big fear about HSM is that they are inscrutable by design. There is no way to verify they are working as advertised. Plus side, all sorts of fancy hardware to protect your keys. Down side, all sorts of fancy hardware that obscures flaws in the implementation.
> The hardware used for each of the issuers is in the attached file. These HSMs are FIPS 140-2 Level 3 compliant.
A FIPS HSM does not forbid one to import keys into it.
Given that 4 is the value shown in this well known XKCD panel, it almost seems like a (im)practical joke or trolling.
One of the issues with sub-CAs (well, X509 certs in general) is that they can be ghosts in that there's no way you'll ever get it as a response on a TLS connection, so automated scanners on that won't find it.
CT should cover this (i.e. expose the cert so you can check for weak keys), but I have no idea if/how CT works for sub-CAs. [ed.: see below, doesn't seem to be in CT]
[ed.: these are technically sub-CAs, but rather cross-signed for bootstrapping a new root CA. Thanks @agwa for pointing that out.]
https://ctlogs.io/search?mode=fingerprint&q=42c4accd12c4f595...
https://ctlogs.dev/search?q=42C4ACCD12C4F59597925E1A08A7A901...
https://ctlogs.io/search?q=EBED6475C2C2D80FCB199F3FE8DD6E6B3...
https://ctlogs.dev/search?q=EBED6475C2C2D80FCB199F3FE8DD6E6B...
Nothing for "CN=EM DV TLS CA - G3B,O=eMudhra BV" (not sure which sha256 this uses, 2nd is the spki sha256)
I should be clear here my understanding is/was these are sub-CAs. Apparently they're (trying to be?) roots themselves? The point of logging a root cert in CT is somewhat moot… though tbh I would argue anything issued by a CA should be in CT logs, even cross-signing another CA that is becoming a root itself.
My understanding is that if something, like a browser, connects to a SSL site, they will either have or can obtain, one way or another, a complete chain all the way up to the root, so there can be no "ghost" certificate. And I would imagine a scanner could do all those same steps, and thus access all the same certs.
Is this a deficiency in scanner software? You seem to indicate it is a fundamental weakness in X509; I'm just trying to understand what.
Could you maybe describe a specific example of what you're describing?
More realistically, especially with sub-CAs, they might only be used on e.g. corporate networks. TLS scanners may not be able to reach any of the hosts that uses such a cert. As the sibling post points out, clients checking for the cert in Certificate Transparency logs counters that (adding a requirement that the cert be publicly known, which TLS/X509 by itself very much does not have) — but of course this only helps if your client does CT…
Particularly concerning is India's UIDAI https://lea.uidai.gov.in/
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
They don't blame the HSM.
Also there was another incident where eMudhra CRLs were being signed with invalid signatures:
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
This bit sets my hair on fire. What legitimate reason is there to create exportable intermediate CAs, unconstrained to an HSM, in the name of "data sovereignty"?
I'm not saying this will never happen, just that rolling out a dramatic infrastructure change for "data sovereignty" just to pull off a SIGINT heist that is almost certain to get your CA distrusted seems like a poor risk/reward calculation. I fully expect IC agencies to get some CAs killed and be fine with that.
> I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates."
Over 10 years ago, Mozilla shipped OneCRL [1]. We remediated this incident in a couple of hours for Firefox users who were protected automatically without the need for updates. Chrome, Apple and Microsoft have similar tooling. Since then, we've built more advanced tooling to handle revocation at scale for leaf certs (CRLite [2]).
However, for the rest of the ecosystem, there has been little to no investment in revocation and for those clients, this is a massive problem. Hopefully, we'll see folks invest now. Efforts like upki [3] look promising.
[1] https://blog.mozilla.org/security/2015/03/03/revoking-interm...
[2] https://hacks.mozilla.org/2025/08/crlite-fast-private-and-co...
[3] https://github.com/rustls/upki
edit: Disabling all the eMudhra CAs in Android settings does nothing, cool?
edit: I guess there's no way to manually ask for CRL update, cool??
edit: Gave up and reinstalled Firefox. Seems to be fine now.
I fully expected disabling them in the phone settings and rebooting would have fixed it, but no?
Don't really wanna reinstall FF and lose all my configruation.
Edit: however on Firefox Focus 157.0.1 they are blocked.
https://discourse.ubuntu.com/t/an-update-on-upki/77063
My interest in building OneCRL support into upki has definitely increased this week. Issue:
https://github.com/rustls/upki/issues/229
I'm hoping some organizations will step forward to fund work on this.
(Note that curl on Ubuntu 26.10 now has CRLite support baked via upki -- unfortunately that doesn't help with this because AIUI CRLite is only for end entities, not for intermediate certificates.)
One blocks everything; one blocks 6 out of 12; one blocks the other 6 out of 12; one blocks nothing.
The middlebox situation, with non-updatable roots, is very annoying. But middleboxes also have unfixable RCEs, so...
It's totally fair to evaluate different projects by how well positioned they are to update their WebPKI trust anchors. I'm just saying, keep perspective.
Being able to quickly update trust roots is at best a hack to paper over the issue when there is no good reason why every podunk CA should have authority to generate certificates for every domain in the first place. It's an inexcusable design flaw. Or a feature that gives power to root stores at the expense of everyones security, from the perspective of the root store operators.
Nobody thinks this is a good thing. It's bad. Hopefully, eMudhra gets distrusted. We've done that to bigger CAs.
This can and will be weaponized for take-overs of systems. They will be of the consumer kind, but they will also be of the industry kind. Let's pray that they will not be of the state or military kind.
It's cute you have a magic DIY CRL. You accepted these assholes into your root program.
They lost the KEYS.
What is the screening process for CA's that can't be bothered to use a HSM?
Even Honest Achmed wasn't this careless.
it's done in the open, and you're even allowed to voice your cute opinion during the multi-week public commentary periods.
here is an overview https://wiki.mozilla.org/CA/Root_Inclusion_Overview
Which is to say, sub-CAs in itself are a giant loophole that needs plugging. Especially since AFAIK nobody is bothering with name constraints (RFC5280 section 4.2.1.10).
They hold intermediate certificates issued by TWO different root CAs, neither of which they operate.
So two separate entities didn't bother to read their certificate policy or inspect their operations. Have we not learned anything from past mistakes?
Roots should be responsible for all their sub-CAs, full stop. If they don't trust them enough to accept that responsibility, then the system is broken.
Issuing sub CAs to non-root operators has historically been a big loophole though it's not as bad these days since multiple root programs now require approval before CAs can do it.
Now it's coming out these guys used "4" as a private key. That's a crazy amount of incompetence and malfeasance. Did they vibe-code a CA? HOW DOES THIS HAPPEN? I am dumbfounded.
You couldn't even do this if you installed Microsoft Small Business Server 2008 today, unpatched, with the CA on default settings.
Obviously the system failed here, but it's a failure of a root certificate operator, not a sub-CA operator as claimed by eqvinox.
(In fairness this is a self-fulfilling prophecy: a lot of validators handle NCs poorly or not at all, so there's no impetus to rely on them. But that's not something we can really save in the Web PKI's current state.)
I agree with your assessment but it's not an excuse to give up without trying. I hope this is just because I'm not looking in the right places, but I don't see people trying.
What really makes me salty is that there was enough energy to mess with the rules around EKU, the primary effect of which is that non-browser software is now starting to just ignore the EKU bits. Even if it would've amounted to nothing, I'd really rather that effort had gone into NCs.
(I really am salty about the EKUs. In a few years the EKU field will just be useless, due to operational annoyance driving devs to ignore it. Except in browsers of course.)
Which, if true, is definitely a lot of fun.
Hope you people are ready for lots and lots of suddenly hackable IoT.
__
Edit: I might also be misinterpreting what I am seeing and it's actually weak logic that created the keys, in the sense that one could guess the private key?
Someone actually qualified please give credible answers. I'm just pretending.
I was not expecting to find a silver lining _this bright_ but you have a point.
Apparently it's even worse
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
Also who hasn't left their private keys hanging around web-accessible, eh? lol
Private keys can be trivially recovered from the public keys because they're tiny. I found D=156, flüpke found D=4.
Yes, that's the entire private key.
4, 56, 66, 115, 155, 156, 230, 237
Yeah that’s just oof. Not hard to imagine the kind of code error that could cause this, but hard to imagine that any serious vendor of cryptography hardware or software could make a mistake like this.
curl https://example.org --connect-to example.org::2.29.126.86:
Thanks to mastodon user q3k@hackerspace.pl.
This thread (linked by the page) seems like a better source: https://groups.google.com/a/mozilla.org/g/dev-security-polic...
Ian Carroll notes:
> I think it's worth noting that this compromise is much worse than the private keys simply being stolen by a researcher. They are inherently weak single-byte keys that anyone can factor, and the issue cannot be contained.
The intermediates seem to be difficult in that the root stores don't seem to have a mechanism for it (… kinda seems like an oversight, at this 20:20 vision point?). If eight of the CA's intermediates are hosed … how many are still "good", and are they good good, or are we just fooling ourselves?
Rust with rustls? Upgrade to rustls-webpki 0.103.16 and/or webpki-roots/webpki-root-certs 1.0.10.
Python? Upgrade certifi to 2026.10.19.
Homebrew ca-certificates? 2026-09-25_1.
Electron? 44.9.1, 43.8.1, or 42.11.16.
Windows is already blocking the compromised CAs as of today with no need for users to upgrade anything.
I bet you have at least one copy of OpenSSL on your Windows system, probably multiple. Using the Windows native APIs is work. OpenSSL just works everywhere.
(Also, "the platform" doesn't technically even provide the cert bundle, depending on exact distro & packages.)
With that being said, that Bugzilla is the best thread to understand "how the fuck did this happen?"
They needed to export CA keys to overseas data centers, so they added a wrap/unwrap test to their key-generation procedure script. Somehow that generated a second set of weak (test?) keys from the CryptoServer Se-Series Gen2 HSM.
Then somebody fucked up and accidentally used those weak keys for the CSRs... then their root and Sectigo signed it. And it shipped.
That's why the private keys are literally one byte.
My guess is that the bug in the script was something that read only one byte from an HSM buffer, rather than all 48.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c11
In that case it was ssh sessions being intercepted.
But my question is if a tor exit node or a vpn provider or your internet provider has a private key to any CA then can they mitm any normal web traffic even though it will show up as https without warnings in your browser?
There is reason why company shouldn't cheap out and use foreign vendors like this....
> An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.
And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
You say this like it's easy and simple and we are simply not choosing to do so.
Building a web-of-trust scheme that works might not be easy, but also it's not like we've been trying exceptionally hard lately either.
The broader issue is that users don't want to (and probably don't have the ability to) actually make any trust decisions; they much prefer to delegate all of the decisions on trustworthiness to somebody else. The PKI root CA system does have its flaws, but on the whole, I suspect that people like Mozilla are better stewards as to whether or not the person vouching for somebody else is trustworthy than the average or even the above-average user.
To your second point, that might be true in a world where the consequences of making a bad trust decision was some kind of hassle related to changing passwords or reporting your cards stolen. But we seem to be moving towards a world where it lands you in a detention facility or some other dire consequence. I think that sooner or later, users are going to realize that they want that control and will be willing to do the corresponding trust hygiene work. Besides, isn't making that kind of work easy the point of technology? We just need to do better re: making it usable rather than investing in systems we know to be unsafe.
Also, the interpersonal web of trust is IMHO quite clearly failing in the age of mass media, and not just recently.
Well, yeah. DANE should have won. NameConstraints would be nice tooo, but who's going to make CAs use it?
And FFS, clients could at least check CRLs once in a while.
Maybe if people implemented name constraints and the CA/B forced CAs to use them. But not as it is; unconstrained sub-CAs are a scourge on the Web PKI.
You've been vocal for a long time against DNSSEC deployment too for no discernable reason.
Why are you against every single thing that tries to make DNS useful for securing anything?
This particular issue has nothing to do with "aligning authentication with the name hierarchy". The exact same thing that happened to EMudhra could happen to any TLD operator. The difference is that we have essentially no recourse when that happens, and far less ability to detect malfeasance.
https://www.nseindia.com/get-quote/equity/EMUDHRA/eMudhra-Li...
D=4. That's the entire private key for one of the certs.
is this related to Government of India related websites and their services/websites like tax filing etc? is the impact expected to be concentrated around indian citizens availing these services?
This means that the people with these keys could MITM almost any https connection people make, not only the ones that this CA issued.
(Of course, the former isn't really intended to constrain trusted Root CAs, moreso internal PKIs. But it is an option people should know about!)
From my limited experience (using them at home) they seem to be fairly well supported in client libraries, actually!
- OpenSSL has full support, https://docs.openssl.org/master/man5/x509v3_config/
- GnuTLS has full support, https://www.gnutls.org/manual/html_node/X509-certificate-API...
- go (crypto/x509) supports everything except DirectoryName constraints, https://pkg.go.dev/crypto/x509#Certificate.Verify
- Python (pyca/cryptography) has full support, https://cryptography.io/en/stable/x509/reference/#cryptograp...
- Windows/Schannel has full support, https://learn.microsoft.com/en-us/windows/win32/seccertenrol...
- Java has full support, https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
- I know that NSS also supports them from experience but (sorry!) couldn't find the docs in a few seconds of searching
- I know that Strongswan supports them well because I actually opened an issue about it a few years back and got it fixed :-)
(Just a few off the top of my head)
…but, yeah, the issue is on layer 9. Even just nailing sub-CAs to ccTLDs breaks the moment a marketing person registers a .lol domain.
Then again, the CA/B is essentially a layer 9 entity. I just refuse to believe this is an unsolvable problem (though it might need NCv2 with a whole bunch more shit to cover more human creativity in policymaking.)
SO, not all CA's are signed by root certificates already on your machine
This has always been incredibly stupid.
Those who manage the root domains should have been able to pin who they trust, and then if you WANTED to you could add someone beyond what .com or .egg or .muffin lists.
To be fair, unless I'm misunderstanding, wouldn't this be a good move?
LE has a good security record and takes itself seriously, versus these smaller CAs that seem to mess up regularly.
Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.
Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.
I don't know how we fix this, but it does seem like a big problem.
The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.
For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.
I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.
Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."
You mean until the DNSsec key rollover completes and they're no longer in that role?
As opposed to what we have now, an indefinite period where crappy non-updated devices will still accept these certs and nothing can be done about it?
The Web PKI works great for browsers. Everything else, not so much.
(To be fair, the crappy devices¹ would probably fail to deal with the periodic DNSsec root key rollovers. But at least that's a fail-closed.)
¹ a label that includes older versions of systemd-resolved, for whatever that may mean
Is the update cadence of CAs that different between the sources? Is the list of CAs included so different from OS vendors and the package managers?
Can you elaborate on this innovation in the email space around PKI?