Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
The king was very careful to ensure all the gatekeepers were following the process correctly. It did not occur to anybody that one of the gatekeepers could be using a dirty rag to jam their gate closed instead of a lock.
Once discovered, the wall could be trivially bypassed. The security of the whole internet was immediately broken. Rebuilding the wall to block up the dodgy gate took much effort from very many people.
This system was invented in the 90s when things were looking up. China was opening up to the world, the Russians were suddenly not evil anymore, and the internet was supposed to bring us all together into one happy global society, kumbaya and all that. Obviously that didn't work out.
It's time for a rethink really. Every CA is now a single point of failure.
Citing Terry Pratchett’s “Interesting Times”.
This is why I went for a "wall with many gates" instead; any bypass makes all gates redundant.
All analogies break down at some level; the main goal is to clearly communicate the most important ideas.
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
However, Kings likely benefitted from the option of a scapegoat, they could always blame someone for forging their unforgeable seal and have them killed, rather than admit that the seal was true.
If we add in attainder and escheat, the King had a pretty strong hand.
As it turns out, these documents were mostly useful for long-lived décisions, which could be overturned, and where a forgery exposed you for a lifetime.
[1]: https://en.wikipedia.org/wiki/Robert_III_of_Artois#The_fraud...
for the non technical folks: imagine there was a password that if you guess it, you can impersonate *any website on the internet*
now imagine the password is literally the number 4- emudhra lawyer maybe (:
> "We set up a system where every government and a bunch of random banks and IT companies in the world holds the secret codes to everyone's encryption. Then we make them pinky swear not to abuse it, with stern talkings to if they do. And now it's all on fire, because one of them was irresponsible!"
You can know the intractability of the problem that PKI is solving, and the (worse) alternatives that have been tried, but that doesn't make this system not-insane to a normal person, and its at least a little bit the fault of the collective "us".
And it's not like it is an impossible problem; we rely on TOFU for SSH, we have remotely-attestable TPMs everywhere, ZKPPs / PAKEs exist, session keys and federation protocols are widespread. Somewhere in the set of "possible solutions" are options that don't sound unhinged when you explain how we got to this eMuhdra situation.
Then, everyone who trusts the CA will trust your certificate by checking it is signed by the CA who should have done the checks to ensure your certificate contains truthful data.
So your layman description of “they have secret codes to everyone’s encryption” is completely wrong.
A modern TLS connection uses an ephemeral key-exchange protocol where both parties opaquely agree to a one-time-use random encryption key that is not knowable to any other party, not even someone who can gain access to either party's private key. This is called Forward Secrecy (originally and sometimes still known as Perfect Forward Secrecy or PFS, but the use of the word "perfect" in any cryptographic context is akin to naming a ship "Unsinkable").
> random banks and IT companies in the world holds the secret codes to everyone's encryption.
That's not true at all.
What they do have is the power to MITM connections IF they first spook DNS or insert themselves into a proxy accepted by the server somehow.
So, if you want to be honest, the original sentence becomes:
> random banks and IT companies in the world holds secret codes that are trusted by everyone to sign information that is used to establish trusted communications. If they wanted to, they could try to tamper with people's network connections and, if successful somehow, pretend to be anyone else that had used their secret codes to sign their information.
Quite a big difference, no?
Your computer has a built-in list of signatures that it trusts and expects to see on the certificates, and one of those signatures was eMudhra’s. These signatures are basically just a proof that you know a secret number without revealing the number itself. If you know the secret number, you can sign the certificate.
There are an unimaginable number of secret numbers. Imagine each atom in the Great Pyramid of Giza is a secret number. And then imagine you have as many Great Pyramids of Giza as there are atoms in the universe. And then double it.
Now imagine you choose any of the numbers represented by all those atoms to sign the certificate with, and you chose 4.
It’s not technically wrong: 4 is indeed a number. But if anyone wants to guess your number, and they start at 0, it won’t take them very long to find your secret number.
Once they’ve found your secret number, they can now sign certificates pretending to be you. So someone could pretend to be your bank, and present a certificate saying “yes, I am your bank” with a signature on it that your computer trusts.
eMudhra sent you a padlock that had the code "4815" promising no one has ever used it or would be able to guess it, but all anyone really needed to try to get in is set the first number to "4" and ignore the rest.
It's not entirely accurate, but its accurate enough for them to get.
As a 90s kid, the Dodge Omni was a common shitbox student car. I think they had like 8 key patterns lol.
My brother’s ignition key unlocked my friends’s trunk. There were a few people who could start other peoples cars. In one case, a soccer player “stole” a football players unlocked car as they looked identical and were parked near each other. In another as a prank we unlocked cars to roll them out of parking spots and physically lift them into unusual places.
Yours is much funnier!
And when that clicks, extend the metaphor, literally, to a key with ~384 grooves.
I start by saying this is about the tech behind the S in HTTPS. I say that the main point of this is that when you talk to foo.com, it has to prove that the people behind foo.com are on the other end, so it's safe to send them your encrypted password or whatever.
Then I say a bit about how the system works by having a small number of trustworthy companies vouch for other people's servers. This doesn't need too much technical detail to get the point across.
Then I say one of those "trustworthy" companies put an encryption key of 4, where it's supposed to be a long number that's hard to crack. This makes the verification easy to forge.
Then I say that everybody who runs a server needs to remove this company's bad key, and Windows or Mac or your phone needs updates too. And that your web browser probably already quietly removed it, which removes a lot of the risk.
I feel like when I explain it this way people get it.
Mine was the lock company selling locks but also making freely available keys that would open any of their locks.
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
1. 4.7 per 100k people per year vs 288.5: https://en.wikipedia.org/wiki/Motor_vehicle_theft#Motor_vehi...
house collapses and people get injured.
concrete manufacture says we no longer build houses cause it’s too small of a business.
“That's the stupidest combination I've ever heard in my life! That's the kinda thing an idiot would have on his luggage!”
Then they locked it with the key you find on a chew toy for a baby.
I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.
This was a good read on the situation, thanks for the link.
Is the Mozilla bugtracker usually a / the Go to party for these issues? Or just a coincidence in this case?
It would need to be a 3, 5, or 7, since a prime is required.
Note that p,q (the primes you’re thinking of) are not actually required for decryption. They enable you to calculate d, the decryption exponent, and also enable decryption-time optimizations (exponentiation with d_p and d_q followed by CRT recombination), but that’s optional. In fact, p,q can be computed from n, e and d.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.
The "4" would be valid in hex, decimal, and even octal.
The key needs to mention which curve it is using.
Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts. "A rose by another name" and so on.
Your own comment has 3 typos and several grammatical errors ("prejudiced to" is wrong, for example). Irony.
Finally, modern usages of "racism" encapsulate all of these meanings. Languages change over time.
> You can be prejudiced to any group of people, doesnt mean that you are biggoted
Yes, that's literally what it means.
All that said, I make this comment as I believe the other commenter's worry is on display im this comment here...
> Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts
These are _not_ all equally bad. Prejudice is foundational to understanding, there is no escape from this. In this understanding, we should treat particular postures of prejudice more serious than collapsing any "bad prejudice" under the same label.
> I have these arguments with people I love who are right in spirit and still dig in the moment the word “racism” lands, because to them it names a thing they’re sure they aren’t
Like I said, it's because of how emotionally charged the word is. People think if they label unfairly discriminatory or prejudiced behavior as something other than racism, it's somehow not as bad as racist behavior.
They are wrong.
Do you think racism is worse than bigotry? Why? I understand that the first word has a more emotional charge in our times. But as far as "bad things to do" go they're equal in gravity. Racism is just a subset of bigotry.
> These are _not_ all equally bad
Why not?
> Prejudice is foundational to understanding
That's one definition of the word - "judging without knowing the facts".
But there's another definition - "an irrational hostile attitude, fear, or hatred towards a particular group, race, or religion" - which was the one we are operating with in this thread.
https://en.wikipedia.org/wiki/Endianness
I had a bunch of openwrt routers based on the ADM5120 SoC, some of them were of different endianness, totally wth:
But either way, I find the reduction of the argument to the semantics of the word racism to be deflection. Wouldn't you agree the original comment was made in poor taste? Why did it need to include a reference to Indian and imply that all things Indian made are cheap or low quality?
We applied this kind of mindset to the Japanese in the 60s and 70s, and look where that got us. I witness it happening again in the 00s and 10s applied to Korean electronics (LG and Samsung mostly) while we were content with minor improvements over the status quo and innovating far too slowly.
Yes, this incident has an almost farcical amount of stupid attached to it, but the fact that it happened to an Indian company is in my eyes only coincidental. There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features.
Curious which language is that, and if "racism" is an accurate translation to English. What you describe sounds more like xenophobia.
> Racism is a form of discrimination in which a person is judged on the basis of his or her ethnic origin, skin color, nationality or religion. It is a form of prejudice and inequality that prevents people from gaining equal opportunities.
While I'm happy to admit that perhaps my use of the word in English is incorrect and its meaning there is more narrow, in Dutch it covers nationality or religion as well. Islamophobia is considered racist ("racistisch") in Dutch, for example.
Let's take your words only.
Manual processes which are bug-prone and faulty are getting outsourced. They handle it with their best knowledge. Some processes/data knowledge is never transferred and some are kept hidden and assumed from outsourced teams but they have different culture and exposure so their understanding of things might be different. Its expectation mismatch not team issues.
American software giants are run by Indians only. So, if they are bug-prone & faulty then how could they reach there. It just that when person study with you or live with you then they have same set of culture & exposure like you have. So, they can manage things effectively.
This isn't what taking responsibility looks like.
> eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery and disclosing promptly
It doesn't end there. It ends when you've investigated how this happened, identified the causes, and changed your procedures, systems, and/or culture to make sure it doesn't happen again.
This seems like a weird statement given that anyone can issue certificates out of the affected CAs.
The more interesting targets will be various REST etc apis being called over https. Lots of people doing API clients have been ignoring the "what if someone tricks me to connect to an evil server" vector, because they assume TLS verification will protect them.
They just didn't mention that now anyone can be their own Certificate Authority.
Is 14 hours a diligent response time for something like this?
It was ~15hrs 30 minutes to first revocation
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
So they didn't even use their own certificates? :)
i'm not sure how much i should read into this, but if you're sending out warnings to people susceptible to ask chatgpt about it, don't say the company is Indian.
As I am writing, my iPhone 15 Pro is still accepting certificates signed by these compromised CAs. The device is "up to date", yet is still running Trust Store Version 2026073000. When I check at [2], certificates issued by 3 eMudhra Intermediate CAs are still being trusted!
Google Chrome revoked these certificates in 1 hour 40 minutes, Mozilla in just under 4 hours.
I am so astonished by how much Apple has failed here. There are potentially huge swathes of people completely oblivious that their phones are susceptible right now to being man-in-the-middle attacked.
Why does the iPhone not utilise a CRL list? Why does it take Apple 5+ days to do what Chrome did in under 2 hours.
It's a similar story for my MacBook, but at least there my Chrome browser _does_ use a CRL (the same can't be said for all the programs installed, however).
[1] https://fenko.nz/blog/emudhra-ca-compromise/#what-happened-i...
https://news.ycombinator.com/item?id=50189521
I suspect Apple published no update and is relying on the CA's CRL and/or OCSP to do its job. But it's rate-limited, the connections time out, and iOS is hard-coded to soft-fail.
Like you my trust store is months old.
That said, I found separate installs of Firefox and Microsoft Edge that accepted *all* the bad certificates without a care.
The only browser that consistently rejected all of them was — you guessed it — Internet Explorer. A browser so old it couldn't even run the test properly but it did what it needed to and blocked all the servers.
I am not smart enough to say how, but the sheer amount of work this created across the industry is monstrous. It probably has cost $billions to mitigate, and those mitigations have been slow and ineffective (see: my iPhone still being vulnerable).
In my experience, seemingly random pieces of software, libraries, and programming languages packaging their own CA stores has made this much harder to mitigate than simply running 'apt-get update ca-certificates'. We've practically just had to update every single thing to mitigate this.
(Even with all the related options disabled, e.g., ocspd on OSX still regularly connects to random certificate related services to "check" on things. Something related to random root certificates I don't have in the system trust store or such, Not entirely sure what this is about and how to get rid of the requests in any other way than blocking things with Little Snitch)
Report OE11044088569486, but currently it's just under review and invisible.
... 5 days later and I know better than to do online banking or messaging on my iPhone 15.
Just now on M1 Pro Safari.
On M1 Prp Firefox: none.
On iPhone 17: Safari: none, Firefox: none.
I just tried on Safari, it still trusted 2/14 compromised chains.
Rubbish.
edit: I went to keychain access app and found all emudhar and em signer certs -> there were four -> clicked get info for each -> under Trust section of the pop-ip -> selected "When using this certificate: never trust".
Now all checks on https://emudhra.we-have-a-problem.com pass for me.
Good that we are now investing thousands of engineering hours globally to fix this.
Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.
Sorry, it's been a long week.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.
My wife still plays her freemium games, the ads just don't load. She took out one of those games when we were sitting at the airport once. She was frustrated that her once-pleasant strategy game was suddenly crammed full of ads for boomer slot machine apps, and I had to explain that the app had probably always been that way.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
https://emudhra.com/en-us/leadership
---
The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
Isn't this what Go programs are to a certain extent?
Let's call it "Let's Decrypt!"
We've had to rework part of our application and certificate use because of this.
Now we are in a situation where we don't really have a practical and correct way to revoke certificates, but many, many things depend on pki.
No, this is not even remotely sensible.
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
If you don't know if you can hot-swap your certs, you probably can't.
I can see the case on phones: the OS may no longer be supported by the vendor, so browsers shipping their own set makes sense. And on embedded there isn't (and on IOT there might not be) an OS, but then you should probably just ship the the single CA that you use, not all of them.
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
Adding "out of the affected CA's" is top PR weasel wording that will slip by non-technical people who read this.
Congratulations to those who wrote or prompted this statement.
There are a lot of reassuring statements from browser makers and others about how their mitigations for exactly this scenario kicked in and protected (some set of) Internet users, but IMHO there is not nearly enough discussion happening about fixing things so those mitigations are not needed to begin with.
The problem is that those same people tend to disappear as soon as you ask "So what should we do instead?"
In the 90s vision of a truly globalised world that might have made sense but in today's geopolitical reality it's insane.
If this had been in place already, instead breaking the internet this would have been purely a regional issue.
Besides, there could be more CAs supporting .in of course. And it becomes a matter of responsibility to the local government.
And national TLDs aren't meant to be trendy anyway. Many have restrictions on them to local businesses anyway, it would make sense to have the local domain authority have control over choosing CAs as well.
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
We have been operating in India for a while. People do not take security, processes, etc., seriously in India. At least most of the organizations do not. Most likely, a human error caused the incident, and guardrails could have prevented it.
The sad part is that even after a major incident like this, the org is probably not going to invest in tools and processes to improve their posture. They will only do what is necessary to meet compliance. This is just the nature of a market driven by low margins and low-quality vendors.
There is no malice here by the org; it is just unfortunate.
It looks like them explaining that certs are only a small part of their business did not fly with shareholders.
Only a few major CAs in jurisdictions that admit heavy scrutiny and legal liability should operate. Country-specific CAs, if any, should only be scoped to signing certificates for their respective ccTLDs.
Also the industry has changed a lot since Let's Encrypt. You could have argued against monopolizing the CA issuance to just a few countries/entities as sovereignty issue, and CAs were indeed exhibiting extreme rent-seeking behavior before Let's Encrypt. With a guaranteed free and accessible alternative in the mix, the weight of the arguments against would change.
You could argue single point of failure, which is fair, so then set up another LE clone which is run in a similar way in another (reasonable) country.
(And there is of course non TLS certs, such as code signing and S/MIME, if that is still a thing. I'm only arguing about the TLS case here.)
Is there some reputation left to salvage here that I'm not aware of? Who would trust them?
But more importantly they don't need to any such name change. For people who matter in India it is just some random incident which Anti-Indian entities are whining about.
Does this mean "trust no indian CAs"? those "Anti-Indian" entities hold a lot of power, and the indian IT industry needs to maintain the trust it has with the rest of the world.
To think of reasoning may help them course correct is wishful thinking. Read about Indian pharma sector and what kind crappy, dangerous drugs they are peddling to world.
This is confident, resurgent India . Reason, competency are racist words if used agaist them.
The technical people will just install a certbot and use the default Let's Encrypt config.
https://emudhra.com/en/press-release/emudhra-to-exit-global-...
Popcorn recommended for the read!
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate "4" as a private key.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week (which you forgot to mention). All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
they should tear down the entire barn
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
What a weak minded statement. Just admit you messed up and call it a day.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
* https://en.wikipedia.org/wiki/Izzat
* Similar? https://en.wikipedia.org/wiki/Face_(sociological_concept)
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
A bad loser tries to save face. A good winner helps them.
The company we're discussing is being a bad loser.
Also, this is the response from official Indian Ministry of Electronics & IT:
…” However, some social media accounts are indulging in fear mongering about the impact of this incident.…”
Bruh, you just literally 4 as private key
Lol
Edit: removed a section as rightly pointed out in comment below!
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
Just the sheer size and diversity of the Indian "boomer class" could put such generalizations in doubt?
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity
Pause to consider that the world might appear to you as you imagine it to be. That is, it might be holding up a mirror in to your own psyche.
People like you are the ones make the world place to live in
Thus
People are the ones that make the world a place to live in.
Thanks for your insightful reflection.
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
It raises the question why these CAs agree to participate on this forum and then complain about doing so.
You apparently don't understand Indians.
(I presume that's a typo.)
e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
For a something this big they'd probably have execs making a visit to a certain forest.
Certainly can't speak for the rest of the world.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)
I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."
Probably just a botched translation though
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
A unit test becoming prod data? That’s really all I’ve got and that’s pretty unforgivable.
Some people just follow a spec and don't stop to think what that spec is supposed to accomplish.
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
Thousand line shell scripts that use expect to drive 30+ answers to a multi-choice interactive tui (that only works on one version of redhat from 2008 or something) is not so unusual, can totally understand how it happened. No excuses in that though.
But in all honesty there are serious questions to be asked about:
a) The initial approval process for ANY CA
b) The ongoing monitoring process for ANY CA
The fact this CA was based in India ultimately has nothing to do with it if we are being completely honest here.
Clearly there is a fundamental problem with the overarching approval and monitoring process which means it could have happened anywhere. This time it just happened to be India.
Secondly, in direct reference to the "far away company" comment India does have quite strict controls on both certain technical things (speaking generally here, not specifically in relation to certs ... e.g. use of sat phones). And in general it is quite difficult (read: almost impossible) to turn up in India and do business as an external company. So there is probably a genuine need for an Indian company CA since not every body can/wants to use Let's Encrypt or whatever.
Microsoft, in particular, had strong transparency around CA approval processes that were reasonable. It was weird.
Far away from the default country?
This is a sample on what it looks like during covid lockdown : https://bugzilla.mozilla.org/show_bug.cgi?id=1728790
https://security.googleblog.com/2014/07/maintaining-digital-...
Quoting from a news report [1]:
> The Ministry of Electronics and Information Technology (MeitY) on Wednesday said it has taken “serious note” of the reported security incident involving eMudhra’s Public Key Infrastructure (PKI) and has formed a special team to investigate the matter.
> The ministry also warned against what it called “fear mongering” on social media over the possible impact of the incident.
The company’s stock is down to the lower circuit on the stock exchanges for the last two days. This exit may be a decision to stem the beating that the stock has taken. I personally believe the stock will sink further, as it deserves to.
Trust once lost, especially in a shameful meme worthy way such as this, isn’t going to be gained back easily.
[1]: https://www.businesstoday.in/technology/story/impact-has-bee...
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
So the top still doesn't understand (it's not diligent), wouldn't you expect the other 99% of the business to be managed just as "diligently"
So far their responses to critical questions in the Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=2083449) have been rather weak and delayed.
went to keychain access app and found all emudhra and emsigner certs -> there were 4 -> clicked "get info" for each -> under Trust section of the pop-ip -> selected "When using this certificate: Never Trust".
Now all checks on https://emudhra.we-have-a-problem.com pass for me.
mitmproxy adds --emudhra option - https://news.ycombinator.com/item?id=50189232 - Oct 2026 (139 comments)
Your MCU likely escaped the eMudhra intermediate CA compromise - https://news.ycombinator.com/item?id=50183280 - Oct 2026 (26 comments)
Eight Bits of Trust: eMudhra's CA keys cracked in milliseconds - https://news.ycombinator.com/item?id=50177916 - Oct 2026 (445 comments)
eMudhra intermediate CA compromise - https://news.ycombinator.com/item?id=50166579 - Oct 2026 (491 comments)
Isn't the problem that anyone can/could issue end entity certificates with those certificates as "parent"?
This incident separately raises deep questions about the competency of the company to do much of anything in the security or compliance space.
Say, like a multi-scale hierarchical half popcount vector (first/second half, first quarter+third quarter/second quarter+fourth quarter down to odd/even bits), where fumbling one private key for another or xkcd-ing 4 as a private key would be detected, but still reveal no information on a truly random series of bits and be understandable by non-technical people in the room.
Apparently they generated them like this: https://xkcd.com/221/
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
It's already in the cab agreement that basically any unlimited intermediate you sign has to follow the same rules. And yeah, as a root in the program, if an intermediate you sign turns out bad, your root cert could get pulled, so you should be careful.
IIRC the Comodo AAA Services root was already in the process of being retired, with its "trust bit" having been removed by Mozilla and Chrome last year. Nuking it completely should have been a no-op for end users.
The whole chaining & cross-signing song-and-dance will never go away completely. There's just no way around if if you want enrollment of a new CA to take less than a decade, or if you want CAs to be able to do any form of non-world-breaking key rotation. Intermediates signed by third-party roots are already treated with the same scrutiny as root certs, so they should have been fine...
But yes, this will undoubtedly result in even stricter verification. For example, some participants are already calling for mandating attestation that a key has been generated inside a HSM running the right config.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
GOOD. RIDDANCE.
You cannot take these people serious. They should never ever touch something serious again.
The blast radius of that is every subdomain of whoever got compromised. Having only ICANN being capable of breaking the entire internet is a world of difference from what we have today, and may be the best we can do... and we should be using that, but it's not perfect either.
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
Of course we wouldn't want ICANN to do that on a whim, or without having a new operator for the TLD ready to go, but there's already process defined for both planned and emergency transfers of operation of a TLD:
https://www.icann.org/en/contracted-parties/registry-operato...
Which is exactly the kind of scenario a lot of people hyperventilated about back when ICANN was formed, for reasons of varying validity, but in this case it would indeed be the least-worst option all around.
ICANN is an organization headquartered in California ... I don't see how they can redelegate .com without approval of the feds.
IMHO, that isn't that compelling of an argument against domain hierarchy certificate issuance. If the feds want to mess with your .com today, they can and then they can get a cert from any CA. Adding a dependency on CAs doesn't remove the dependency on the TLD operator.
If you wanted to stick with x.509 certs and you had an ICANN root that would sign Name Constrained TLD intermediate(s), and the registry would sign name constrained Domain Intermediates, and all those require certificate transparency, then we'd be about where we are with WebPKI, but without trusting any parties you don't already have to trust.
You can disagree! But then: let's just keep the premises clear. You think DNSSEC makes sense because you also think states will lose control over their registry operators.
#1 Executive bonuses
#2 Shareholder value
...
...
--//--
...
...
#NaN customer trust
#NaN security
With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI),
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal.
Fuck these clowns, in India to get an import/export license you had get a hardware key via USB from them to access the government portal. Worse yet, their website is designed to look eerily similar to a government website and a lot of people won't even be able to tell it's a private entity.
Now imagine, all the customers who had purchased this key from them have their private details exposed. There was absolutely no mention of this anywhere. Not even as much as an apology. This letter is full of arrogance, deflection and insult to injury. I hope they burn to their ground. It's a really shame the government isn't even stepping in to do anything about this as their customers are the ones that are most impacted because they mandated the stupid hardware key requirement. What a joke of a situation.
In any other country, everyone involved would be fired and there would be serious consequences for such a level of breach (Eg. Equifax). In India, even a class action suit would lead nowhere and the fact that these assholes would be operational still despite this is extremely infuriating.
Translation: We are exiting this business because people are being mean, and we don't get paid enough.
WTF!
They also say: > We believe an organization's character is shown in how it responds when something goes wrong
I totally agree with this. I will judge their character based on this: No apology for wreaking havoc on the world, no admission of fault, whining about people getting pissed at the monumental incompetence, saying that they are exiting the business because "sour grapes" rather than admit they have lost the trust within the community. <sigh>
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
I will never trust any brand that even remotely sounds like this company.
"Oh sorry, we erased the evidence because we no longer had use for it"
> Ready to Try?
> Talk to our team about how eMudhra can help secure your digital workflows with PKI, eSignatures and identity solutions.
> Connect with sales
From everything that came out they seem terribly unqualified.
This is problem with Indian companies, their founder and management always try to hire cheap labor and penny pinch on everything. And, this is the result we see again and again.
it doesn't matter that they left the business, it matters that _they_ continue to be investigated
Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 They're being asked some very pointed questions about why they encountered problems with the key generation process at the time but didn't properly investigate.
Don't these people pay any attention to politics?
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
First one I clicked on?
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
For example, node, java, and python all have vendored roots and don't check any CRLs.
Well, what about them? If you've got access to the private key, nothing is going to stop you from generating a cert violating CAA records - and it won't even be easy to detect! They are intended to protect against, say, someone hacking into your server, running ACME to get a cert, cleaning up their traces, and then intercepting your traffic for 90 days. CAA records assume the CA is trustworthy and following the rules.
There used to be a pinning scheme called HPKP, but the trouble with it is it's very easy to accidentally (or via malicious third-party action) end up in a situation where your web site becomes unreachable by people who've visited it previously because the key changed and the pin prevents their browser recognizing the new key as valid:
My trust in those vetting the CAs is completely broken now.
You might have heard of DigiCert or IdenTrust, but what, exactly, makes them trustworthy, and why eMudhra wouldn't be considered trustworthy (a week ago)?
That’s sort of the problem the CA forum (badly) addresses. The consumers of the CAs are not specifically exposed to failures of the CA they choose.
this lets them exit this without public humiliation of getting made fun (and example of) in the mailing lists.
Analog
sure eMudhra needs to be investigated and confirmed that even a small part of their business hadn't been used to compromise (it doesn't matter that they left the business, it matters that the investigation continues)
but also we need to see how it is this even happened, who dropped the ball and approved the bad code/behavior to enter in the first place??
or is this a structural issue where we need to introduce more checks/balances (tests)
Surely they should be investigated by the local regulator.
https://www.deccanherald.com/india/meity-forms-team-to-probe...
By signing eMudhra's CA, sectigo put their own responsability on the table
And by doing so without sufficient checking of the signee's quality, they prove to the world that they shall not be trusted
~ Not a quote from the press release
You can't vibe code your way out of trust after causing a $100M dollar stock price crash over the silliness of how weak their root CA private key was.
I think that's for the best; there are much bigger problems with DANE/DNSSEC than TLS reliability.
Fortunately, the WebPKI has been ratcheting (slowly) up security over the last 15 years, so when nothing else happens, which is what I predict, at least we'll be in a better place a decade from now.
The reason we don't have DANE support in browsers is of course because Google decided not to implement it in Chrome, because (ultimately) the key lengths securing the root DNS server zones weren't large enough to suit them (at the time, 1024 bits):
https://www.imperialviolet.org/2015/01/17/notdane.html
And if Google wasn't going to do it, it didn't make sense for any other browser to do it either.
However, I believe the root zone keys are now longer, so I could (potentially) see Google implementing DANE in Chrome now, maybe with a Chrome-specific restriction of "TLSA records that have keys under 2048 bits in their chain of trust are ignored" (since the root zone keys are not the only keys in the chain and they want to eliminate 1024-bit RSA from Chrome). And if Google does it in Chrome, then everybody else has to do it too, so the situation flips.
More to the point I would trust IANA and ICANN, faults and all, to not screw up the DNS/DNSSEC root, much more than I trust every CA in the default browser trust stores to get X.509 right all day, every day.
(Of course it's fine to use AI tools! We use them heavily ourselves. Just please don't use them to generate or edit text to be read by fellow humans on Hacker News itself.)
It's impossible to know for sure what was LLM generated or edited, but we have to rely on imperfect classifiers or else the site would be completely flooded by genai text, which is not what the community here wants to read, nor in the intended spirit of HN.
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
I wish any company that gets sunk by this a very happy go-out-of-business.
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps.
source:
What's the eMudhra security scare that has rattled the internet?
https://theprint.in/tech/whats-the-emudhra-security-scare-th...
If they go to the trouble of wrapping them, use real keys. Or don't do it.
Also, it's a big "human error" to not notice this for a year and a half.
It is more likely they had a script which placed a parameter in the wrong field, due to something like using positional arguments. Or they misread the documentation for some interface.
Still incompetence. And I would love to see my theory confirmed or refuted.
Generating a key with "4" was probably done as a joke, but also is insane to have ever lived outside some devs computer.
[1] https://www.youtube.com/watch?v=hXUqwuzcGeU
As a consequence, these businesses are very well protected from normal market forces and are pretty much above the law. Had this incident not been an international one, they had a very good chance of getting away with it, and the people reporting this would have landed in jail.
Would you say the system forces people to be like that or people are the reason why the system is like that?
Would you do business (i.e. trust) with someone that comes from that system?
When working with different businesses across the globe, I’ve noticed Indian correspondence to incidents tends to be more a show of strength even if it’s transparently obvious that there’s no merit to the content.
Whereas western European correspondence is often more apologetic and empathetic. Even if it might sound insincere to those affected.
But it’s all just words at the end of the day. It never means much unless the business is willing to back real change to mitigate similar issues in the future and resolve undisclosed existing problems. However there’s no way for us to know if this is happening or not.
"We admit no wrongdoing. We are only settling to move past this. Our business is strong/secure/competent."
It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care.
Yeah, the proportion if anything is too small. This is the sort of company that should not be doing anything related to security.
Waiting for a release like "... the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP 1FA place in our business. In view of this, after detailed deliberations our board has concluded that withdrawing from global OTP 1FA is the right decision for eMudhra and its shareholders"
At what point does it become Russian roulette with a Glock?
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
It’s get worse when these companies work with regulators to create more regulation that they can sell more of their services.
https://www.amazon.co.uk/Compliance-Industrial-Complex-Opera...
Companies already control all the traffic flow on their infrastructure using end-point security solutions, the same can be extended at ISP level, and then Country level
Typically they do this for non pay but they've dropped the hammer multiple other times incorrectly (and good luck getting a RFO out of anyone there).
Only when fully satisfied they'd have given license to operate this security business.
>According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla).
https://www.fairssl.se/en/articles/emudhra-ca-noegler-knaekk...
If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place.
As a trust provider you get one chance. Squander it and you're out. That is how it should be.
In any legal dispute involving them, Exhibit A will be "look at how they ran their SSL/TLS business". That won't go well.
Unsurprising if you are familiar with the indian honor code „izzat“. To quote an old kiwifarms thread:
Hi. I didn't see this term mentioned in the thread, so I wanted to share this all with you. It's about Izzat. Some posts have mentioned 'saving face', which is Izzat, but I wanted to elaborate on it.
Izzat has no direct translation into English. We only have terms that can broach the same concept such as 'honour' or 'reputation' or 'face'. Izzat is so much more than that. It's a zero-sum game of collective honour shared by whole groups of people, all of whom take it very, very seriously. A system like this isn't just foreign to Enlightenment values, but I'd argue it's antithetical to every sensible form of governance on the planet. It will destroy any system that assumes good faith.
Izzat conflicts are not about who is right and who is wrong. It's about who wins and who loses. This means it's a zero-sum game where just about any action is justified (including murder) to restore the lost Izzat. Izzat is a limited social currency and the easiest way to get it is to take it from someone else. Winning is righteous in Izzat. Losing is unrighteous. This means that if someone plays the game of izzat well enough, they can get away with just about anything (murder, rape, scamming, cheating, stealing). The only morality in Izzat is the protection of your group's collective ego. The only appropriate response when your Izzat is attacked is the complete destruction of whoever insulted it.
Two people get into an argument. They might escalate, but chances are it won't be that bad. i.e. 'Sorry I broke your garden gnome.' With two Indians in an argument, the stakes are always deadly thanks to Izzat. Neither of them can back down, nor can they admit fault. Admitting fault is seen as deliberately humiliating yourself. Not only that, but because Izzat is shared, you are shaming everyone who shares your Izzat. So, admitting fault or taking responsibility for a problem is a form of social suicide. This means even if the dispute was over something completely fucking stupid or trivial (like a broken garden gnome), it could spiral into a decades long honour feud. Izzat is also the reason why police are reluctant to get involved in disputes. Because Izzat is a zero-sum game, all participants are involved in the game. If a policeman sides with one family over another in an Izzat dispute, this means that he's deliberately taking the side of that family and dishonouring the other. This marks him for retribution by the offending party. Without parties being able to be impartial, then centralized authority cannot effectively function.
Let's say someone in India complains about a broken water pipe. Instead of the problem being addressed, the official responsible for the water pipe denies it's a problem and counterattacks him instead, because daring to question his efficacy in his role was challenging his Izzat. So the official destroys the person who brought the problem up. The water pipe never gets fixed.
Izzat is also the reason why Indian managers are so infamous for hiring more Indians. It's because from the manager's perspective, he's using his position to gain an invisible social currency. Merit and actual qualifications come second to that idea. If he hired a westerner, he would not gain or even lose Izzat by doing so. He has a very strong cultural incentive not to be impartial. Meanwhile, if he can strong-arm dozens of Indians into a company, he is gaining huge quantities of Izzat and conspirators who owe an absolute debt to him.
It's not uncommon to see Indians gloat about their success in the west. And yes, izzat is very much a system that enables short term success. But the fundamental reality is the prosperity that these Indians find so attractive in the first place wouldn't exist if the west practiced something similar to Izzat. Our systems can only exist on the assumption of good faith, and not a majority of people exploiting them for destructive short term gain. On top of that, if an Indian causes the systems and companies he comes into contact with to collapse, then he can just go back to India with his plunder. He has no stakes in the long-term prosperity, functionality or stability of these systems. The stakes are completely asymmetrical in the Indian's favour.
You all know about that infamous video of that Indian scamming a food bank. Once again, winning is righteous. Losing is not. By employing 'clever' means (jugaad), he successfully extracted more resources for himself and his family. From the perspective of jugaad, a rule is not something to respect, but merely an obstacle to the Indian's own gain. Since he cheated the system and wasn't caught, he is seen as a righteous and dignified man in the Izzat framework.
I read a story in India about a man offering to pay another man to use a public toilet instead of defecating openly. The latter left and came back with several friends and beat the former to death. The problem was not that he was defecating openly, but that he was criticised for doing so. And with Izzat, it's not an eye-for-an-eye. Izzat is often an eye-for-a-whole-head. Disproportionately and brutally annihilating your enemies is the correct move to make because you take back your lost Izzat and then some. This kind of vindictiveness would be rightly seen as horrifying and disgusting in most of the world.