Railway Security Disclosure
198 points by passenger 2 hours ago | 78 comments

vldszn 20 minutes ago
Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.

And it’s all just getting started imo.

https://blog.cloudflare.com/emudhra-certificate-compromise

https://www.wiz.io/blog/php2shell-php-rce

reply
netvarun 3 minutes ago
I think the pingcap hack might be the biggest of them all, somehow it has gone under the radar.

[0] https://news.ycombinator.com/item?id=50173716

reply
tom1337 13 minutes ago
reply
vldszn 9 minutes ago
Madness..
reply
bokchoi 8 minutes ago
And litellm too
reply
vldszn 5 minutes ago
true =(
reply
wunderwuzzi23 12 minutes ago
[dead]
reply
theturtletalks 2 hours ago
Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:

https://x.com/wickedguro/status/2113015432974405658

reply
lifeisloving 5 minutes ago
Why is a 300k mrr company (his tiwtter bio) running on raleway?
reply
fbrncci 47 minutes ago
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
reply
jazzyjackson 12 minutes ago
Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?
reply
jjulius 26 minutes ago
Prioritize your mental health, go enjoy your vacation.
reply
bryancoxwell 19 minutes ago
Strongly concur
reply
Liuser 12 minutes ago
Not your problem. Enjoy your vacation. Bus factor of 1 is poor planning of leadership.
reply
MattIPv4 2 hours ago
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
reply
numbsafari 15 minutes ago
Environment variables are a terrible place to store secrets.

I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.

The 12 Factor manifesto was 100% wrong on this point.

Use environment variables to point to where secrets are located, not to store their values.

reply
tptacek 4 minutes ago
Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.
reply
zaik 6 minutes ago
systemd makes it easy: https://systemd.io/CREDENTIALS/
reply
martinald 2 hours ago
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.

Fwiw it sounds similar to the hack over the weekend that opencode suffered.

reply
0x1F8B 28 minutes ago
The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.
reply
Fordec 50 minutes ago
I fully believe the only sustainable future is that hot patching will need to itself become fully automated. Hot patch now with a minimal allowed amount of line changes to head off the AIs, review it in office hours for auditing and the team is actually awake.

My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.

reply
munificent 48 minutes ago
> hot patching will need to itself become fully automated.

Great, even less control over the things we use and more power ceded to corporations.

reply
Fordec 44 minutes ago
Due to the small nature of hot patching (eg. bump this dependency, rotate these keys, turn off that config, follow the runbook) it will unlikely need to be a frontier model and is likely able to be done LocalLLM/Jev style decision making rather than having to hand all the keys to frontier model operators.
reply
stabbles 36 minutes ago
Hot patching is itself a liability
reply
Fordec 15 minutes ago
Not reacting is a liability. Blue-Green deployment is a liability. Code is itself a liability. Choosing this career at all is a liability.
reply
yakz 51 minutes ago
It's only just starting. It'll get worse before it gets better.
reply
reconnecting 60 minutes ago
I think the only way out is to go back to on-prem for everything, like in the 00s.

LAN cables will go up in price soon.

reply
seanw444 9 minutes ago
Ha, Cyberpunk 2077 is becoming reality. We're proposing the Blackwall.
reply
solyin 56 minutes ago
How does on-prem help if your product is a SaaS.
reply
lawrenceduk 42 minutes ago
You open a SaaS shop where people come to use it
reply
codeulike 38 minutes ago
Instead of being a SaaS it will need to just become a S
reply
80fj0348i 37 minutes ago
You put the SaaS into a Box your customers can install into their datacenter!
reply
doubled112 11 minutes ago
What a brave new concept!

Sometimes I do miss owning the hardware to run a thing.

What I don't miss was having all of those appliances run a different base distro of Linux, each with varying levels of update support, while the security team keeps reading security scans to me.

reply
reaperducer 12 minutes ago
Like back when you could rent a Google appliance.
reply
Sivart13 50 minutes ago
surely once there's been enough patches the code will be perfect though?
reply
arbayi 4 minutes ago
i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.
reply
fyzix 46 minutes ago
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
reply
frio 9 minutes ago
Yep. What an incredibly frustrating notice. I like Railway and I guess their hands are tied here, but wow.
reply
tossstone 33 minutes ago
I've already taken my applications offline and will be migrating to a new host...
reply
DrawTR 2 hours ago
Jeez, what a rough past 24 hours for all security folk. Hang in there
reply
llama052 29 minutes ago
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.

Then you have real security incidents that you need to tune into constantly.

reply
jacquesm 2 hours ago
It's going to be a rough decade.
reply
rectang 57 minutes ago
I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.

The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.

reply
aaron_m04 7 minutes ago
Business requirements will have to change. All your competitors are under the same pressure with the same shitty set of options, after all.
reply
jacquesm 28 minutes ago
I think it will kill open source to some degree. I found it already very hard to imagine how the endless queues of issues were being dealt with so diligently, but at least there was a human on the other side. The torrent that is unleashed now is not manageable on a human scale. It's a highly automated arms race with a bunch of good-willing humans caught in the crossfire between large companies and criminals duking it out.
reply
rectang 10 minutes ago
Could open source activity transition towards provably correct software, with far more modest feature sets?

I don't have experience with proof based tools, so I'm not sure what's feasible.

reply
cortesoft 56 minutes ago
It feels like the only solution is going to be proactively running your own AI agents looking for patches and applying them.
reply
rectang 50 minutes ago
The issue is where to direct the LLM's attention.

Prompting the LLM to "look for vulnerabilities in PHP's multipart upload handling code" is going to find stuff that "look for vulnerabilities in PHP" won't.

reply
cyberpunk 21 minutes ago
Or, you know, use client certificates so the bots can’t hit you.

Those can leak / be stolen of course, but a bunch of my clients php apis survived as without a cert they’re told to fuck right off.

It does make their client onboarding a bit harder but it’s a trade off which seems to have been the right one.

reply
jacquesm 26 minutes ago
Yes, so who will pay for that? Open source so far meant you would use your brains and your creativity, now you need to use your money too if you want to keep pace. From a 'guts, credits & glory' proposition to 'slop processor if you can afford it' is not really an upgrade.
reply
throwaway613746 58 minutes ago
[dead]
reply
merlindru 29 minutes ago
job security!
reply
sergiotapia 4 minutes ago
Actually I will move my projects away from Railway. With AI what it is today, no reason to shy away from hosting on actual VPS.

What is the best VPS to use today? Is it still hetzner or ovh?

reply
AJRF 60 minutes ago
Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?
reply
Aboutplants 24 minutes ago
Not a bad idea
reply
tom1337 7 minutes ago
It is… for their business.
reply
minraws 2 hours ago
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.

Like did someone vibe code too hard, how do env secrets leak for everyone...

reply
9cb14c1ec0 57 minutes ago
Easy, all the bad guys need to do is access whatever system stores the decryption keys.
reply
torben-friis 12 minutes ago
>We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.

Is that an accusation? As in, major AI players taking out competitors ?

reply
alpineman 60 minutes ago
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
reply
paxy 2 hours ago
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
reply
tokioyoyo 53 minutes ago
Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.

It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.

Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.

reply
nrmitchi 28 minutes ago
> It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!”

This isn't in the article? They do not at all claim "the attacked used AI!".

The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.

reply
arcanemachiner 2 hours ago
> your own red teaming should have found them before the attacker did

Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.

reply
gkoberger 60 minutes ago
Unless they edited the post, they never said this. You should delete this; it's false information.
reply
mbreese 49 minutes ago
I didn’t actually see that in this post at all. They mention that the end goal of the attackers might be to get to AI inference providers. But I don’t think they are crying “AI was used” here.

Or did I miss something?

reply
barrrrald 2 hours ago
This one doesn’t?
reply
shinycode 23 minutes ago
I’m lucky I moved away from it to host my own db with coolify
reply
dgellow 18 minutes ago
That’s coming for all platforms unfortunately
reply
eecks 50 minutes ago
Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36
reply
wateralien 2 hours ago
Is this our new AI daily reality?
reply
iririririr 56 minutes ago
companies probably have php and Python applications exposed to the web and nobody knows.

the speed of irresponsible deployments is infinite now, by design

reply
gbnwl 2 hours ago
The past two days have been awful as someone who uses Railway.
reply
goforbg 7 minutes ago
the worst service I've signed up for has to be railway. What a terrible product.
reply
sergiotapia 50 minutes ago
If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.
reply
lijok 2 hours ago
Damn I guess all those l33t hiring practices were for nothing
reply
rvz 49 minutes ago
Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.

Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.

reply
Keloran 2 hours ago
Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
reply
esseph 2 hours ago
Fuck, I thought this was about trains
reply