Stop MitM on the first SSH connection, on any VPS or cloud provider
77 points by JoachimSchipper 3 days ago | 45 comments

theteapot 3 hours ago
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution.

Maybe I'm missing something but SSH already has a built-in solution for this, key-certs. Just sign the server key with a private CA key you trust.

reply
londons_explore 7 hours ago
A big class of attacker is nation state attackers who do not want to risk discovery.

A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.

For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.

Suddenly, nation state attackers won't be interested in MITM-ing at all.

Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??

reply
hnlmorg 7 hours ago
Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?

If so, the legitimate server wouldn’t have anything in their logs that would help detect such an attack.

OpenSSH does log other telemetry though.

reply
flumpcakes 6 hours ago
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?

Fingerprints are derived from the certificates/private keys. Unless I don't understand some basic crypto, or SSH works in some obtuse way, I do not think it would be possible for the MITM attacker to present the server with the true client's fingerprint unless they also had had the client's private key.

reply
hnlmorg 6 hours ago
Ah thanks for the explanation. It’s been a long long time since I’ve delved this deeply into the topic.
reply
cesarb 4 hours ago
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?

The client sends not only the public key, but also a signature, and that signature depends on the output from the key exchange, so it's "bound" to the shared keys negotiated between the client and the server. If the MITM server does separate key exchanges with the client (pretending to be the real server) and the server (pretending to be the real client), the signature won't match; if it forwards the key exchange between the real client and the real server, it won't be able to decrypt the packets.

That's the best thing about SSH public key authentication (and HTTPS client certificates): even when MITM can impersonate the server to the client (because the client didn't verify the host key), it can't impersonate the client to the real server.

reply
mbxy 3 hours ago
Let me see if I understand correctly: Client takes its own public key and the server's public key and creates this signature.

MITM can take its public key and the client's public key and send the resulting signature to the server instead of forwarding what it received from the client.

Do pretty much the same exact thing: MITM PK + Server's PK -> Client. Now client has a signature as well. The signatures that client and server have are different but that is OK as long as MITM can see and change all communication.

It has been a while since I went through the details of the protocol, so I must be missing something. What is it?

reply
iberator 2 hours ago
how would you create REAL write only logs?

syslog > /dev/lpt0 printer?

reply
daveguy 5 hours ago
Because log processing is handled in the kernel/root/system? Is this a trick question?

See also: rsyslogd

reply
dboreham 9 minutes ago
I think the idea is the attacker didn't compromise both the local machine and the remote log sink machine. If you want to get really fancy the techniques used in cert revocation logs/blockchains could be used.
reply
sieabahlpark 5 hours ago
[dead]
reply
kayson 2 hours ago
I've been fighting against this chicken and egg problem in my homelab. I have a step CA SSH CA set up along with automated deployment, bootstrapping, and cert renewals with an ansible playbook. It sets StrictHostKeyChecking yes in the ssh_config for my domain so I'm happily protected against a very unlikely attacker that has a foothold in my network. Theoretically, it means I should never have to worry about host keys again.

Unfortunately, ansible does everything over.... SSH. So if I spin up a new VM or host, I have to manually trust the certificate for the first connection, which is the whole point of this article. I always have console access so I can log in and check the pub key.

There are various ways around this, including the authors suggestion with cloud init. None are very helpful for new physical hosts, though. I'm leaning towards a feature step CA supports that lets hosts authenticate themselves with an X509 cert, which you can easily get with ACME. It's so easy that you could even do it over console on a new physical host.

What I really wish is that there were something like the ACME TLS-ALPN challenge but for SSH servers. They can already present a self signed certificate, so it would just be a matter of connecting all the plumbing.

reply
denkmoon 55 minutes ago
Though the name is cloud-init, you can cloud-init physical hosts. Seems excessive though, how often are you adding new physical hosts to the homelab?
reply
washingupliquid 4 hours ago
I'm supposed to believe MitM with the same exact keypair is somehow possible? Private keys are never exchanged. Did everybody forget how crypto works?

Yes you implicitly trust the public key on first login.... then just... immediately compare it with what's on your box?

Might as well seal your doors with duct tape to prevent ghosts from entering your home because this is equally effective.

reply
tptacek 2 hours ago
The author of this post understands that private keys are never exchanged. Read it more carefully.
reply
leni536 3 hours ago
How do you compare? What trusted channel do you use to retrieve the real public key?
reply
washingupliquid 2 hours ago
Public keys go over untrusted channels. That's why they're public.

I'm not confident you understand how crypto works.

You do realize the entire threat model here is a house of cards perched atop someone else's software hosted on someone else's hardware all of which you implicitly trust and discard in favor of some unlikely cloak and dagger interception scheme.

reply
comex 2 hours ago
Public keys can go over channels that an attacker can read. They cannot go over channels that an attacker can modify. (Which would include the SSH connection itself, until such time as you’ve verified the key through a trustworthy channel.)
reply
washingupliquid 2 hours ago
A public key is useless without the private key. Which the attacker in this unlikely scenario doesn't have.

So you login the first time and they either match, or they don't. If they don't you start over. The end.

Ignore the fact that most people will probably use the box to host a poorly coded vulnerable service anyway.

reply
asalahli 6 hours ago
It baffles me why VPS providers don't display the fingerprints on their dashboards or expose them through the API.
reply
dspillett 3 hours ago
A few do. Though a more common way of securing the loop is to take the desired user public key during the build process.

Assuming you trust that the host control panel (or API server) hasn't been hacked, which you are assuming anyway if you trust a host fingerprint given to you that way, that should be as secure. For a small bit of extra assurance, to protect from an extra very unlikely attack, generate a new key pair just for this VM's creation so that you know you aren't connecting to some other VM that happens to have a known public key of yours and you've been redirected to by DNS poisoning.

reply
comex 2 hours ago
AFAIK this is not secure. The SSH protocol doesn’t require the server to know the client’s public key for the connection to succeed. Instead, the client actively sends its public key to the server while authenticating, and the server just decides whether to allow a connection or not. OpenSSH will typically match the client’s key against a fixed list of keys, but nothing stops a rogue server from allowing connections from any client key. In fact, OpenSSH itself can be configured to do this using AuthorizedKeysCommand.
reply
INTPenis 7 hours ago
The author essentially bootstraps their servers with a known trusted host key, so that first connection is recognized, instead of having to trust a new and recently generated host key when you first connect.

It's a neat little trick if you're often deploying VPS in shared cloud environments.

reply
flumpcakes 6 hours ago
This is something I have struggled with:

How to deploy secrets during bootstrap to a new virtual machine running in the Cloud that does not leave a trace in the infrastructure. And in a way that I can completely automate the deployment.

One answer is providing the secrets in cloudinit - but this leaves a trail on the host/provider's infrastructure, I do not know if those configs I paste into the portal then get saved off somewhere.

The other option (more secure) is having the keys/secrets generated on the host itself at first boot. But then this is difficult to automate as I would need to scrap them (even just the public parts) in a secure way. One option would be to have the public keys printed to the terminal/VNC - but this is much more trouble than it is worth to automate.

I'm not sure on a good solution. This is taking quite an adversarial security model though, assuming the host/provider is not completely trustworthy. Of course not owning the hardware means that the host/provider could be performing other attacks without my knowledge (copying memory, etc.)

reply
unsnap_biceps 6 hours ago
I've been meaning to investigate Nitro Enclaves[1] for exactly this but haven't made the time yet. Have you looked into them and found them lacking?

[1] https://aws.amazon.com/ec2/nitro/nitro-enclaves/

reply
IgorPartola 6 hours ago
1. Use cloudinit but give it a one time password to download the secrets on first boot.

2. Use certificates and your own CA.

3. Use the virtual serial console for first login.

4. Use cloudinit to add a custom software repo, then use that to install a custom package that does the initial work.

reply
anygivnthursday 4 hours ago
You can bootstrap from your custom ISO with some embedded starter key, upload ISO, loads into RAM and opens SSH, connect and run a playbook to encrypt the root drive where you deploy your OS with your SSH key. If you went with encrypted root, you might need to enter pass via console, or dropbear server in a pre-boot env you connect to via SSH to enter the key if you dont trust the console, or setup some custom network unlock mechanism, etc. But once unlocked your provider can still dump your keys from memory. There are also things like AMD SEV-SNP for some more confidential use cases.
reply
athrowaway3z 4 hours ago
So this is a tangent on a thought i had after reading the title, but it might be a cool idea that I'll not have the time to do anything with so feel free to use it:

Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.

But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.

You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.

reply
lvlabguy 6 hours ago
MitM is not possible if one uses public key authentication.
reply
gruez 6 hours ago
I was about to downvote this for being obviously false, but after some research this does appear to be true, because ssh uses some channel binding mechanism to prevent your public key authentication from being replayed/reused by the "man" in the middle.
reply
ekr____ 5 hours ago
This is one of those situations where it's necessary to be very precise about the security properties.

Specifically, if you bind authentication to the connection, then an attacker who impersonates the server (in this case because it's the first connection, but in other settings because they have a fake certificate), then client authentication is not portable to another connection, so the attacker can't mount a classic MITM attack. However -- and this is a big however -- that doesn't mean that there aren't serious security problems. For example:

* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.

* If you download some file (e.g., a script) from the server and then trust it, the attacker can use that to provide a malicious script.

reply
gruez 5 hours ago
>* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.

That's much harder to pull off though, because you need to replicate the environment close enough so that the victim doesn't suspect anything. Do they put their config files in /var/lib or random docker volumes? Do they use docker compose or docker-compose, etc.

reply
ekr____ 5 hours ago
Sure. I'm not saying it's not better to use public key authentication (it is!). Just that it's still possible to have problems.
reply
lvlabguy 2 hours ago
Basically, the client signs the shared key obtained through Diffie-Hellman key exchange, which then gets verified by the server. This ensures that the client and the server have the same shared key, hence no man-in-the-middle.
reply
crypt0r84 7 hours ago
provision the hosts with an SSH CA, use the CA as a trust root in openssh. they are various version out there from the big players.
reply
IgorPartola 6 hours ago
There was ages ago a project called monkeysphere that let you sign the host ssh key with your gpg key and verify it automatically. The downside was that it was very slow.
reply
TacticalCoder 2 hours ago
> inject a temporary SSH host (private) key via cloud-init...

Reading the comments here I'm tempted to believe that if cloud-init is available and if we consider Heztner (and OVH etc.) provides a secure access to cloud-init (i.e. the box running cloud-init is really the box you think it is), then there are many different ways to solve this problem.

reply
zuzululu 6 hours ago
but how real is this threat? you buy a box from hetzner or some VPS provider and its rooted when you try to ssh ?
reply
tptacek 2 hours ago
If you don't assume the network between you and Hetzner is compromised, you might as well just use rsh instead of ssh.
reply
projektfu 5 hours ago
You get a box and someone is sitting on its IP address (in the middle), proxying to the real one, so everything is getting logged. Other comments say that this mitm stops working when you use public key authentication.
reply
cesarb 4 hours ago
> Other comments say that this mitm stops working when you use public key authentication.

It doesn't completely stop working; a MITM can still pretend to be the server, it just can't authenticate to the real server on your behalf. You could be doing all your work in a fake server controlled by the attacker, while the real server sits there untouched.

reply
projektfu 3 hours ago
If that were the case then I don't think the OP approach works either.
reply
tabwidth 2 hours ago
[dead]
reply
riffic 4 hours ago
zero touch prod might be something folks should keep as a north star - SSH should be break glass and a last resort if possible.

https://www.usenix.org/conference/srecon19emea/presentation/...

reply
skydhash 7 hours ago
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).

To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.

It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).

Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.

reply
dboreham 9 minutes ago
Attacker might have MITMed the console.
reply
PunchyHamster 6 hours ago
Seems like way easier way would be sshing for the first time and just typing `sudo reboot`. If VM reboots, it is yours

Or cat-ing some secrets that would be on target machine but not attacker

reply