1. SSH into the Pi
2. Issue the Wake-on-LAN packet to boot the server
3. Tunnel that server's SSH port to my laptop
4. SSH into the initramfs SSH server (I use TinySSH), enter the decryption key
5. Wait for server to come up, then access via Tailscale
This is more complicated than the author's setup in that it requires another device (the Pi), but it's simpler in that you don't need to have the initramfs on your Tailnet.
If you set this up once ("this" meaning adding networking, SSH and tailscale inside initramfs), you can just do the same thing for the next server you set up, and you don't have to worry about the failure of one node affecting the other(s).
But scaling also isn't really a parameter I (or the author) are optimizing for: we have a single beefy server we do all our work on, and a thin laptop client we want to access the server from, remotely and booting an encrypted root partition.
I don't necessarily understand the deployment question. If it's about the Raspberry Pi, I just do my updates when I don't need to use it to boot the server.
I use OpenSuse so I had to use the guide for Fedora, but there were some differences as far as I remember.
/usr/lib/dracut/modules.d/46cryptssh:
#!/bin/bash
check() {
require_binaries sshd || return 1
return 0
}
depends() {
return 0
}
install() {
if [ ! -e /etc/ssh/dracut ]; then
mkdir /etc/ssh/dracut &&
tmp=$(mktemp -d) &&
head -c128 /dev/random > $tmp/passphrase &&
ssh-keygen -t ed25519 -f /etc/ssh/dracut/ssh_host_ed25519_key -N"$(base64 < $tmp/passphrase)" &&
tpm2_createprimary -C o -c $tmp/primary.ctx &&
tpm2_pcrread -o $tmp/pcr.bin sha256:0,7
tpm2_createpolicy --policy-pcr -l sha256:0,7 -f $tmp/pcr.bin -L $tmp/pcr.policy
tpm2_create -C $tmp/primary.ctx -L $tmp/pcr.policy -i $tmp/passphrase -c $tmp/seal.ctx &&
tpm2_evictcontrol -C o -c $tmp/seal.ctx -o /etc/ssh/dracut/seal || {
rm -r $tmp /etc/ssh/dracut
exit 1
}
rm -r $tmp
fi
for file in /etc/ssh/dracut/*; do
inst_simple "$file" "/etc/ssh/${file#/etc/ssh/dracut/}"
done
mkdir -p -m 0700 "$initdir"/root/.ssh
/usr/bin/install -m 600 /etc/ssh/dracut_authorized_keys "$initdir"/root/.ssh/authorized_keys
inst_binary /usr/sbin/sshd
inst_binary /usr/sbin/ssh-keygen
inst_binary /usr/bin/tpm2_unseal
inst_binary /usr/bin/base64
inst_simple /usr/lib/libtss2-tcti-device.so
inst_simple "$moddir"/cryptsshd.service "$systemdsystemunitdir"/cryptsshd.service
inst_simple "$moddir"/sshd_config /etc/ssh/sshd_config
inst_binary /usr/lib/ssh/sshd-session
inst_binary /usr/lib/ssh/sshd-auth
mkdir -p -m 0755 "$initdir"/var/empty/sshd
mkdir -p -m 0755 "$initdir"/usr/share/empty.sshd
mkdir -p -m 0755 "$initdir"/var/log
touch "$initdir"/var/log/lastlog
systemctl -q --root "$initdir" enable cryptsshd
}
[Unit]
Description=OpenSSH Daemon for Disk Encryption Passphrase
DefaultDependencies=no
Before=cryptsetup.target
After=network-online.target
[Service]
Type=notify-reload
ExecStartPre=/bin/sh -c '/usr/bin/ssh-keygen -p -f /etc/ssh/ssh_host_ed25519_key \
-N "" -P "$(/usr/bin/tpm2_unseal -c /etc/ssh/seal -p pcr:sha256:0,7 | base64)"'
ExecStart=/usr/bin/sshd -D
KillMode=process
Restart=always
[Install]
WantedBy=sysinit.target
It's still probably vulnerable to a cold boot attack, since the key is decrypted CPU-side. It would be interesting to perform the actual key operations on the TPM itself to prevent this.
Other options that I've investigated that involve having a second server:
* A second server with Tang, and Clevis in the initramfs OS
* Keylime
Putting tailscale in the initramfs, and then updating the certs on a frequent enough schedule, seems risky to me. I've already played around with limine enough that I know I don't want to install much in the initramfs...
Realistically for a home server what you are worried about is someone breaking in and selling your drives on Facebook marketplace rather than the FBI raiding your nextcloud server. So TPM automated unlock is perfectly sufficient.
If someone steals the entire machine, the drives will unlock themselves automatically. I don't think it's worth the risk to assume a hypothetical thief is too lazy to check if there's any valuable data on the disks. At the very least, they'll probably check for crypto wallets.
With something like Clevis and Tang, you can set it up so it only auto unlocks while connected to your home network, or do something more complex as needed
Of course, a thief could try to bypass the login screen by e.g. booting with a different kernel command line, or a different initramfs. If you want to avoid this vulnerability, TPM unlock can be configured as a very fragile house of cards - the tiniest change and it falls down. The jargon for this is "binding to PCRs"
Having key off-machine mitigates a lot of that.
> Unless the junkie who stole your server has an unpatched debian login bug,
the key for disk decryption is in memory at that point. There are methods to take it out of it
E.g. systemd measures basically everything that is part of the boot process (kernel, kernel cli, initrd, ...[1]) into different PCRs, so if any of those are different they result in differen PCR values and won't unlock the boot device (depending on which PCRs you decided to encrypt against). I forgot what excatly it measures, but I remember that some PCRs also get measured during the switch_root operation from initrd -> rootfs which can be used to make something only unlock in the initrd.
If this fails you can always manually input the decryption key and reregister with the TPM. The whole point of this setup is you can't just use a bootable USB to reset the devices password.
Just use dm-verity for remote servers.
If you have disk encryption, your data now requires the police to force you to produce a password, which may or may not be within their powers, depending on the jurisdiction.
It’s strictly better to have full disk encryption and remote unlocking than no disk encryption at all, because it prevents such „system was switched off by accident“ attacks.
They have kits that allow them to unplug the server from the wall without interrupting power supply, specifically so they don't lose the decryption keys.
If confidentiality during remote unlock matters, seal the LUKS key in TPM2 tied to PCR values using systemd-cryptenroll or use Clevis with Tang over TLS with strict server certificate pinning, accept the operational cost of re-sealing keys after kernel or firmware updates, and keep an offline recovery key because trusting the local console is asking for trouble.
Maybe I have a server at home, with a locked cabinet and vibration sensors, that houses a server or two and they all use full disk encryption, but I still want to be able to reboot them without having to connect a physical keyboard to them. So no one has physical access, not even me, but I still want to be able to reboot them.
Or countless of other scenarios where it could be useful to be able to remotely unlock FDE.
The argument was that physical access gives up the FDE key.
Or is there already a solution to this that I've been missing? (Yeah, KVM/IPMI/etc, I know, but not all hosters make it easy to get to that.)
The better solution is to use tpm, unified kernel image and secure boot skipping the network unlock.
The whole process is like this -
1. enable secure boot;
2. generate and install your own secure boot keys (using sbctl);
3. use clevis to enable automatic unlocking of the root fs only when secure boot check passes;
4. generate the unified kernel image (in EFI partition) that is signed by your secure boot key;
4. use efibootmgr to enable booting of said kernel image.
(5.) If your CPU supports it, enable memory encryption in BIOS (to mitigate cold boot attacks).
The unified kernel image doesn't accept additional kernel parameters, so only parameters that are set during generation of the initram are used. The secure boot makes sure no one else has tampered with the boot chain. And TPM stores the disk key securely.
You can still add some additional network level check to make sure that your computer is in your expected location before unlocking.
And you can also include some recovery tools + dropbear in your initram (within the unified kernel image), if you expect that you will have to do some recovery from the other side of the world.
If you must do such upgrades, solutions include hot standby hardware, IPMI, an on-site tech with a screen and keyboard, or moving everything to the cloud.
When rebooting a FileVault encrypted machine, where it normally "hangs" asking for a user to unlock it, you can now SSH into the machine, but instead of getting a prompt it interprets your SSH login as a user logging in, hangs up, and proceeds to booting up.
The bad news: `The capability to unlock the data volume over SSH appeared in macOS 26 Tahoe.`
Debian has (or had; at least my Devuan still has) a simple shell script as first init. Was an interesting read and helped me understand were to add my remote rootfs decryption.
https://salsa.debian.org/kernel-team/initramfs-tools/-/blob/...
The `base` hook installs the shell PID 1, the `systemd` hook installs systemd as PID1. The default hook setup was changed with the latest'ish release to default too the `systemd` hook setup.
Shell `init`; https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio...
Give it a go: https://aur.archlinux.org/packages/mkinitcpio-wifi
I've done stuff with mkinitcpio / initramfs on arch before, can't remember exactly what for. I still run arch on my main laptop. I'm running nixos on my home server though, and adding something like this is so easy by comparison.
I once built a demo-ish encrypted network boot system using similar initrd techniques. It's a fun hack working in the preboot environment.
It's not a huge problem but it certainly means some recovery scenarios would be painful.
I've only seen it on some paranoid-level devices in industry (typically devices handling biometric identity verification services).
IIRC this one is a Linux image that boots up, unlocks the normal Bitlocker partition via whatever mechanism you need, then hands control back to the Windows bootloader to continue onwards.
https://winmagic.com/en/products/full-disk-encryption-for-wi...
https://pikvm.org/ and similar exist.
This is maybe not as good as the article solution, because it requires you to secure the pi too.
Mandos works with initramfs images created by both initramfs-tools and Dracut, and is present in Debian since 2011, so no need to use a third-party package.
I want something like "kexec" but taking over the entire hardware at the lowest possible level.
Edit, found this:
https://github.com/marcan/takeover.sh
But it's not as low level as I hoped, though it keeps the network running which is nice :)
Mandos is also for Debian (and all Debian-based systems), and not Red Hat (although there is nothing preventing a port, since Mandos now also supports initramfs images created by Dracut).
Finally, Mandos was initially created in in 2007, many years before tang/clevis, and literally by a person in a dorm room, not by whatever academics/scientists who seem to have dreamed up the cryptography/protocol tangle that is tang/clevis. Mandos has changed some since then, from initially using broadcast UDP and X.509 certificates, to using DNS-SD and raw public keys², but has otherwise remained very similar to its initial design.
1. <https://www.dns-sd.org/>
2. <https://www.rfc-editor.org/rfc/rfc7250>
I think FOSDEM had a talk a few years ago about that solution and their tang server. It is very similar in concept. They use McCallum-Relyea exchange, and we use PGP encryption, but the basics are the same in that you need information stored at the server and information stored in the unencrypted initramfs in order to be combined and turned into the encryption key. If my understanding of the McCallum-Relyea exchange is correct, they also combine the key creation and perfect forward security of the transaction into a single protocol, while we use TLS and client keys for the perfect forward security aspect and PGP for data at rest.
With tang you need to verify the hash of the tang server in order to prevent a MITM threat from extracting the server side part during the key creation process, while with Mandos you configure it manually with configuration files.
Tang uses REST and JSON. Mandos sends a single protocol number and then sends the data. Both operate primarily through local LAN, through Mandos client also support as an option to use global reachable ip addresses.
Outside of those design differences, there are some packaging aspects. Tang was designed for Redhat/Dracut, while Mandos was designed for Debian and initramfs-tools (Today Mandos also support Dracut, but it is not packaged for Redhat). Redhat packaging has been requested multiple times for Mandos, but neither of us two developer are Redhat users.
To add some historical context, I recall a Debconf BoF by the developers of initramfs-tools about if they were going to continue developing initramfs-tools or giving up and port everything to Dracut, and the silence was fairly deafening. People did not want to give up on initramfs-tools, but everyone recognized the massive duplication that those two project are. Similarly, the reason why Redhat had chosen initially to develop Dracut rather than just port initramfs-tools is also fairly big mystery, and is generally considered a Not Invented Here syndrome. Today there are however some distinct difference in design between the two systems.
I set up what you are suggesting (sort of anyway[1]) on a personal VPS to reboot after updates, that require one. I just generate an initrd in tmpfs that contains a keyfile[2] and kexec with that ephemeral initrd; The newest kernel can be found by looking at what the /boot/vmlinuz symlink points to. Been running this for years. It is 100% reliable, and simple. And, for the purposes of this box, secure enough.
For remote unlocks from initial power on, Debian has had that since forever using keyscripts and dropbear in the initrd.
[1] You could pull the key from memory, and use that to unlock the disk from within the generated initrd, but it would be more work than just setting up a keyfile in advance. It was my first thought as well.
[2] Easiest way was to use a mount namespace to use a diff crypttab file that points to the keyfile, since cannot specify crypttab location when creating the initrd. E.g.,
(mkinitramfs is usually wrapped by update-initramfs, but calling it directly allows specifying a location)Also most distros don't support using kexec for kernel upgrades anyway.
Then you turn off the server, and just start a vm with the captured init and capture the key.
Now you can decrypt the server offline with all the time in the world.
That only works with RAID 1. If the server uses RAID 5 or RAID 6, this won’t work.
> extract what you need
Well, yes. This is addressed in the FAQ.
> or change the image.
> Then you turn off the server, and just start a vm with the captured init and capture the key.
Well, as explained in the FAQ, an attacker will have to do so quickly, before the Mandos server decides that the Mandos client has been offline for too long, and disables that client. The default value is five minutes, but is configurable per client.
5 minutes is plenty to boot initrd from a vm... what's that gonna take? 10 seconds?
With RAID 5, pulling 1/3 gives you only partial data, and pulling 2/3 removes too much so the system can't run.
But even if it is, you could just pull one after the other and wait for the resilver before pulling the next one (you will hear if it resilvers automatically)