We describe the procedure for emergency access to SSH hosts with hardware keys
In this post, we will develop a procedure for emergency access to SSH hosts using offline security keys. This is just one of the approaches, and you can adapt it for yourself. We will store the SSH certification authority for our hosts on a hardware security key. This scheme will work on almost any OpenSSH, including SSH with single sign-on.
Why all this? Well, this is an emergency option. This is a backdoor that will allow you to access your server in the event that for some reason nothing else helps.
Why use certificates instead of public/private keys for emergency access?
- Unlike public keys, certificate validity can be very short. You can generate a certificate valid for 1 minute or even 5 seconds. After this period, the certificate will become unusable for new connections. This is perfect for emergency access.
- You can create a certificate for any account on your hosts and, if necessary, send such “one-time” certificates to colleagues.
What you need
- Hardware security keys that support resident keys.
Resident keys are cryptographic keys that are completely stored inside the security key. Sometimes they are protected by an alphanumeric PIN code. The public part of the resident key can be exported from the security key, if necessary, together with the private key descriptor. Support for resident keys are, for example, USB-keys of the Yubikey 5 series. It is desirable that they are intended only for emergency access to the host. For this post I will use only one key, but you must have an additional one for backup.
- A safe place to store these keys.
- OpenSSH version 8.2 or higher on your local computer and on the servers that you want to access emergency. Ubuntu 20.04 ships with OpenSSH 8.2.
- (optional, but desirable) CLI certificate verification tool.
First you need to create a certification authority, which will be located on the hardware security key. Insert the key and run:
$ ssh-keygen -t ecdsa-sk -f sk-user-ca -O resident -C [security key ID]
As a comment (-C), I indicated email@example.com so as not to forget which security key this certification authority belongs to.
In addition to adding the key to Yubikey, two files will be generated locally:
- sk-user-ca, a handle to the key that refers to the private key stored in the security key
- sk-user-ca.pub, which will be the public key for your certification authority.
But don’t worry, Yubikey has one more private key that cannot be retrieved. Therefore, everything is reliable here.
On hosts as root, add (if you have not already added) the following to your SSHD configuration (/etc/ssh/sshd_config):
Then on the host add the public key (sk-user-ca.pub) to/etc/ssh/ca.pub
Reboot the server:
systemctl sshd restart
Now we can try to access the host. But first, we need a certificate. Create a key pair that will be associated with the certificate:
$ ssh-keygen -t ecdsa -f emergency
Certificates and SSH Couples
Sometimes it is tempting to use a certificate as a replacement for a public/private key pair. But for user authentication, a single certificate is not enough. Each certificate also has a private key associated with it. That's why we need to generate this pair of "emergency" keys before we issue a certificate to ourselves. It is important that we show the signed certificate to the server, indicating the key pair for which we have a private key.
Thus, the public key exchange is still alive and well. It rolls even with certificates. Certificates simply save the server from having to store public keys.
Next, create the certificate itself. I need ubuntu user authorization in a 10 minute interval. You can do it your own way.
$ ssh-keygen -s sk-user-ca -I test-key -n ubuntu -V -5m:+5m emergency
You will be asked to sign the certificate with your fingerprint. You can add additional usernames, separated by commas, for example, -n ubuntu, carl, ec2-user
That's it, now you have a certificate! Next, you need to specify the correct permissions:
$ chmod 600 emergency-cert.pub
After that you can familiarize yourself with the contents of your certificate:
$ step ssh inspect emergency-cert.pub
Here's what mine looks like:
emergency-cert.pub Type: firstname.lastname@example.org user certificate Public key: ECDSA-CERT SHA256:EJSfzfQv1UK44/LOKhBbuh5oRMqxXGBSr+UAzA7cork Signing CA: SK-ECDSA SHA256:kLJ7xfTTPQN0G/IF2cq5TB3EitaV4k3XczcBZcLPQ0E Key ID: "test-key" Serial: 0 Valid: from 2020-06-24T16:53:03 to 2020-06-24T17:03:03 Principals: ubuntu Critical Options: (none) Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc
Here, the public key is the emergency key we created, and sk-user-ca is associated with the certification authority.
Finally, we are ready to run the SSH command:
$ ssh -i emergency ubuntu@my-hostname ubuntu@my-hostname:~$
- Now you can create certificates for any user on a host who trusts your certification authority.
- You can remove emergency. You can save sk-user-ca, but you do not need it, since it is also on the security key. You might also want to remove the original PEM public key from your hosts (for example, in ~/.ssh/authorized_keys for the ubuntu user) if you used it for emergency access.
Emergency Access: Action Plan
Insert the security key and run the command:
$ ssh-add -K
This way you add the public key of the certification authority and the key descriptor to the SSH agent.
Now export the public key to make the certificate:
$ ssh-add -L | tail -1 > sk-user-ca.pub
Create a certificate with an expiration date of, for example, no more than an hour:
$ ssh-keygen -t ecdsa -f emergency $ ssh-keygen -Us sk-user-ca.pub -I test-key -n [username] -V -5m:+60m emergency $ chmod 600 emergency-cert.pub
And now again SSH:
$ ssh -i emergency username@host
If your.ssh/config file causes some kind of connection problems, you can run ssh with the -F none option to get by without it. If you need to send a certificate to a colleague, the easiest and safest option is Magic Wormhole . To do this, you only need two files — in our case, emergency and emergency-cert.pub.
What I like about this approach is the hardware support. You can put the security keys in a safe and they will not go anywhere.
Epic servers are cheap VPS with powerful processors from AMD, CPU core frequency up to 3.4 GHz The maximum configuration allows you to solve almost any task - 128 CPU cores, 512 GB RAM, 4000 GB NVMe. Join now!