SSH on Linux: Install OpenSSH, Connect, and Set Up Keys

SSH has two sides: the client (ssh), which you run on your machine, and the server (sshd), which runs on the machine you connect to. Almost every Linux desktop has the client already. The server is often missing on desktop installs and minimal images. This post covers both sides, from installing the server to a key-only setup.

Install and start the SSH server

Ubuntu and Debian

sudo apt update
sudo apt install openssh-server
sudo systemctl enable --now ssh

The Ubuntu Server installer has an "Install OpenSSH server" option and can import your public keys from GitHub or Launchpad, so on a fresh server it may already be there.

On Debian and Ubuntu the service is called ssh. On Ubuntu 22.10 and later it uses systemd socket activation: ssh.socket listens on port 22 and starts ssh.service on the first connection. Seeing ssh.service as inactive right after install is normal. Check the socket:

systemctl status ssh.socket
sudo ss -tlnp | grep ':22'

RHEL, AlmaLinux, Rocky, Fedora

sudo dnf install openssh-server
sudo systemctl enable --now sshd

Here the service is called sshd, and it is usually installed and enabled by default on server installs.

Open the firewall

# Ubuntu with ufw
sudo ufw allow OpenSSH
sudo ufw enable

# RHEL family with firewalld (ssh is usually allowed in the default zone already)
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload

Allow SSH before you run ufw enable on a remote machine, or you cut off your own session. On cloud VMs, the provider's security group or firewall rule also has to allow TCP 22 from your address.

Connect

Find the server's address on the server itself with ip -br addr or hostname -I. Then from your machine:

ssh alice@203.0.113.10
ssh -p 2222 alice@203.0.113.10          # non-default port
ssh -i ~/.ssh/work_ed25519 alice@host   # specific key
ssh alice@203.0.113.10 'df -h /'        # run one command and exit

If you leave out the username, ssh uses your local username.

On the first connection you are asked to confirm the server's host key fingerprint. On a server you control, compare it with the output of ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the server. After you accept, the key goes into ~/.ssh/known_hosts.

If the host key later changes, ssh refuses to connect with a "REMOTE HOST IDENTIFICATION HAS CHANGED" warning. After a legitimate rebuild, remove the old entry with ssh-keygen -R 203.0.113.10. If you did not rebuild the server, find out why it changed before you connect.

Key-based login

Passwords can be guessed and phished. Keys cannot be brute-forced in practice.

ssh-keygen -t ed25519 -C "alice@laptop"
ssh-copy-id alice@203.0.113.10
ssh alice@203.0.113.10                  # should not ask for the account password now

Set a passphrase on the key when ssh-keygen asks. ssh-agent, which most desktop sessions start for you, keeps the decrypted key in memory so you type the passphrase once per session.

ssh-copy-id appends your public key to ~/.ssh/authorized_keys on the server. Without it, do the same by hand and fix permissions, because sshd ignores keys in files that others can write to:

mkdir -p ~/.ssh && chmod 700 ~/.ssh
cat >> ~/.ssh/authorized_keys      # paste the .pub line, then Ctrl+D
chmod 600 ~/.ssh/authorized_keys

Turn off password logins

Once key login works, disable passwords. Put your settings in a drop-in file instead of editing the main config:

sudo tee /etc/ssh/sshd_config.d/10-hardening.conf <<'EOF'
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
EOF

sudo sshd -t                     # syntax check, no output means OK
sudo systemctl reload ssh        # sshd on RHEL family

Ubuntu, Debian 12+ and RHEL 9+ read /etc/ssh/sshd_config.d/*.conf at the top of sshd_config. For most settings, sshd uses the first value it finds, so a file named 10-... beats a later line in the main file. That matters on cloud images, where cloud-init often writes 50-cloud-init.conf with PasswordAuthentication yes. To see what is actually in effect:

sudo sshd -T | grep -Ei 'passwordauth|permitroot|port '

Keep your current session open and test from a second terminal before you log out.

PermitRootLogin defaults to prohibit-password (root can log in with a key only). no is stricter and fine if everyone has their own account with sudo.

Changing the port

A different port reduces log noise from scanners. It does not add real security. On RHEL with SELinux, label the port first: sudo semanage port -a -t ssh_port_t -p tcp 2222.

On Ubuntu 24.04 and later, the listening port belongs to ssh.socket, and a systemd generator reads Port from the sshd config. After setting Port 2222 in a drop-in, regenerate and restart the socket:

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket

Open the new port in the firewall before you close the old one.

A client config file

Long command lines belong in ~/.ssh/config:

Host web1
    HostName 203.0.113.10
    User alice
    Port 2222
    IdentityFile ~/.ssh/work_ed25519

Host db1
    HostName 10.0.1.20
    User alice
    ProxyJump web1

Now ssh web1 is enough, and ssh db1 connects to a private host through web1 as a jump host. The same works ad hoc with ssh -J alice@203.0.113.10 alice@10.0.1.20.

Copying files and forwarding ports

scp report.pdf web1:/tmp/                 # copy a file
rsync -avz ./site/ web1:/var/www/site/    # sync a directory, resumable
sftp web1                                 # interactive transfer
ssh -N -L 5432:localhost:5432 db1         # local port 5432 -> PostgreSQL on db1

Since OpenSSH 9.0, scp uses the SFTP protocol under the hood. The -L example lets a local client connect to localhost:5432 and reach a database that only listens on the server's loopback.

Cloud alternatives

On cloud VMs you often do not need open port 22 at all:

aws ssm start-session --target i-0123456789abcdef0        # AWS SSM Session Manager
aws ec2-instance-connect ssh --instance-id i-0123456789abcdef0
gcloud compute ssh my-vm --zone europe-west1-b --tunnel-through-iap

Session Manager needs the SSM agent and an instance role, and gives you IAM-based access with logging and no inbound rule. EC2 Instance Connect pushes a short-lived key. gcloud compute ssh manages keys for you.

Troubleshooting

Start with verbose output: ssh -v alice@203.0.113.10.

• Connection timed out: a firewall, security group or wrong IP. Nothing answered at all.
• Connection refused: the host answered, but nothing listens on that port. Check systemctl status ssh (or sshd) and the port.
• Permission denied (publickey): wrong user, wrong key, or bad permissions on ~/.ssh or authorized_keys on the server.
• Server-side logs: sudo journalctl -u ssh (-u sshd on RHEL family), or /var/log/auth.log and /var/log/secure.

Quick reference

sudo apt install openssh-server             # Ubuntu/Debian server
sudo systemctl enable --now ssh             # sshd on RHEL family
sudo ufw allow OpenSSH                      # firewall on Ubuntu
ssh user@host                               # connect
ssh-keygen -t ed25519                       # create a key
ssh-copy-id user@host                       # install it on the server
sudo sshd -t && sudo systemctl reload ssh   # validate and apply config
sudo sshd -T                                # effective server config
ssh -J bastion user@private-host            # through a jump host
ssh -v user@host                            # debug a failing connection