Mastering Secure SSH Access to Azure VMs: Beyond the Basics

Think you know how to SSH into your Azure VM? Think again. It’s not just about connecting—it’s about transforming access into a secure, seamless, and automated gateway that protects your environment and accelerates deployment workflows.

Securely accessing Azure Virtual Machines (VMs) is foundational for managing cloud infrastructure effectively. While the basics of SSH (Secure Shell) are widely understood—you generate a keypair, open port 22 in the network security group, then connect—the reality of handling production-grade environments requires going beyond these basics.

In this post, I’ll walk you through advanced yet practical techniques that improve your security posture, streamline operations, and help you manage Azure VMs more efficiently.

Why Go Beyond Basic SSH?

• Reduce attack surface: Simply opening port 22 publicly invites brute force attempts.
• Enforce least privilege: Rely on certificate-based authentication instead of passwords or static keys.
• Automate access management: Dynamically provision and revoke access without manual intervention.
• Enhance auditability: Know exactly who connected, when, and with what privileges.
• Improve workflow efficiency: Use tooling to automate your SSH access seamlessly into CI/CD pipelines.

Step 1: Use SSH Key Pairs Instead of Passwords

If you haven’t started here yet—do it now.

Azure recommends uploading an SSH public key when creating your VM rather than enabling password authentication.

Generate an SSH Key Pair (if you don’t have one)

ssh-keygen -t rsa -b 4096 -C "your.email@example.com"

This creates a private key (~/.ssh/id_rsa) and a public key (~/.ssh/id_rsa.pub).

When provisioning your Azure VM using the CLI:

az vm create \
  --resource-group myResourceGroup \
  --name myVM \
  --image UbuntuLTS \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_rsa.pub

This ensures password login is disabled and only your corresponding private key can log in.

Step 2: Lock Down Your Network Security Group

Opening port 22 to the whole internet (0.0.0.0/0) is a bad practice for any production workload.

Tip: Restrict SSH access to known IP ranges or use Azure Bastion or VPN for a safer jumpbox.

Example rule that allows SSH only from your current IP:

MY_IP=$(curl -s https://ipinfo.io/ip)/32
az network nsg rule create \
    --resource-group myResourceGroup \
    --nsg-name myNSG \
    --name AllowMyIPSSH \
    --protocol Tcp \
    --direction Inbound \
    --priority 1000 \
    --source-address-prefixes $MY_IP \
    --source-port-ranges '*' \
    --destination-address-prefixes '*' \
    --destination-port-ranges 22 \
    --access Allow

Step 3: Use Azure AD Authentication for SSH

Azure now supports integrating Linux VM login with Azure Active Directory (AAD). This allows centralized identity management and eliminates the need for key distribution.

Enabling AAD Login on Your VM

1. Assign the Azure AD login role to users or groups.
2. Enable AAD login on the VM by installing the Azure-Linux VM AAD extension.

Example CLI commands:

# Assign role at resource level:
az role assignment create \
   --assignee "user@domain.com" \
   --role "Virtual Machine Administrator Login" \
   --scope "/subscriptions/{subscription-id}/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVM"

Once set up, users can use their Azure AD credentials via az ssh.

Step 4: Use az ssh for Simplified Connection Management

The Azure CLI now has built-in support for SSH (az ssh). It helps you connect without exposing keys manually because it handles token exchange and key injection automatically when using AAD-enabled VMs.

az ssh vm -n myVM -g myResourceGroup

Benefits:

• No need for managing private keys locally.
• Supports multi-factor authentication through AAD.
• Seamless integration with Kubernetes nodes or scale sets.

Step 5: Automate SSH Access with Just-in-Time (JIT) Access

Azure Security Center offers Just-in-Time VM access, vetting inbound rules dynamically when needed.

You can request temporary open ports & get notified upon usage—a huge step up from permanently open inbound SSH ports.

To enable JIT:

az security jit-policy update -n 'default' -g myResourceGroup \ 
--vm-name myVM --ports '[{"number":22,"protocol":"*"}]'

Then approve requests on-demand via portal or API before opening port 22 temporarily.

Step 6: Harden Your Linux VM OS for SSH

Even after securing network and identity layers, hardening inside the OS matters:

• Change default SSH port (beware of automation impacts)
• Disable root login by editing /etc/ssh/sshd_config:

PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

• Limit user logins via AllowUsers azureuser directive in sshd_config
• Enable Fail2Ban or similar intrusion prevention tools to block repeated unauthorized attempts

After changes restart sshd:

sudo systemctl restart sshd

Bonus Tip: Use SSH Agent Forwarding & ProxyJump For Multi-Hop Connections

If you manage multiple VMs behind jump hosts/bastion VMs, configure ~/.ssh/config like so:

Host bastion-host
   HostName bastion.cloud.azure.com
   User azureuser
   IdentityFile ~/.ssh/id_rsa

Host internal-vm
   HostName internal-vm.private.ip.address 
   User azureuser 
   ProxyJump bastion-host    

This way,

ssh internal-vm

will automatically hop through your bastion securely using your agent keys — no need to copy private keys everywhere!

Wrapping Up

Mastering secure SSH access to your Azure VMs means more than just connectivity; it means building trust in your cloud infrastructure operations while enabling agility. Leveraging SSH keys properly, integrating with Azure AD, automating access controls with Just-in-Time policies, and hardening OS configs together create a strong security fabric around your critical workloads.

Try implementing these steps next time you provision or manage your Azure Linux VMs — your ops team will thank you, and so will your security audit reports!

If you found this helpful or want deeper walkthroughs on automating these workflows with Terraform or ARM templates, drop a comment below!

Happy cloud managing!