SSH Into a Docker Container—Responsible Debugging Without Eroding Security

“Never SSH into a container”—the mantra echoes across DevOps handbooks and conference talks. In reality, the boundary is not so crisp. Sometimes, container introspection is the only way to uncover subtle issues that defy logs and metrics.

Consider a production environment running Docker Engine v24.0 and containers based on a hardened Alpine Linux base image. An unfamiliar PID spikes CPU load, logs are stale, and the CI/CD pipeline can’t replicate the issue. What next? Direct shell access, if handled with strict controls, is often the least disruptive answer.

When Direct Access Makes Sense (Despite Purism)

• Forensics on a process leaking file descriptors (see lsof output directly)
• Environment drift—external dependencies present in runtime but absent at build
• Real-time patch for a critical service to restore SLAs before a rebuild

Note: All such interventions should be ephemeral. Embed a high-friction approval step before reaching for remote shells in production.

Pitfalls: Common SSH Mistakes in Docker Contexts

1. docker exec with root:

docker exec -it <container_id> /bin/sh

This works, but beware: running as root sidesteps all container-user safeguards. On a misconfigured host with a shared PID namespace, this can even leak privileges (see Docker CVE-2019-5736).

2. Installing OpenSSH server in the container:
Bloats the image, increases attack surface, and encourages persistent per-container login—antithetical to immutable infrastructure.

3. Exposing SSH ports (-p 2222:22):
Negates any network isolation the orchestrator provides. If you must do this, double-check firewalls and use temporary subnets.

4. Leaving traces:
Manual fixes and tweaks risk drift between the deployed container and CI artifacts.

Secure Patterns for Interactive Access

Shell Access via Docker Exec—Restrict Privileges

Most issues are diagnosable with one-off exec commands:

docker exec -it --user $(id -u appuser) <container_name> sh

• Specify --user; never assume root is required—introspect the container’s /etc/passwd.
• Optionally, combine with Docker’s User Namespaces (experimental in some versions; check daemon.json: userns-remap).
• Log all docker exec invocations with tools like auditd.

Ephemeral Debug Containers—Sidecar Approach

Leverage an external container for indirect investigation:

docker run --rm -it --network container:<target_container> debian:12-slim bash

Variants can mount volumes or inject debugging binaries (strace, tcpdump) not present in minimal images.

Non-obvious tip: Some orchestrators (K8s, Nomad) allow joining an ephemeral container to a live pod for forensic inspection.

Temporary SSHD: Only When Absolutely Required

Not recommended for ongoing ops, but sometimes unavoidable (e.g., regulatory compliance audits).

Minimal Dockerfile for temp SSHD:

FROM alpine:3.19
RUN apk add --no-cache openssh \
    && ssh-keygen -A \
    && echo "PermitRootLogin prohibit-password" >> /etc/ssh/sshd_config \
    && echo "PasswordAuthentication no" >> /etc/ssh/sshd_config
COPY id_rsa.pub /root/.ssh/authorized_keys
EXPOSE 2222
CMD ["/usr/sbin/sshd", "-D", "-e", "-p", "2222"]

Launch & connect:

docker build -t debug-sshd .
docker run --rm -d --name debug_ssh -p 2222:2222 debug-sshd
ssh -i ~/.ssh/id_rsa -p 2222 root@localhost

Important: Destroy this container after investigation. Always use key-based authentication. Rotate keys pre– and post-use.

Kubernetes and Orchestration: Native Exec Beats SSH

In a cluster context, kubectl exec or equivalent is almost always preferable.

kubectl exec -it <pod-name> --container <container-name> -- sh

• RBAC governs access; central audit trails are available.
• Container runtimes (containerd, CRI-O) expose standardized exec APIs; no open ports.
• If running a distroless image, check for the presence of /bin/sh or /busybox/sh. If unavailable, inject a debug container (see Ephemeral Containers in K8s).

Cautions, Logging, and Clean-Up

• Never persist manual changes. Always bake fixes back into a new image/tag.
• Audit everything: docker exec logs, SSH access logs (see /var/log/secure in container, if present).
• In production, require approval (e.g., PagerDuty, change ticket) before direct access.
• Clean up temp containers, credentials, and open ports immediately.

Reality Check: When Rules Yield to Requirements

Is this ideal? No. The immutable-container model is there for a reason. But operational reality—especially on legacy platforms or high-stakes prod emergencies—can demand direct access. The key is discipline: revert containers after intervention, never allow “permanent” SSH, and scrutinize all access through the same lens as production firewall changes.

Known issue: Some cloud providers (notably managed K8s with custom network overlays) may block kubectl exec or severely rate-limit out-of-band access. Plan for a fallback (e.g., SSM on AWS, direct node access for last-resort debugging).

Summary Table: Access Methods

Practical example:
Diagnosing a process inside a Compose service:

docker exec -it --user www-data web_1 sh -c "ps aux | grep php-fpm"

Non-obvious tip:
Attach with nsenter instead of docker exec if you lose the Docker socket but retain host root—sometimes useful in catastrophic recovery:

PID=$(docker inspect --format '{{.State.Pid}}' <container_id>)
nsenter -t "$PID" -a

Final word:
Guard access with discipline—treat every shell as a temporary escalation, and always restore immutability. The most secure container is the one you never SSH into; the most practical engineer knows when to break that rule, fix the issue, and document everything.