Migrating from Docker to LXC: A Practical Guide to Container Paradigm Shift

Docker’s abstraction fits 12-factor apps, but what about those times you need real init, granular cgroup configuration, or kernel-level customization? Standard Docker containers hit a wall in complex, stateful, or OS-intensive workloads. LXC isn’t new—it’s a mature solution for “system containers” offering much tighter integration with the underlying OS. This guide distills lessons learned from hard migrations: where Docker falls short and LXC takes the baton.

When Does LXC Make Sense?

Typical scenario: you try to launch a multi-service application or legacy software suite in Docker, but either systemd fails to initialize or the network stack needs per-container tweaks. Docker, by design, limits direct access to the host—making it tricky (or hacky) to tune low-level networking, cgroups, or mount points.

Core advantages of LXC in practice:

• Transparent access to Linux namespaces and complete cgroup control.
• Containers can run full init systems (e.g., systemd, OpenRC), unlocking broader compatibility for legacy or “fat” workloads.
• Native manipulation of container filesystems and direct device passthrough.
• Tighter performance profiles for stateful components (e.g., databases with special I/O requirements).

Table 1.

Real-World Migration Preparation

Transitioning from Docker to LXC isn’t a matter of swapping YAMLs. Expect to rethink initialization, networking, and filesystem access. Gotcha: direct Docker image import isn’t supported by LXC; you’ll be assembling your containers more like you do with chroots or classic VMs.

Checklist—before starting actual builds:

• Audit all Docker containers. Identify which expect systemd, rely on advanced networking (macvlan, host mode), or use storage plugins.

• Note every custom Docker volume, secret, and entrypoint.

• Check for kernel feature support (CONFIG_USER_NS, CONFIG_NET_NS, etc.):

  lxc-checkconfig
  # Watch for 'missing' or 'disabled' flags.
  

• Identify the Docker base images in use. LXC templates rarely include Alpine or scratch; conversions for minimal images may require manual package installation.

Stepwise Migration: Docker to LXC

1. LXC Installation and Baseline Configuration

On Ubuntu 22.04:

sudo apt-get update
sudo apt-get install lxc lxc-templates
# Verify service/component status
sudo systemctl status lxc
lxc-checkconfig | grep missing

Common misstep: missing kernel support for user namespaces or specific controllers. Some production kernels (notably RHEL derivatives) require explicit tuning.

2. Creating a Clean Container

Start with an OS environment matching your original base—which likely differs from a Docker image. Example for an Ubuntu workload:

sudo lxc-create -n myapp-lxc -t ubuntu -- -r jammy

• Older Docker containers (e.g., using deprecated CentOS versions) may need you to import local rootfs tarballs.
• Container naming best practice: use simple, unique names. LXC gets cranky with special characters.

Start and attach:

sudo lxc-start -n myapp-lxc -d   # daemonized container
sudo lxc-attach -n myapp-lxc

Side note: lxc-attach does not perform a full login; for systemd debugging, use machinectl inside the container if available.

3. Migrating Application and Configuration

Suppose your Dockerfile ran Nginx with a bundled config. You’ll transfer the runtime logic, not the Docker layering.

• Install packages manually:

  apt-get update && apt-get install -y nginx
  

• Copy in configuration:
  Use SSH, rsync, or mount a host directory. Example using LXC mount:

  # In container config
  lxc.mount.entry = /srv/nginx_conf /var/lib/lxc/myapp-lxc/rootfs/etc/nginx none bind,ro 0 0
  

• Start services via init system:

  systemctl start nginx
  systemctl enable nginx
  

If your Docker container ran multiple daemons (DB and web), LXC allows standard init orchestration.

Gotcha: Some lightweight Docker images omit /sbin/init or systemd. In LXC, always start with a full OS template unless deliberately minimizing the userland for security or footprint.

4. Network Topology

Docker’s network modes don’t map directly. In LXC, you explicitly define veth pairs, bridges, or even assign real VLANs.

NAT bridge example:

Edit /etc/lxc/default.conf:

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx

Set up lxcbr0:

ip link add lxcbr0 type bridge 2>/dev/null || true
ip addr add 10.0.3.1/24 dev lxcbr0 || true
ip link set lxcbr0 up
iptables -t nat -C POSTROUTING -s 10.0.3.0/24 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s 10.0.3.0/24 -j MASQUERADE

Known issue: Networking and firewalling defaults differ from Docker. You may need to explicitly tweak sysctl parameters for IP forwarding.

5. Storage: Mounts, Volumes, and Persistence

LXC expects explicit mount instructions in each container’s config file (/var/lib/lxc/<name>/config). Unlike Docker’s managed volumes, here you deal directly with filesystems, devices, or loopbacks.

Example: Host bind mount

lxc.mount.entry = /data /var/lib/lxc/myapp-lxc/rootfs/mnt/data none bind 0 0

For more isolation, consider overlayfs or LVM volumes. SELinux and AppArmor may block initial attempts—always check dmesg and audit.log on errors like:

mount: permission denied

6. Container Lifecycle & Automation

Typical workflow:

• Snapshot before risky changes:

  lxc-snapshot -n myapp-lxc
  

• Automate with bash or Ansible.
• For batch operations (e.g., bulk updates), loop over container names with the lxc CLI.

LXD brings a higher-level management API (REST, clustering, image publishing), but remains a distinct project—evaluate migration effort before jumping.

Additional Lessons from the Field

• Start with non-critical workloads. LXC config syntax is less forgiving; minor typos can create silent failures.
• Resource monitoring: Use in-container htop, but check actual limits on the host—cgroup rules apply differently.
• Log locations: LXC errors often land in /var/log/lxc/<name>.log or under journald; don’t chase Docker’s logs.
• Use prebuilt templates wisely: Community OS images often lag behind upstream releases.

Non-obvious tip:
To run Docker inside an LXC container (for build or CI), add:

lxc.apparmor.profile = unconfined
lxc.cgroup.devices.allow = a
lxc.mount.auto = proc:rw sys:rw

Beware—security trade-off. Only for isolated build sandboxes.

Final Note

Migration from Docker to LXC isn’t invertible—some orchestration abstractions (auto networking, image layering) disappear. But for ops teams needing transparent system-level containers, the trade-off means clarity and control. Take it step by step, document quirks, and expect a few surprises along the way.

Questions or hit a kernel quirk during migration? Log the details—solutions often lurk in the small print of release notes or mailing lists.