Efficient Strategies for Upgrading from CentOS 7 to CentOS 8 with Minimal Downtime

Upgrading CentOS 7 hosts is rarely a seamless affair. The OS leap (7, kernel 3.10, to 8, kernel 4.18) includes changes in almost every subsystem: from sysctl defaults to DNF package management, systemd unit semantics, PAM modules, and Python interpreter versions. Relying on an "in-place" approach is analogous to a live-patch on a brittle codebase: possible, but almost always sub-optimal outside of test benches.

Below is an actionable, production-focused workflow for minimizing downtime and risk during CentOS 7-to-8 transitions—assuming virtualized workloads, but with side notes for bare metal.

Motivation: Why Transition, and Why Now?

CentOS 7's full support sunset (June 2024) marks the start of the entropy curve—CVE gaps, kernel vulnerabilities, EOL application dependencies. Performance uplift is measurable: OpenSSL 1.1 in CentOS 8, post-4.x kernels, improved I/O scheduling, Podman-native containers replacing Docker, and up-to-date systemd for advanced service orchestration.

Miss the window, and application migrations become harder—Python 2 dies, container builds fail, backports aren’t maintained.

Challenge Overview

Key blockers:

• No officially-supported in-place upgrade from CentOS 7 to 8 (unlike 6→7 with redhat-upgrade-tool).
• Significant package tree changes (yum replaced by dnf, many libraries dropped, SELinux policy changes).
• Configuration drift accumulated from years of minor upgrade and patching.
• Application state that can’t easily be rebuilt from scratch (gotcha: database migrations with legacy collations or C extensions tied to glibc).

Migration Strategy: Staged Cutover

Imperative: Never touch the only copy of production data during OS migrations. Spin up parallel instances, validate workloads, then migrate state. This approach enables atomic cutover with rollback, and staged validation.

1. Inventory and Full Backups

Start with a complete system inventory:

• Enumerate running and enabled systemd units:

  systemctl list-units --type=service --state=running
  

• Dump package list:

  rpm -qa --qf "%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n" > pkgs-$(hostname)-$(date +%F).txt
  

• Archive /etc, /var/lib, and custom application state. For quick, consistent capture:

  tar czf /var/backups/etc-$(date +%F).tgz /etc
  

Snapshot VM (if virtualized) or LVM volumes:

lvcreate -L 10G -s -n root_snap /dev/centos/root

Note: For bare metal, offline backups using rescue images or network block-level tools (Clonezilla, Borg) are safer.

2. Provision Clean CentOS 8 Hosts

Provision equal-size VMs/servers running CentOS 8.6. Use identical disk layouts and attach separate storage volumes for application data if possible.

Validate environment parity:

• Network interfaces, VLAN tags, routes
• SELinux enforcing/permissive states (getenforce)
• Locale and time zone

Use official mirrors or internal repositories to pre-stage packages and avoid last-minute surprises from broken upstream mirrors.

3. Automate Baseline Configuration

Rebase critical configuration using Ansible, Puppet, or similar. Avoid hand-conversion of files—automate with templated roles to ensure idempotency.

Example: deploying Nginx + certs, with service enabled and custom firewall rules. (ansible snippet)

- hosts: centos8
  become: yes
  tasks:
    - name: Install Nginx
      dnf:
        name: nginx
        state: present
    - name: Push ssl certs
      copy:
        src: files/certs/
        dest: /etc/nginx/ssl/
    - name: Open HTTP/S in firewalld
      firewalld:
        zone: public
        service: "{{ item }}"
        permanent: yes
        state: enabled
      loop: ["http", "https"]
    - name: Enable and start nginx
      service:
        name: nginx
        enabled: yes
        state: started

Gotcha: Systemd unit names or locations may change between 7 and 8 (double-check custom units).

4. Application Migration: Dry Run & Data Sync

Initiate stateless services first—static sites, reverse proxies. For data-heavy systems (DB, stateful containers):

• Dry run rsync with --dry-run or -n:

  rsync -avz --delete -n /var/lib/mysql/ user@centos8:/var/lib/mysql/
  

• Identify permission or SELinux mismatches (restorecon -Rv /var/lib/mysql; check audit.log for denials).
• Achieve near-zero downtime cutover: Run initial sync while CentOS 7 node is live; at final switchover, halt service, catch up with final rsync, then start on target.

Sample staged rsync:

rsync -az --delete /var/www/ user@centos8:/var/www/
systemctl stop httpd
rsync -az --delete /var/www/ user@centos8:/var/www/

Tip: For high-write DBs, consider logical replication to the new host prior to cutover (e.g., MySQL GTID, PostgreSQL streaming replication).

5. Traffic Shift: DNS or Load Balancer Cutover

After validation (test / status checks, application pings):

• If using a load balancer (e.g., HAProxy, F5), add new CentOS 8 backend, dial up weight, and monitor.
• For DNS-based services, reduce DNS TTL to low value (60s) ahead of switch, point A record to the new IP, then observe system logs (journalctl -xe, application logs) for new errors.

Known issue: "SSL handshake failed" often appears if TLS parameters or certs aren’t faithfully restored.

6. Cleanup and Legacy Decommission

Post-migration, maintain the CentOS 7 node in a "frozen" state (network-restricted, read-only) for at least 72 hours until confident in system stability. Only remove after confirmatory backup retention and validation.

Alternative: In-Place Upgrade (Use With Caution)

Red Hat’s leapp tool can attempt an in-place upgrade, but official support on CentOS is absent and breakages are frequent (kernel modules, custom drivers, EPEL packages).

Minimum viable approach:

yum install epel-release -y
yum install leapp leapp-repository -y
leapp preupgrade  # Review /var/log/leapp/leapp-report.txt
leapp upgrade
reboot

Errors like:

FAIL: Unable to migrate package: foo-1.2.3.el7.x86_64

are routine. Post-upgrade, system may require manual fixes for network scripts (/etc/sysconfig moves), lost modules, or broken SELinux contexts. Use only for non-critical instances, and snapshot first.

Recap: Minimal Downtime Upgrade Flow

Side note: For containerized workloads, replatform to RHEL UBI or AlmaLinux—future-proofs against further EOL drama.

Upgrading CentOS never fits a one-size-fits-all narrative. Document oddities, adapt per workload, and force as much automation as realistically possible. No UI for panic recovery; only good backups and solid process.