Solutions Log
Docker containers going dark on Rocky Linux 9 host
Preamble
Docker container networking on RHEL-based Linux hosts depends on a set of dynamically managed iptables/nftables rules that Docker injects at startup. These rules handle NAT, port publishing, and traffic forwarding for container bridge networks. Because other system components also interact with the host's netfilter subsystem, updates to those components can disrupt Docker's networking rules if the Docker service is not restarted afterward.
Issue Description
Docker containers running on a Rocky Linux 9 host became unreachable from external hosts. The containers appeared healthy and running normally from the Docker host itself, but all inbound network traffic from outside the host was silently dropped. No error messages were logged by Docker or the application workloads inside the containers. Restarting the Docker service resolved the issue.
Affected Systems
- Rocky Linux
- Fedora
- RHEL
Cause
dnf-automatic applied package updates to the host at 06:34 UTC on 2026-06-05 (transaction ID 14). The following packages were updated:
gnutlsopenssl,openssl-fips-provider,openssl-libssystemd,systemd-libs,systemd-pam,systemd-rpm-macros,systemd-udev
The systemd upgrade is the most likely trigger. Upgrading systemd, particularly systemd-udev, which manages device and network interface events, can affect the host's netfilter state in ways that invalidate Docker's iptables rules without causing Docker to restart or log any errors. Because Docker only injects its required firewall rules at startup, it has no mechanism to detect or recover from rules being externally invalidated while it is running.
The precise mechanism by which the systemd upgrade disrupted Docker's iptables rules could not be confirmed from available logs, as Docker itself logged no events during or after the update window. The causal relationship is strongly implied by the timing of the transaction and the nature of the packages updated.
Resolution
The immediate issue was resolved by restarting the Docker service, which caused Docker to re-establish its required iptables/nftables rules. Docker package updates were also applied at the same time, though these are not believed to be the cause of the disruption.
To prevent recurrence, dnf-automatic was configured to no longer apply updates automatically by setting apply_updates = no in /etc/dnf/automatic.conf. Package updates will instead be applied manually during scheduled maintenance windows, ensuring that any service restarts required as a consequence of updates can be performed in a controlled and deliberate order.
Supplemental: Path to Discovery
The issue was initially suspected to be a firewalld/iptables conflict, which is a well-documented interaction problem on RHEL-based hosts running Docker. However, review of the dnf-automatic transaction history narrowed the cause to a specific update event:
dnf history info 14
Transaction 14 confirmed that dnf-automatic had run at 06:34 UTC, updating systemd and openssl/gnutls packages. Notably absent were any firewalld, nftables, kernel, or Docker package updates, which ruled out the most commonly cited causes of this class of issue.
A journal query covering the transaction window was run to look for corroborating log evidence:
journalctl --since "2026-06-05 06:34:00" --until "2026-06-05 06:40:00" \ -u docker -u systemd-udev -u firewalld
This returned no entries, confirming that Docker was not restarted during or after the update, and logged no awareness of its networking rules being disrupted. This is consistent with the silent failure mode: Docker's iptables rules were invalidated externally, but Docker had no mechanism to detect or report this condition.
The systemctl list-units | grep -i docker output confirmed the service unit name as docker.service and showed all container scopes in a running state, consistent with containers being alive but unreachable due to missing forwarding rules.
The root cause attribution to the systemd upgrade remains inferred rather than directly evidenced, as the log record does not capture the specific netfilter event that invalidated Docker's rules.