Docker桥接网络容器在主机无重启切换网络时DNS解析失败问题

问题背景 #

预期行为 #

运行在bridge网络下的Docker容器，当主机网络在无重启情况下切换（network1为192.168.1.*段、network2为192.168.2.*段）时，DNS解析应当正常生效。

实际行为 #

主机网络切换后容器内DNS解析失败。

复现步骤 #

1. 创建名为abc_network.ctr的容器，不指定network_mode参数，默认使用bridge网络，容器内部使用Docker嵌入式域名服务器
2. 创建第二个容器xyz.ctr，将其network_mode设置为container:abc_network.ctr
3. 查看xyz.ctr容器内/etc/resolv.conf内容，此时域名解析正常，内容如下：

root@d3e0dc71f370:~# cat /etc/resolv.conf 
nameserver 127.0.0.11
options ndots:0

4. 查看宿主机/etc/resolv.conf内容，此时域名解析正常，内容如下：

root@RP4-2711-c03112-1xb0e6f3b8:~# cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.1.1

5. 不重启设备，拔出网线接入另一个子网，将主机网络从network1切换至network2
6. 在Docker容器内执行以下命令测试域名解析：

• 执行docker exec -it xyz.ctr bash进入容器
• 执行apt-get update，返回结果如下：

root@d3e0dc71f370:~# apt-get update
Err:1 http://security.debian.org/debian-security buster/updates InRelease
  Temporary failure resolving 'security.debian.org'
Err:2 http://deb.debian.org/debian buster InRelease                  
  Temporary failure resolving 'deb.debian.org'
Err:3 http://deb.debian.org/debian buster-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists... Done    
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.

• dig命令输出如下：

root@15c396b06f57:~# dig

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>>
;; global options: +cmd
;; connection timed out; no servers could be reached

7. 此时查看宿主机/etc/resolv.conf，已更新为新子网的DNS配置，宿主机域名解析正常，内容如下：

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.2.1

8. 宿主机上apt-get、dig命令均运行正常无报错

环境信息 #

docker version输出：

Docker version 20.10.6, build 370c289

docker info输出：

root@RP4-2711-c03112-1xb0e6f3b8:~# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 22
  Running: 18
  Paused: 0
  Stopped: 4
 Images: 98
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.17-v7l+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 3.44GiB
 Name: RP4-2711-c03112-1xb0e6f3b8
 ID: C66C:3DAT:6DLH:PHJA:JR2N:KURB:PTMQ:BHUP:XRIH:FE3H:VSEZ:B7K3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.128.0.1/10, Size: 16

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

额外环境说明：测试设备为Raspberry Pi-4B、Intel NUC，该问题不特定于单一运行环境 #

根因说明 #

该问题是Docker 20.10.6及更早版本的已知缺陷：dockerd进程启动时会一次性缓存宿主机的上游DNS地址，bridge网络使用的嵌入式DNS服务（127.0.0.11）不会监听宿主机resolv.conf的动态变更，当主机切换网络后，缓存的旧DNS地址失效，就会导致容器DNS解析超时。同时使用container:abc_network.ctr的共享网络栈模式时，两个容器共用一套DNS配置，进一步导致变更无法自动同步。 #

解决方案 #

临时修复（无需重启容器） #

执行以下命令重启dockerd进程，强制重新读取宿主机最新的DNS配置：

systemctl restart docker

注意：如果开启了Live Restore功能，重启dockerd不会影响运行中容器，否则容器会被强制停止。

永久修复 #

1. 升级Docker版本到20.10.10及以上，该版本已修复DNS缓存不自动更新的问题
2. 若无法升级Docker，可在/etc/docker/daemon.json中固定公共DNS地址，避免随子网切换变动：

{
  "dns": ["223.5.5.5", "114.114.114.114"]
}

配置后重启dockerd生效：

systemctl daemon-reload
systemctl restart docker

内容的提问来源于stack exchange，提问作者CheTan Neve