Rancher logging fluentbit启动报错：重复挂载/var/lib/docker/containers

问题现象 #

部分rancher-logging-root-fluentbit Pod可正常启动，其余Pod启动时报错：

Error: Error response from daemon: Duplicate mount point: /var/lib/docker/containers

已排查暂停状态Pod、异常状态Helm应用，未重启问题节点。

问题Pod详情 #

Name:             rancher-logging-root-fluentbit-scjn4
Namespace:        cattle-logging-system
Priority:         0
Service Account:  rancher-logging-root-fluentbit
Node:             xxxx
Start Time:       xxxx
Labels:           app.kubernetes.io/managed-by=rancher-logging-root
                  app.kubernetes.io/name=fluentbit
                  controller-revision-hash=5b6c67854b
                  pod-template-generation=3
Annotations:      checksum/fluent-bit.conf: 2b08687b2f14ac5fece45523412a2ba2669a33cc4e0e2c4479b752e92e511045
                  cni.projectcalico.org/containerID: fa706d155fa5e571893dfcf92bab107d0f2a3aeb0df0b3c7817223d2a757f949
                  cni.projectcalico.org/podIP: 10.42.0.84/32
                  cni.projectcalico.org/podIPs: 10.42.0.84/32
Status:           Pending
IP:               10.42.0.84
IPs:
  IP:           10.42.0.84
Controlled By:  DaemonSet/rancher-logging-root-fluentbit
Containers:
  fluent-bit:
    Container ID:   
    Image:          rancher/mirrored-fluent-fluent-bit:1.9.3
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100M
    Requests:
      cpu:        100m
      memory:     50M
    Environment:  <none>
    Mounts:
      /buffers from buffers (rw)
      /fluent-bit/etc/fluent-bit.conf from config (rw,path="fluent-bit.conf")
      /tail-db from positiondb (rw)
      /var/lib/docker/containers from varlibcontainers (ro)
      /var/lib/docker/containers/ from extravolumemount0 (ro)
      /var/log/ from varlogs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hvr65 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  varlibcontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers/
    HostPathType:  
  varlogs:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:  
  extravolumemount0:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers/
    HostPathType:  
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-logging-root-fluentbit
    Optional:    false
  positiondb:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  buffers:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-hvr65:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 cattle.io/os=linux:NoSchedule
                             node-role.kubernetes.io/controlplane=true:NoSchedule
                             node-role.kubernetes.io/etcd=true:NoExecute
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  4m51s                   default-scheduler  Successfully assigned cattle-logging-system/rancher-logging-root-fluentbit-scjn4 to xxx
  Warning  Failed     2m49s (x12 over 4m50s)  kubelet            Error: Error response from daemon: Duplicate mount point: /var/lib/docker/containers
  Normal   Pulled     2m35s (x13 over 4m50s)  kubelet            Container image "rancher/mirrored-fluent-fluent-bit:1.9.3" already present on machine

问题原因与解决方法 #

从Pod挂载配置可见，容器内的/var/lib/docker/containers路径被两个HostPath卷（varlibcontainers和extravolumemount0）重复挂载，触发Docker daemon报错。

解决方法：
安装Rancher Logging Helm包时，若手动指定默认值/var/lib/docker作为Docker根目录，会触发额外卷挂载逻辑，导致重复挂载。将Docker根目录配置留空（默认即为/var/lib/docker），重新部署Helm应用后，Pod即可正常启动。

内容的提问来源于stack exchange，提问作者Marco Brunet