HAproxy静态Pod在containerd节点CPU占100%，求排查方案

弃用Docker shim，将kubelet容器引擎切换为containerd后，haproxy静态Pod在节点上CPU占用率达到100%。由于资源限制对静态Pod不生效，无法通过该方式解决问题。该Pod在Podman及K8s使用Docker作为容器引擎时运行正常，现寻求涉及kubelet、containerd和haproxy的排查方法。

HA-Proxy版本信息 #

HA-Proxy version 2.0.25-6986403 2021/09/07 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
    prometheus-exporter

Available filters :
    [SPOE] spoe
    [COMP] compression
    [CACHE] cache
    [TRACE] trace

静态Pod规格（修正语法错误后） #

apiVersion: v1
kind: Pod
metadata:
  name: static-web
  labels:
    role: myrole
spec:
  containers:
    - name: web
      image: haproxy:v2.0.0
      volumeMounts:
        - mountPath: /usr/local/etc/haproxy/haproxy.cfg
          name: configfile
      ports:
        - name: web
          containerPort: 6443
          protocol: TCP
  volumes:
  - name: configfile
    hostPath:
      path: /etc/kubernetes/config/haproxy.cfg
      type: FileOrCreate

1. 定位haproxy高CPU的具体来源 #

• 用top或htop找到占用CPU的haproxy进程PID
• 执行perf top -p <haproxy_pid>，查看函数调用栈，确定是网络IO、正则匹配、Lua脚本还是SSL处理等模块导致高负载
• 执行strace -p <haproxy_pid> -c统计系统调用频率，排查是否存在异常频繁的epoll_wait、connect或read/write调用

2. 对比Docker与containerd的运行环境差异 #

• 检查容器cgroup配置：分别在两种环境下执行cat /proc/<pid>/cgroup，对比CPU调度策略、内存限制等配置
• 核对网络配置：查看容器的ip addr、route输出，确认网络模式、DNS配置是否一致，排查是否因DNS解析异常导致haproxy不断重试
• 验证挂载文件：确认haproxy.cfg在两种环境下的权限、内容完全一致，排查containerd下挂载是否存在文件读取异常

3. 排查haproxy配置与运行参数问题 #

• 检查haproxy.cfg是否存在逻辑错误：比如无限循环的健康检查、错误的后端服务器配置导致连接重试
• 临时关闭多线程：在配置中添加nbthread 1，观察CPU占用是否下降，排查多线程适配问题
• 启用调试日志：在配置中设置log stdout format raw debug，输出详细日志，查看是否有异常报错

4. 检查kubelet与containerd的交互逻辑 #

• 查看kubelet日志：journalctl -u kubelet，确认静态Pod创建过程是否存在重复启动容器等异常
• 查看containerd日志：journalctl -u containerd，排查容器启动、运行时的权限、资源隔离配置是否正确
• 对比容器启动参数：用docker inspect <container_id>和crictl inspect <container_id>查看启动参数，确认--cap-add、--sysctl等配置是否一致

5. 直接在containerd中测试haproxy #

• 跳过kubelet，用ctr命令直接启动容器，挂载相同配置文件，观察CPU情况：

  ctr images pull docker.io/library/haproxy:v2.0.0
  ctr run --rm -v /etc/kubernetes/config/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg docker.io/library/haproxy:v2.0.0 test-haproxy
  

  如果直接运行也出现高CPU，说明问题出在containerd与haproxy的适配，而非kubelet静态Pod逻辑

内容的提问来源于stack exchange，提问作者Raj