LinkerD sidecar报证书过期但linkerd check显示证书有效问题

• LinkerD通过cert-manager安装部署，已为所有linkerd命名空间配置对应颁发者，证书支持自动续期
• 执行./linkerd check命令未返回任何错误
• 颁发者linkerd-trust-anchor、webhook-issuer均为有效状态
• 证书linkerd-identity-issuer、linkerd-policy-validator、linkerd-proxy-injector、linkerd-sp-validator均有效、未过期

• 尝试应用策略A时收到服务端报错，信息如下：

Error from server (InternalError): error when creating ".\templates\servers.yaml": Internal error occurred: failed calling webhook "linkerd-policy-validator.linkerd.io": Post "https://linkerd-policy-validator.linkerd.svc:443/?timeout=10s": x509: certificate has expired or is not yet valid

• sidecar容器日志中存在Failed to connect error=invalid certificate: CertExpired错误，日志原文如下：

[     0.000771s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.001299s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.001311s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.001313s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.001314s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.001316s]  INFO ThreadId(01) linkerd2_proxy: Local identity is gpproxyserver-tlm.rainbowstaging-main.serviceaccount.identity.linkerd.cluster.local
[     0.001321s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001323s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003971s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=192.168.163.207:8090}: rustls::session: Sending fatal alert BadCertificate
[     0.004033s]  WARN ThreadId(01) policy:watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=192.168.163.207:8090}: linkerd_reconnect: Failed to connect error=invalid certificate: CertExpired
[     0.032277s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: gpproxyserver-tlm.rainbowstaging-main.serviceaccount.identity.linkerd.cluster.local
[    24.056844s]  WARN ThreadId(01) inbound:server{port=8000}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=192.168.163.207:8086}: rustls::session: Sending fatal alert BadCertificate
[    24.057004s]  WARN ThreadId(01) inbound:server{port=8000}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=192.168.163.207:8086}: linkerd_reconnect: Failed to connect error=invalid certificate: CertExpired
[    24.074130s]  WARN ThreadId(01) outbound:server{orig_dst=10.98.31.55:5672}: rustls::session: Sending fatal alert BadCertificate
[    24.132043s]  INFO ThreadId(01) inbound:server{port=8000}: linkerd_proxy_http::upgrade: tcp duplex error: client: Broken pipe (os error 32)
[  2117.365594s]  WARN ThreadId(01) inbound:server{port=8000}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=192.168.163.207:8086}: rustls::session: Sending fatal alert BadCertificate
[  2117.369126s]  WARN ThreadId(01) inbound:server{port=8000}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=192.168.163.207:8086}: linkerd_reconnect: Failed to connect error=invalid certificate: CertExpired
[  2117.377137s]  WARN ThreadId(01) outbound:server{orig_dst=10.98.31.55:5672}: rustls::session: Sending fatal alert BadCertificate
[  2117.388693s]  INFO ThreadId(01) inbound:server{port=8000}: linkerd_proxy_http::upgrade: tcp duplex error: client: Broken pipe (os error 32)

按优先级依次排查以下问题：

1. 节点时间不同步
   该类certificate has expired or is not yet valid报错绝大多数由节点时间偏移导致。在所有K8s节点（含控制面、工作节点）执行timedatectl或date命令校验当前系统时间，确认与标准UTC时间偏差不超过60秒。若存在时间偏移，先修复节点NTP时间同步服务，时间恢复正常后证书校验错误会自动消失。
2. 控制平面Pod未加载最新续期证书
   cert-manager自动续期Secret中的证书后，已运行的LinkerD控制平面Pod不会自动热更新挂载的证书文件，会一直使用启动时加载的旧证书直到重启。
   执行以下命令校验Secret中存储的证书实际有效期：

   # 校验policy-validator证书有效期
   kubectl -n linkerd get secret linkerd-policy-validator-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates
   # 校验identity、dst、policy等控制平面服务使用的证书有效期
   kubectl -n linkerd get secret linkerd-identity-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates
   

   若Secret中证书为最新未过期状态，直接滚动重启linkerd命名空间下所有工作负载即可加载新证书：

   kubectl -n linkerd rollout restart deploy
   kubectl -n linkerd rollout restart statefulset
   

3. Webhook配置的CA bundle与当前信任根不匹配
   执行以下命令查看webhook配置中写入的CA证书：

   kubectl get mutatingwebhookconfiguration linkerd-policy-validator-webhook-config -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -dates
   

   若返回的证书已过期或与当前linkerd-trust-anchor证书不一致，重新执行LinkerD信任根更新流程，同步最新CA到所有webhook配置中。

内容的提问来源于stack exchange，提问作者Bouklan