使用Terraform升级kube-prometheus-stack遇两类错误，如何修复？

升级版本信息 #

错误1：Webhook证书认证失败 #

错误日志：

failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.infra.svc:443/admission-prometheusrules/mutate?timeout=30s": x509: certificate signed by unknown authority

修复步骤： #

• 方案一：重启operator并重新生成证书

  1. 删除operator的webhook证书Secret：

     kubectl delete secret -n infra kube-prometheus-stack-operator-webhook-cert
     

  2. 重启operator部署，触发证书重新生成：

     kubectl rollout restart deployment kube-prometheus-stack-operator -n infra
     

• 方案二：手动更新Webhook的CA Bundle

  1. 获取集群根CA证书内容：

     kubectl get secret -n kube-system default-token-$(kubectl get secret -n kube-system | grep default-token | awk '{print $1}') -o jsonpath='{.data.ca\.crt}'
     

  2. 用获取到的CA字符串更新MutatingWebhookConfiguration：

     kubectl patch mutatingwebhookconfiguration prometheusrulemutate.monitoring.coreos.com \
       -p '{"webhooks":[{"name":"prometheusrulemutate.monitoring.coreos.com","clientConfig":{"caBundle":"<替换为上一步获取的CA字符串>"}}]}'
     

错误2：Deployment/DaemonSet Selector字段不可变 #

错误日志：

Error: cannot patch "kube-prometheus-stack-prometheus-node-exporter" with kind DaemonSet: DaemonSet.apps "kube-prometheus-stack-prometheus-node-exporter" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"kube-prometheus-stack", "app.kubernetes.io/name":"prometheus-node-exporter"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable && cannot patch "kube-prometheus-stack-kube-state-metrics" with kind Deployment: Deployment.apps "kube-prometheus-stack-kube-state-metrics" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"kube-prometheus-stack", "app.kubernetes.io/name":"kube-state-metrics"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

修复步骤： #

Kubernetes中Deployment和DaemonSet的spec.selector字段是不可变的，新旧Chart版本的Selector标签规范不一致导致冲突，解决方法如下：

1. 删除冲突的无状态资源（node-exporter和kube-state-metrics均为无状态，删除重建不会丢失数据）：

   kubectl delete daemonset kube-prometheus-stack-prometheus-node-exporter -n infra
   kubectl delete deployment kube-prometheus-stack-kube-state-metrics -n infra
   

2. 重新执行Terraform部署或Helm升级：

   terraform apply
   # 或使用Helm命令
   # helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack -n infra -f your-values.yaml
   

注意：如果不想删除资源，可以临时修改Helm values，让新版本的Selector与旧版本保持一致，但这只是临时方案，长期来看建议重建以对齐新版本的标签规范。

内容的提问来源于stack exchange，提问作者Abdullah Khawer