K8s升级至1.23后Istio1.8.3 Pod就绪探针失败报CRD查询错误

• 早期通过Rancher UI部署Istio 1.8.3版本，初始运行状态正常：Istio Pod、Ingress Gateway Pod均可稳定运行，业务应用可通过Istio正常对外提供服务
• 集群升级阶段：先后将Kubernetes集群版本从1.21升级至1.22，再升级至1.23版本，升级完成后重启kubelet时，Istio Pod未出现启动异常
• 故障触发：后续因其他问题对节点执行重启操作后，Istio Pod随节点重启，当前Pod状态显示为Running，但Readiness Probe（就绪探针）持续检测失败
• 核心报错信息：

failed to list CRDs: the server could not find the requested resource

Istiod Pod完整运行日志 #

stream logs failed container "discovery" in pod "istiod-5fbc9568cd-qgqkk" is waiting to start: ContainerCreating for istio-system/istiod-5fbc9568cd-qgqkk (discovery)
2022-06-27T05:35:32.772949Z    info    FLAG: --log_rotate_max_age="30"
2022-06-27T05:35:32.772952Z    info    FLAG: --log_rotate_max_backups="1000"
2022-06-27T05:35:32.772955Z    info    FLAG: --log_rotate_max_size="104857600"
2022-06-27T05:35:32.772958Z    info    FLAG: --log_stacktrace_level="default:none"
2022-06-27T05:35:32.772963Z    info    FLAG: --log_target="[stdout]"
2022-06-27T05:35:32.772971Z    info    FLAG: --mcpInitialConnWindowSize="1048576"
2022-06-27T05:35:32.772974Z    info    FLAG: --mcpInitialWindowSize="1048576"
2022-06-27T05:35:32.772977Z    info    FLAG: --mcpMaxMsgSize="4194304"
2022-06-27T05:35:32.772980Z    info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-06-27T05:35:32.772982Z    info    FLAG: --monitoringAddr=":15014"
2022-06-27T05:35:32.772985Z    info    FLAG: --namespace="istio-system"
2022-06-27T05:35:32.772988Z    info    FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2022-06-27T05:35:32.772999Z    info    FLAG: --plugins="[authn,authz,health]"
2022-06-27T05:35:32.773002Z    info    FLAG: --profile="true"
2022-06-27T05:35:32.773005Z    info    FLAG: --registries="[Kubernetes]"
2022-06-27T05:35:32.773008Z    info    FLAG: --resync="1m0s"
2022-06-27T05:35:32.773011Z    info    FLAG: --secureGRPCAddr=":15012"
2022-06-27T05:35:32.773013Z    info    FLAG: --tlsCertFile=""
2022-06-27T05:35:32.773016Z    info    FLAG: --tlsKeyFile=""
2022-06-27T05:35:32.773018Z    info    FLAG: --trust-domain=""
2022-06-27T05:35:32.801976Z    info    klog    Config not found: /var/run/secrets/remote/config[]
2022-06-27T05:35:32.803516Z    info    initializing mesh configuration ./etc/istio/config/mesh
2022-06-27T05:35:32.804499Z    info    mesh configuration: {
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "envoyAccessLogService": {

        },
        "envoyMetricsService": {

        },
        "proxyMetadata": {
            "DNS_AGENT": ""
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": true
}
2022-06-27T05:35:32.804516Z    info    version: 1.8.3-e282a1f927086cc046b967f0171840e238a9aa8c-Clean
2022-06-27T05:35:32.804699Z    info    flags: 
2022-06-27T05:35:32.804706Z    info    initializing mesh networks
2022-06-27T05:35:32.804877Z    info    mesh networks configuration: {
   "networks": {
   }
}
2022-06-27T05:35:32.804938Z    info    initializing mesh handlers
2022-06-27T05:35:32.804949Z    info    initializing controllers
2022-06-27T05:35:32.804952Z    info    No certificates specified, skipping K8S DNS certificate controller
2022-06-27T05:35:32.814002Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:33.816596Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:35.819157Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:39.821510Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:47.823675Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:36:03.827023Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:36:35.829441Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:37:35.831758Z    error    kube    failed to list CRDs: the server could not find the requested resource

故障本质是版本兼容性问题：

• Kubernetes 1.22版本正式移除了apiextensions.k8s.io/v1beta1版本的CRD API，仅保留稳定版apiextensions.k8s.io/v1接口
• Istio 1.8.3发布时间早于K8s 1.22版本，其discovery组件默认调用旧的v1beta1接口查询CRD列表，接口被移除后直接返回资源不存在的错误
• 集群升级后重启kubelet未触发故障，是因为当时运行的旧Istio Pod已在内存中加载了CRD缓存，不需要重新全量拉取CRD列表；节点重启后Pod重建，启动流程中需要重新调用API拉取CRD数据，旧接口不可用直接导致启动失败，就绪探针检测不通过。

• 临时应急方案：若短时间无法升级Istio，可手动将集群内所有Istio相关CRD转换为v1版本存储，同时修改Istiod绑定的集群角色，授予v1版本CRD的读写权限，重启Istiod Pod即可恢复服务
• 长期稳定方案：将Istio版本升级至1.10及以上版本，Istio从1.10版本开始正式兼容Kubernetes 1.22+版本，默认使用v1版本CRD接口，不存在该兼容性问题

升级Istio前必须全量备份所有Istio自定义资源（包括Gateway、VirtualService、DestinationRule、AuthorizationPolicy等），避免升级过程中配置丢失影响业务正常运行。

内容的提问来源于stack exchange，提问作者RajNikhil Marpu