Jenkins X 3基于Terraform在EKS部署的安装故障求助

我是Jenkins X新手，使用Terraform结合AWS Secrets Manager在EKS上安装Jenkins X 3时遇到问题。已向内部Bitbucket集群仓库jx3-eks-asm提交测试提交，但错误依旧，推测是boot pod运行失败导致。此前通过调整Kuberhealthy及其他Helm Chart版本解决过部分问题，已确认满足terraform、aws-cli、aws-iam-authenticator和wget等前置要求。禁用Kuberhealthy后健康检查命令无法运行，将其版本更新至0.0.90左右解决了一个问题，但现在出现新错误。

核心错误（jx-git-operator命名空间boot pod日志） #

error validating "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-bot-token-kuberhealthycheck.yaml": error validating data: ValidationError(KuberhealthyCheck.spec.podSpec.containers[0]): unknown field "restartPolicy" in io.github.comcast.v1.KuberhealthyCheck.spec.podSpec.containers; if you choose to ignore these errors, turn validation off with --validate=false
error validating "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-kuberhealthycheck.yaml": error validating data: ValidationError(KuberhealthyCheck.spec.podSpec.containers[0]): unknown field "restartPolicy" in io.github.comcast.v1.KuberhealthyCheck.spec.podSpec.containers; if you choose to ignore these errors, turn validation off with --validate=false
error validating "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-kuberhealthycheck.yaml": error validating data: ValidationError(KuberhealthyCheck.spec.podSpec.containers[0]): unknown field "restartPolicy" in io.github.comcast.v1.KuberhealthyCheck.spec.podSpec.containers; if you choose to ignore these errors, turn validation off with --validate=false
make[1]: Leaving directory '/workspace/source'
make[1]: *** [versionStream/src/Makefile.mk:324: kubectl-apply] Error 1
error: failed to regenerate: failed to regenerate phase 1: failed to run 'make regen-phase-1 NEW_CLUSTER=false' command in directory '.', output: ''
make: *** [versionStream/src/Makefile.mk:269: regen-check] Error 1

环境版本信息 #

kubectl版本 #

kubectl version
Client Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:38:50Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.13-eks-0a21954", GitCommit:"6305d65c340554ad8b4d7a5f21391c9fa34932cb", GitTreeState:"clean", BuildDate:"2023-04-15T00:33:45Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}

jx健康检查状态（jx health status -A） #

jx health status -A
                          NAME                     NAMESPACE                        STATUSERROR MESSAGE
                     daemonset                  kuberhealthy                   OK
                    deployment                  kuberhealthy                   OK
           dns-status-internal                  kuberhealthy                   OK
                    jx-install               jx-git-operator                         ERRORlatest boot job jx-boot-1997a686-37fb-4704-af81-8505a7518877 has been running for more than 30m0s, it could be stuck
                                                                                          latest boot job jx-boot-1997a686-37fb-4704-af81-8505a7518877 has a failed run
                 jx-pod-status                  kuberhealthy                         ERRORpod: jenkins-x-chartmuseum-6cd76fd747-4hwrc in namespace: jx is in pod status phase Pending
                                                                                          pod: jx-build-controller-b6bdc5b6d-ch2xh in namespace: jx is in pod status phase Pending
                                                                                          pod: jx-pipelines-visualizer-7b685f9c79-bdx5k in namespace: jx is in pod status phase Pending
                                                                                          pod: lighthouse-foghorn-cff4fd4f5-r76kr in namespace: jx is in pod status phase Pending
                                                                                          pod: lighthouse-keeper-57dfddb5f9-plfjn in namespace: jx is in pod status phase Pending
                                                                                          pod: lighthouse-webhooks-85f985b664-9wdgx in namespace: jx is in pod status phase Pending
                                                                                          pod: nexus-nexus-776c96f565-vxp42 in namespace: jx is in pod status phase Pending
                    jx-secrets                  kuberhealthy                         ERRORERROR, Secrets Manager can't find the specified secret.
                                                                                          ERROR, Secrets Manager can't find the specified secret.
                                                                                          ERROR, Secrets Manager can't find the specified secret.
                                                                                          ERROR, Secrets Manager can't find the specified secret.
                                                                                          ERROR, Secrets Manager can't find the specified secret.
                                                                                          ERROR, Secrets Manager can't find the specified secret.
                                                                                          ERROR, Secrets Manager can't find the specified secret.

秘钥验证结果（jx secret verify） #

jx secret verify
SECRET                                       STATUS
jx-production/tekton-container-registry-auth key tekton-container-registry-auth missing properties: .dockerconfigjson
jx-staging/tekton-container-registry-auth    key tekton-container-registry-auth missing properties: .dockerconfigjson
jx/jenkins-maven-settings                    key jx-maven-settings missing properties: settingsXml, securityXml
jx/jenkins-x-chartmuseum                     valid: jx-admin-user/BASIC_AUTH_PASS, jx-admin-user/BASIC_AUTH_USER
jx/jx-basic-auth-htpasswd                    key jx-basic-auth-htpasswd missing properties: token
jx/jx-basic-auth-user-password               valid: jx-basic-auth-user/password, jx-basic-auth-user/username
jx/lighthouse-oauth-token                    key lighthouse-oauth missing properties: token
jx/nexus                                     valid: jx-admin-user/password
jx/tekton-container-registry-auth            key tekton-container-registry-auth missing properties: .dockerconfigjson
jx/tekton-git                                key jx-pipeline-user missing properties: token, username

排查建议 #

1. 修复KuberhealthyCheck资源字段错误

   • 错误显示restartPolicy字段被放在了KuberhealthyCheck.spec.podSpec.containers[0]下，该字段实际属于podSpec层级（而非容器层级）。需要修改对应的3个yaml文件，将restartPolicy移到spec.podSpec根节点下：
     调整前：

     spec:
       podSpec:
         containers:
         - name: check
           restartPolicy: OnFailure
     

     调整后：

     spec:
       podSpec:
         restartPolicy: OnFailure
         containers:
         - name: check
     

2. 解决Secrets Manager秘钥缺失问题

   • 确认Terraform是否正确创建了所需的Secrets Manager资源，或是否手动创建了对应秘钥条目
   • 核对秘钥名称、ARN与Jenkins X配置是否一致
   • 检查boot pod的服务账号是否具备访问Secrets Manager的IAM权限

3. 处理Pending Pod问题

   • 执行kubectl describe pod <pod-name> -n jx查看Pending Pod的具体事件，排查资源不足、镜像拉取失败、存储卷挂载错误等原因
   • 结合秘钥验证结果，优先解决tekton-container-registry-auth缺失问题，确保镜像拉取秘钥配置正确

4. 调整Kuberhealthy版本兼容性

   • 当前使用的Kuberhealthy 0.0.90版本可能与Jenkins X 3配置模板不兼容，尝试匹配Jenkins X版本流推荐的Kuberhealthy版本，确保CRD定义与配置文件字段一致

内容的提问来源于stack exchange，提问作者cnu