在AWS EKS部署Quarkus Native应用遇exec format error问题

问题描述 #

在Mac M1设备上构建Quarkus-Camel Native应用，本地minikube环境运行正常，已确认镜像的os/arch为linux/arm64，但部署到AWS EKS时出现日志错误：exec ./application: exec format error。另有一个通过完全相同方式构建部署的应用可正常运行，已尝试AWS官方EKS Pod状态排障文档中的所有步骤，仍未找到问题原因。

相关信息 #

Pod状态（kubectl describe pod） #

Name:             integration-order-6dbccccb7d-qm9sb
Namespace:        development
Priority:         0
Service Account:  integration-order
Node:             ip-192-168-31-231.ap-southeast-2.compute.internal/192.168.31.231
Start Time:       Sun, 12 Mar 2023 21:49:05 +1100
Labels:           app.kubernetes.io/name=integration-order
                  app.kubernetes.io/version=1.0.0
                  pod-template-hash=6dbccccb7d
Annotations:      app.quarkus.io/build-timestamp: 2023-03-12 - 09:59:34 +0000
                  app.quarkus.io/commit-id: b888f2e29a05851b20d400386d70f3b269c3ed7a
Status:           Running
IP:               192.168.6.84
IPs:
  IP:           192.168.6.84
Controlled By:  ReplicaSet/integration-order-6dbccccb7d
Containers:
  integration-order:
    Container ID:   containerd://6cd1d5c2459d00ea17e28528a06e73846cdad78cb2f4d03696427f88f2107bd2
    Image:          fluentintegrations/integration-order:1.0.0
    Image ID:       docker.io/fluentintegrations/integration-order@sha256:eb42f2faf67cd61b0eb8e805aa6a46d5649db1b4aafd448319cf50bec6f476f2
    Port:           8087/TCP
    Host Port:      0/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 12 Mar 2023 21:49:22 +1100
      Finished:     Sun, 12 Mar 2023 21:49:22 +1100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 12 Mar 2023 21:49:06 +1100
      Finished:     Sun, 12 Mar 2023 21:49:06 +1100
    Ready:          False
    Restart Count:  2
    Liveness:       http-get http://:8087/q/health/live delay=30s timeout=10s period=30s #success=1 #failure=3
    Readiness:      http-get http://:8087/q/health/ready delay=45s timeout=10s period=45s #success=1 #failure=3
    Environment:
      KUBERNETES_NAMESPACE:  development (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rmczh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-rmczh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  20s               default-scheduler  Successfully assigned development/integration-order-6dbccccb7d-qm9sb to ip-192-168-31-231.ap-southeast-2.compute.internal
  Normal   Pulled     3s (x3 over 20s)  kubelet            Container image "fluentintegrations/integration-order:1.0.0" already present on machine
  Normal   Created    3s (x3 over 20s)  kubelet            Created container integration-order
  Normal   Started    3s (x3 over 20s)  kubelet            Started container integration-order
  Warning  BackOff    2s (x3 over 18s)  kubelet            Back-off restarting failed container

Maven构建命令 #

mvn clean package -Pnative -e \
-Dmaven.test.skip=true \
-Dquarkus.native.remote-container-build=true \
-Dquarkus.kubernetes.deploy=true \
-Dquarkus.container-image.push=true

application.properties配置 #

quarkus.http.port=8087

# quarkus kubernetes extension
quarkus.container-image.group=fluentintegrations
quarkus.container-image.name=integration-order
quarkus.kubernetes.namespace=development
quarkus.kubernetes.readiness-probe.initial-delay=45s
quarkus.kubernetes.readiness-probe.period=45s
quarkus.kubernetes.liveness-probe.initial-delay=30s
quarkus.kubernetes.liveness-probe.period=30s
quarkus.kubernetes.image-pull-policy=ifNotPresent
quarkus.kubernetes.service-type=LoadBalancer

排查建议 #

• 确认EKS节点架构：执行kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture查看节点架构。AWS EKS默认节点多为x86_64，arm64镜像无法在x86节点上直接运行，需构建多架构镜像或切换arm64节点组。
• 验证远程镜像架构：用docker manifest inspect fluentintegrations/integration-order:1.0.0确认远程仓库中镜像的实际架构，避免本地镜像与远程镜像不一致（比如push操作未成功覆盖旧镜像）。
• 强制指定构建架构：在Maven构建命令中添加-Dquarkus.native.container-build-args="--platform linux/arm64"，确保远程构建容器以arm64架构编译镜像，避免远程环境默认使用x86。
• 对比正常应用配置：将当前应用的pom.xml、构建命令、application.properties与正常运行的应用逐行对比，重点检查Quarkus Native相关配置（如基础镜像版本、插件参数），排查版本或依赖差异。
• 在EKS节点测试镜像：通过SSH或kubectl debug node/[节点名称] -it --image=busybox进入节点，直接运行docker run fluentintegrations/integration-order:1.0.0，获取更详细的错误输出。
• 检查镜像启动配置：用docker inspect fluentintegrations/integration-order:1.0.0 | grep -A 10 "Entrypoint"查看镜像的启动命令，或修改Pod的command为["/bin/sh"]进入容器，检查./application文件的权限和架构（用file application命令）。

内容的提问来源于stack exchange，提问作者eistropfen