GCP实例组无法启动容器：Container-Optimized OS异常排查

问题背景 #

使用同一实例模板创建单个VM时，Container-Optimized OS能正常启动容器；但创建托管实例组（MIG）后，容器完全未启动，日志无容器启动相关记录。对比单个实例与MIG实例的gcloud compute instances describe输出，仅网络接口及托管相关配置存在差异。此外，SSH到单个实例会显示容器操作提示，而MIG实例无此提示。

实例模板定义如下：

creationTimestamp: '2022-11-09T03:25:29.896-08:00'
description: ''
id: '757769630202081478'
kind: compute#instanceTemplate
name: server-using-docker-hub-1
properties:
  canIpForward: false
  confidentialInstanceConfig:
    enableConfidentialCompute: false
  description: ''
  disks:
  - autoDelete: true
    boot: true
    deviceName: server-using-docker-hub
    index: 0
    initializeParams:
      diskSizeGb: '10'
      diskType: pd-balanced
      sourceImage: projects/cos-cloud/global/images/cos-stable-101-17162-40-20
    kind: compute#attachedDisk
    mode: READ_WRITE
    type: PERSISTENT
  keyRevocationActionType: NONE
  labels:
    container-vm: cos-stable-101-17162-40-20
  machineType: e2-micro
  metadata:
    fingerprint: 76mZ3i--POo=
    items:
    - key: gce-container-declaration
      value: |-
        spec:
          containers:
          - name: server-using-docker-hub-1
            image: docker.io/rinbar/kwik-e-mart
            env:
            - name: AWS_ACCESS_KEY_ID
              value: <redacted>
            - name: AWS_SECRET_ACCESS_KEY
              value: <redacted>
            - name: SECRET_FOR_SESSION
              value: <redacted>
            - name: SECRET_FOR_USER
              value: <redacted>
            - name: MONGODBURL
              value: mongodb+srv://<redacted>@cluster0.<redacted>.mongodb.net/kwik-e-mart
            - name: DEBUG
              value: server:*
            - name: PORT
              value: '80'
            stdin: false
            tty: false
          restartPolicy: Always
        # This container declaration format is not public API and may change without notice. Please
        # use gcloud command-line tool or Google Cloud Console to run Containers on Google Compute Engine.
    kind: compute#metadata
  networkInterfaces:
  - kind: compute#networkInterface
    name: nic0
    network: https://www.googleapis.com/compute/v1/projects/rons-project-364411/global/networks/default
    stackType: IPV4_ONLY
    subnetwork: https://www.googleapis.com/compute/v1/projects/rons-project-364411/regions/me-west1/subnetworks/default
  reservationAffinity:
    consumeReservationType: ANY_RESERVATION
  scheduling:
    automaticRestart: true
    onHostMaintenance: MIGRATE
    preemptible: false
    provisioningModel: STANDARD
  serviceAccounts:
  - email: 629139871582-compute@developer.gserviceaccount.com
    scopes:
    - https://www.googleapis.com/auth/devstorage.read_only
    - https://www.googleapis.com/auth/logging.write
    - https://www.googleapis.com/auth/monitoring.write
    - https://www.googleapis.com/auth/servicecontrol
    - https://www.googleapis.com/auth/service.management.readonly
    - https://www.googleapis.com/auth/trace.append
  shieldedInstanceConfig:
    enableIntegrityMonitoring: true
    enableSecureBoot: false
    enableVtpm: true
  tags:
    items:
    - http-server
selfLink: https://www.googleapis.com/compute/v1/projects/rons-project-364411/global/instanceTemplates/server-using-docker-hub-1

排查与解决步骤 #

• 验证MIG与实例模板的关联正确性
  执行gcloud compute instance-groups managed describe <mig-name>，确认MIG关联的实例模板名称与问题中的模板一致，避免使用了未包含容器声明的旧模板。

• 检查MIG实例的元数据完整性
  执行gcloud compute instances describe <mig-instance-name> --format="value(metadata.items)"，确认gce-container-declaration字段存在且内容与单个实例完全一致，未被MIG的自定义配置覆盖。

• 查看容器服务启动日志
  SSH到MIG实例后，执行以下命令排查容器服务状态：

  • 检查Docker服务状态：systemctl status docker
  • 检查containerd服务状态：systemctl status containerd
  • 查看容器启动相关日志：journalctl -u cos-container-shim，该日志会记录COS是否读取到容器声明、启动容器时的错误信息。

• 测试镜像拉取能力
  在MIG实例中手动执行docker pull docker.io/rinbar/kwik-e-mart，确认能正常拉取镜像，排除Docker Hub速率限制或网络策略导致的镜像拉取失败。

• 排查启动脚本干扰
  检查实例模板是否包含startup-script或其他自定义元数据，这些内容可能干扰COS的默认容器启动逻辑。若存在无关的启动脚本，移除后重新创建MIG实例测试。

内容的提问来源于stack exchange，提问作者Ron Inbar