GCE实例被MIG自动重建求助：CPU高致健康检查失败

我们的Managed Instance Group（MIG）仅部署1台GCE实例（当前应用不支持自动扩缩容），健康检查策略为TCP端口22，间隔30秒，连续3次失败则判定实例不健康。近期多次无规律出现健康检查失败，导致实例被自动重建，重建前VM CPU利用率高达~99%。

监控与重建日志 #

CPU利用率超标告警日志 #

{
  "insertId": "vwsb19f4r1p5f",
  "labels": {
    "activity_type_name": "ViolationOpenEventv1",
    "policy_id": "17302346025290670292",
    "resource_name": "gcp-project-name gcp-gce-name",
    "started_at": "1674896821",
    "terse_message": "CPU utilization for gcp-project-name gcp-gce-name with metric labels {instance_name=gcp-gce-name} *is above the threshold of 0.850 with a value of 0.989*.",
    "verbose_message": "CPU utilization for gcp-project-name gcp-gce-name with metric labels {instance_name=gcp-gce-name} *is above the threshold of 0.850 with a value of 0.989*.",
    "violation_id": "0.mt6pqza8uq6c"
  },
  "logName": "projects/gcp-project-name/logs/monitoring.googleapis.com%2FViolationOpenEventv1",
  "receiveTimestamp": "2023-01-28T09:07:01.311681568Z",
  "resource": {},
  "timestamp": "2023-01-28T09:07:01Z"
}

日志触发后，该GCE在Stackdriver日志浏览器中无后续输出，10-15分钟后触发实例重建，日志如下：

MIG自动重建实例日志 #

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "message": "Instance Group Manager 'projects/1048357249635/zones/europe-north1-b/instanceGroupManagers/gcp-project-name-managed-instance-group' initiated recreateInstance on instance 'projects/1048357249635/zones/europe-north1-b/instances/gcp-project-name-bbjn'. *Reason: Instance eligible for autohealing: instance unhealthy.*"
    },
    "authenticationInfo": {
      "principalEmail": "system@google.com"
    },
    "serviceName": "compute.googleapis.com",
    "methodName": "compute.instances.repair.recreateInstance",
    "resourceName": "projects/gcp-project-name/zones/europe-north1-b/instances/gcp-project-name-bbjn",
    "request": {
      "@type": "type.googleapis.com/compute.instances.repair.recreateInstance"
    }
  }
}

已尝试的无效方案 #

• 将实例配置从n1-highcpu-4升级至n1-highcpu-8
• 修改健康检查策略：间隔改为1分钟，连续失败次数改为5次

额外观察到的细节 #

• 问题发生前实例完全无响应，无法建立IAP隧道或通过控制台SSH连接
• 无法进一步排查根因（因实例无响应）
• 偶然发现ETL作业的子进程占用大量CPU，进程命令：

  /usr/lib64/google-cloud-sdk/platform/bundledpythonunix/bin/python3 /usr/lib64/google-cloud-sdk/bin/bootstrapping/bq.py query --use_legacy_sql=false
  

• 问题发生时，BQ在途查询数和插槽利用率显著降低，推测实例无法连接BQ API导致任务堆积
• 该GCE部署在项目的私有VPC中
• 自定义监控任务在问题发生时停滞，无法将数据同步至GCS

求助 #

恳请有类似经验或解决方案的同行提供帮助。

内容的提问来源于stack exchange，提问作者Souvik Das