Vertex AI托管笔记本创建定时执行计划报错：无法提交计划

问题描述 #

在Vertex AI Workbench托管笔记本中，交互式执行和**单次执行（One-time execution）均正常，但创建定时重复执行（Schedule-based recurring executions）**时，提交等待约30秒后报错：2: Unknown Error.: Unable to submit schedule，执行参数与单次执行完全一致。

完整错误日志如下：

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 2,
      "message": "Error 0: googleapi: Error 503: The service is currently unavailable.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.DebugInfo",
    "detail": "[ORIGINAL ERROR] RPC::DEADLINE_EXCEEDED: Deadline exceeded while waiting for channels to become available, and/or be able to accept RPCs. Channel count = 0. Queue size = 0. Deadline=28.992677276s; Extensible Stubs capped the outgoing deadline: see go/stubs-longest-allowed-deadline; \ncom.google.net.rpc3.RpcException: DEADLINE_EXCEEDED: Deadline exceeded while waiting for channels to become available, and/or be able to accept RPCs. Channel count = 0. Queue size = 0. Deadline=28.992677276s; Extensible Stubs capped the outgoing deadline: see go/stubs-longest-allowed-deadline; \n\tat com.google.apphosting.admin.mixer.actions.CreateApplicationAction.checkRpcPresentStatus(CreateApplicationAction.java:194)\n\tSuppressed: com.google.common.labs.concurrent.LabsFutures$LabeledExecutionException: GraphFuture{key=@com.google.apphosting.admin.mixer.actions.CreateApplicationAction$CreateReturnMessage com.google.protobuf.Message} failed: com.google.net.rpc3.RpcException: DEADLINE_EXCEEDED: Deadline exceeded while waiting for channels to become available, and/or be able to accept RPCs. Channel count = 0. Queue size = 0. Deadline=28.992677276s; Extensible Stubs capped the outgoing deadline: see go/stubs-longest-allowed-deadline; \n"
  }
]
, backendError"
    },
    "authenticationInfo": {
      "principalEmail": "xxxxxxxxxx@developer.gserviceaccount.com",
      "serviceAccountDelegationInfo": [
        {
          "firstPartyPrincipal": {
            "principalEmail": "xxxxxxxxxx@gcp-sa-notebooks.iam.gserviceaccount.com"
          }
        }
      ]
    },
    "requestMetadata": {
      "requestAttributes": {},
      "destinationAttributes": {}
    },
    "serviceName": "notebooks.googleapis.com",
    "methodName": "google.cloud.notebooks.v1.NotebookService.CreateSchedule",
    "resourceName": "projects/MY-PROJECT/locations/europe-west4/schedules/test_rnn_aria__1673534440196",
    "resourceLocation": {
      "currentLocations": [
        "europe-west4"
      ]
    }
  },
  "insertId": "184n01ud3oul",
  "resource": {
    "type": "audited_resource",
    "labels": {
      "method": "google.cloud.notebooks.v1.NotebookService.CreateSchedule",
      "service": "notebooks.googleapis.com",
      "project_id": "MY-PROJECT"
    }
  },
  "timestamp": "2023-01-12T14:42:42.563449708Z",
  "severity": "ERROR",
  "logName": "projects/MY-PROJECT/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "id": "projects/MY-PROJECT/locations/europe-west4/operations/operation-XXXXXXXXXX",
    "producer": "notebooks.googleapis.com",
    "last": true
  },
  "receiveTimestamp": "2023-01-12T14:42:43.526355300Z"
}

排查与解决步骤 #

从日志中的DEADLINE_EXCEEDED和503: The service is currently unavailable可以看出，问题核心是服务端超时或临时不可用，结合场景可按以下步骤排查：

• 检查目标区域服务健康状态：登录GCP控制台，查看Notebooks服务在europe-west4区域的运行状态，确认是否存在临时故障或维护。
• 验证服务账号权限：
  • 确认执行定时任务的服务账号（gcp-sa-notebooks.iam.gserviceaccount.com）拥有notebooks.schedules.create权限，同时需具备Cloud Scheduler、Cloud Functions（若涉及）的相关权限，以及笔记本输出存储桶的读写权限。
  • 在项目IAM页面检查该服务账号是否被授予Notebooks Admin角色，或包含上述权限的自定义角色。
• 尝试用gcloud命令行创建定时任务：界面操作的超时限制可能较严格，使用命令行工具可配置更灵活的参数，示例命令：

  gcloud notebooks schedules create SCHEDULE_NAME \
    --location=europe-west4 \
    --vm-image-project=deeplearning-platform-release \
    --vm-image-family=common-cpu \
    --notebook-file=gs://PATH/TO/YOUR/NOTEBOOK.ipynb \
    --schedule="0 1 * * *"
  

• 检查VPC网络配置：
  • 若笔记本部署在自定义VPC中，确保防火墙规则允许出站流量访问Notebooks API和Cloud Scheduler API。
  • 若启用了私有服务访问，确认Notebooks API端点在VPC内可达。
• 重试或切换区域测试：
  • 临时服务不可用通常是短暂的，间隔一段时间后重新提交定时任务。
  • 若条件允许，切换到其他稳定区域（如europe-west1）创建笔记本和定时任务，排除区域特定问题。

内容的提问来源于stack exchange，提问作者MariusPontmercy