使用自定义Docker镜像连接GCP AI平台Notebook实例遇连接拒绝

Let’s break down the issues you’re facing and walk through actionable fixes based on the logs you shared:

Key Observations from Your Logs #

• The initial Connection refused on SSH ties back to permission errors in the serial logs (PermissionDenied for the guest agent) and clock synchronization issues (Clock Unsynchronized). GCP’s authentication systems rely on accurate time, so a misaligned clock can break token validation.
• Later, when you could SSH but port 8080 failed, it means the Jupyter server (or notebook service) didn’t start correctly inside your custom image.
• The pull access denied error when launching the image as a VM confirms that the instance’s service account lacks permissions to pull your custom image from GCR.

Step-by-Step Fixes #

1. Grant Correct Permissions to the Notebook Service Account #

AI Platform Notebooks uses a dedicated service account to pull images and manage instances. You need to give it access to your GCR repository:

• First, get your project number:

  PROJECT_NUMBER=$(gcloud projects describe $PROJECT --format="value(projectNumber)")
  

• Then add the Storage Object Viewer role to the notebook service account (this lets it pull images from GCR):

  gcloud projects add-iam-policy-binding $PROJECT \
    --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-notebooks.iam.gserviceaccount.com" \
    --role="roles/storage.objectViewer"
  

2. Fix Clock Synchronization #

A desynchronized clock can cause permission checks to fail even if your roles are correct. Here’s how to resolve it:

• On the running instance, restart the NTP service to sync the clock:

  sudo systemctl restart ntpd
  # For newer Debian/Ubuntu systems using chrony:
  sudo systemctl restart chronyd
  

• To prevent this in future instances, add this step to your Dockerfile to ensure NTP is configured on image startup:

  FROM gcr.io/deeplearning-platform-release/base-cpu:latest
  # Ensure NTP is enabled and running
  RUN apt-get update && apt-get install -y ntp && \
      sudo systemctl enable ntpd
  

3. Preserve the Base Image’s Startup Logic #

The base Deep Learning Container has a dedicated startup script that launches the Jupyter server and configures the instance. If your Dockerfile overrides CMD or ENTRYPOINT, you’ll break this flow.

• Make sure your Dockerfile doesn’t replace the base image’s entrypoint. For example:

  FROM gcr.io/deeplearning-platform-release/base-cpu:latest
  # Your custom setup (install packages, etc.)
  RUN apt-get update && apt-get install -y git
  # DO NOT add CMD/ENTRYPOINT here unless you explicitly call the base script:
  # CMD ["/opt/deeplearning/start.sh"]
  

• If you need to add custom startup steps, create a script that calls the base script at the end, then set that as your entrypoint.

4. Diagnose Port Forwarding Failures #

Once you can SSH into the instance, check if the Jupyter service is running:

• List active Jupyter processes:

  ps aux | grep jupyter
  

• Check the Jupyter log for startup errors:

  cat /var/log/jupyter.log
  

• Common issues here are missing dependencies in your custom image that the base image relies on for Jupyter to start.

5. Verify Image Pull Permissions #

If you still get pull access denied when launching instances, test the service account’s ability to pull the image locally:

• Create a key for the notebook service account:

  gcloud iam service-accounts keys create key.json \
    --iam-account=service-${PROJECT_NUMBER}@gcp-sa-notebooks.iam.gserviceaccount.com
  

• Activate the service account and try pulling the image:

  gcloud auth activate-service-account --key-file=key.json
  docker pull gcr.io/${PROJECT}/my-custom-image:latest
  

• If this fails, double-check that the service account has the correct roles in IAM and that the GCR repository’s permissions are set to allow access.

Wrap-Up #

Most of these issues stem from missing permissions for the notebook service account or broken startup logic in your custom image. The clock synchronization problem was likely a contributing factor that triggered the initial permission errors.

内容的提问来源于stack exchange，提问作者divBy0