在GCP容器优化OS中配置GPU监控的问题及解决方法

原配置尝试 #

用户参照Google Cloud官方GPU监控文档进行配置，修改部分代码后仍无法正常运行，以下是其使用的简化版cloud config内容：

- path: /etc/scripts/gpumonitor.sh
  permissions: "0644"
  owner: root
  content: |
    #!/bin/bash
    echo "Starting script..."
    sudo mkdir -p /etc/google
    cd /etc/google
    sudo git clone https://github.com/GoogleCloudPlatform/compute-gpu-monitoring.git
    echo "Downloaded Script..."
    echo "Starting up monitoring service..."
    sudo systemctl daemon-reload
    sudo systemctl --no-reload --now enable /etc/google/compute-gpu-monitoring/linux/systemd/google_gpu_monitoring_agent.service
    echo "Finished Script..."
- path: /etc/systemd/system/install-monitoring-gpu.service
  permissions: "0644"
  owner: root
  content: |
    [Unit]
    Description=Install GPU Monitoring
    Requires=install-gpu.service
    After=install-gpu.service

    [Service]
    User=root
    Type=oneshot
    RemainAfterExit=true
    ExecStart=/bin/bash /etc/scripts/gpumonitor.sh
    StandardOutput=journal+console
    StandardError=journal+console
runcmd:
    - systemctl start install-monitoring-gpu.service

最优解决方案 #

后续验证得出最优方案：构建包含GPU监控脚本的Docker容器，再按照Google Cloud容器优化OS GPU运行文档的指引，将GPU设备传入容器中运行。

内容的提问来源于stack exchange，提问作者bb197