Lifecycle Configuration文件无法自动终止Amazon SageMaker Studio实例
SageMaker代码编辑器闲置自动终止LCC配置无效问题排查
我正在配置生命周期配置(LCC)文件,为Amazon SageMaker代码编辑器实现闲置阈值触发的自动终止功能。按照相关方案操作后,能在SageMaker Studio界面看到并选择该LCC文件,但实例闲置1小时后并未关机(测试时设置为60秒也无效)。使用的LCC文件内容如下:
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # SPDX-License-Identifier: MIT-0 #!/bin/bash set -eux ASI_VERSION=0.3.0 # User variables [update as needed] IDLE_TIME_IN_SECONDS=60 # in seconds, change this to desired idleness time before app shuts down # System variables [do not change if not needed] CONDA_HOME=/opt/conda/bin LOG_FILE=/var/log/apps/app_container.log # Writing to app_container.log delivers logs to CW logs. SOLUTION_DIR=/var/tmp/auto-stop-idle # Do not use /home/sagemaker-user PYTHON_PACKAGE=sagemaker_code_editor_auto_shut_down-$ASI_VERSION.tar.gz PYTHON_SCRIPT_PATH=$SOLUTION_DIR/sagemaker_code_editor_auto_shut_down/auto_stop_idle.py # Installing cron sudo apt-get update -y sudo sh -c 'printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d' sudo apt-get install -y cron # Creating solution directory. sudo mkdir -p $SOLUTION_DIR # Downloading autostop idle Python package. echo "Downloading autostop idle Python package..." curl -LO --output-dir /var/tmp/ https://github.com/aws-samples/sagemaker-studio-apps-lifecycle-config-examples/releases/download/v$ASI_VERSION/$PYTHON_PACKAGE sudo $CONDA_HOME/pip install -U -t $SOLUTION_DIR /var/tmp/$PYTHON_PACKAGE # Touch file to ensure idleness timer is reset to 0 echo "Touching file to reset idleness timer" touch /opt/amazon/sagemaker/sagemaker-code-editor-server-data/data/User/History/startup_timestamp # Setting container credential URI variable to /etc/environment to make it available to cron sudo /bin/bash -c "echo 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI' >> /etc/environment" # Add script to crontab for root. echo "Adding autostop idle Python script to crontab..." echo "*/2 * * * * /bin/bash -ic '$CONDA_HOME/python $PYTHON_SCRIPT_PATH --time $IDLE_TIME_IN_SECONDS --region $AWS_DEFAULT_REGION >> $LOG_FILE'" | sudo crontab -
排查与修复步骤
- 确认cron服务运行状态:进入代码编辑器终端,执行
sudo service cron status检查cron是否正常运行。若未启动,执行sudo service cron start启动服务,并通过sudo systemctl enable cron设置开机自启。 - 查看日志定位错误:执行
cat /var/log/apps/app_container.log查看脚本执行日志,排查是否存在依赖缺失、路径错误或权限问题。 - 固定环境变量值:cron环境中可能无法正确获取
AWS_DEFAULT_REGION,修改crontab命令,直接替换为具体区域(如us-east-1),避免依赖环境变量。 - 验证脚本路径有效性:执行
ls /var/tmp/auto-stop-idle/sagemaker_code_editor_auto_shut_down/auto_stop_idle.py,确认脚本文件是否存在。若路径错误,检查LCC文件中的SOLUTION_DIR和PYTHON_SCRIPT_PATH配置。 - 优化cron命令格式:原命令中的
/bin/bash -ic可能导致环境变量加载异常,修改为直接调用conda的Python路径,并添加错误日志输出:*/2 * * * * $CONDA_HOME/python $PYTHON_SCRIPT_PATH --time $IDLE_TIME_IN_SECONDS --region us-east-1 >> $LOG_FILE 2>&1 - 检查IAM权限:确保SageMaker执行角色拥有
sagemaker:StopApp权限,否则脚本无法触发应用终止操作。 - 修复计时文件权限:执行
sudo chown -R sagemaker-user:sagemaker-user /opt/amazon/sagemaker/sagemaker-code-editor-server-data/data/User/History,确保闲置计时文件可被正确读写更新。
内容的提问来源于stack exchange,提问作者Cyrus Mohammadian
相关产品推荐
相关产品推荐

