You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

AWS ParallelCluster Graviton2下slurmrestd崩溃API连接被拒

avatar
DevOps 小满
2026-9-3
AWS ParallelCluster Slurm Rest API 连接被拒故障排查

问题背景

参照 AWS Research Workshop 教程使用 AWS ParallelCluster 部署集群时，CloudFormation 资源部署完成后无法通过 Rest API 连接集群。
当前环境信息：

  • Slurm 版本：21.08.8
  • ParallelCluster 版本：3.1.4
  • 集群架构：Graviton2 ARM 处理器
  • 正常表现：可正常 SSH 登录头节点提交计算作业，slurmctld 服务运行稳定
  • 故障表现：通过 SageMaker notebook 访问头节点 8082 端口接口时始终提示连接被拒

现有配置

ParallelCluster 集群配置 YAML 

Region: ${REGION}
Image:
  Os: alinux2
SharedStorage:
  - Name: myebs
    StorageType: Ebs
    MountDir: /shared
    EbsSettings:
      VolumeType: gp2
      Size: 200
HeadNode:
  InstanceType: c6g.medium
  Networking:
    SubnetId: ${SUBNET_ID}
    ElasticIp: true 
  Ssh:
    KeyName: ${KEY_NAME}
  CustomActions:
    OnNodeConfigured:
      Script: ${POST_INSTALL_SCRIPT_LOCATION}
      Args:
        - ${POST_INSTALL_SCRIPT_ARGS_1}
        - ${POST_INSTALL_SCRIPT_ARGS_2}
        - ${POST_INSTALL_SCRIPT_ARGS_3}
        - ${POST_INSTALL_SCRIPT_ARGS_4}
        - ${POST_INSTALL_SCRIPT_ARGS_5}
        - ${POST_INSTALL_SCRIPT_ARGS_6}
        - ${POST_INSTALL_SCRIPT_ARGS_7}
        - ${POST_INSTALL_SCRIPT_ARGS_8}
        - ${POST_INSTALL_SCRIPT_ARGS_9}
  Iam:
    AdditionalIamPolicies:
      - Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
    S3Access:
      - EnableWriteAccess: true
        BucketName: '*'
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: q1
      CapacityType: ONDEMAND
      ComputeResources:
        - Name: cr1
          InstanceType: c6g.2xlarge
          MinCount: 0
          MaxCount: 20
          Efa:
            Enabled: false
      CustomActions:
        OnNodeConfigured:
          Script: ${POST_INSTALL_SCRIPT_LOCATION}
          Args:
            - ${POST_INSTALL_SCRIPT_ARGS_1}
            - ${POST_INSTALL_SCRIPT_ARGS_2}
            - ${POST_INSTALL_SCRIPT_ARGS_3}
            - ${POST_INSTALL_SCRIPT_ARGS_4}
            - ${POST_INSTALL_SCRIPT_ARGS_5}
            - ${POST_INSTALL_SCRIPT_ARGS_6}
            - ${POST_INSTALL_SCRIPT_ARGS_7}
            - ${POST_INSTALL_SCRIPT_ARGS_8}
            - ${POST_INSTALL_SCRIPT_ARGS_9}
      Iam:
        AdditionalIamPolicies:
          - Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
        S3Access:
          - EnableWriteAccess: true
            BucketName: '*'
      Networking:
        SubnetIds:
          - ${SUBNET_ID}
        AssignPublicIp: true
        PlacementGroup:
          Enabled: true

slurm.conf 配置内容 

#
# Example slurm.conf file. Please run configurator.html
# (in doc/html) to build a configuration file customized
# for your environment.
#
#
# slurm.conf file generated by configurator.html.
#
# See the slurm.conf man page for more information.
#
# CLUSTER SETTINGS
ClusterName=mypc6g2
SlurmUser=slurm
SlurmctldPort=6820-6829
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurm.state
SlurmdSpoolDir=/var/spool/slurmd
SwitchType=switch/none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ReconfigFlags=KeepPartState
#
# CLOUD CONFIGS OPTIONS
SlurmctldParameters=idle_on_node_suspend,power_save_min_interval=30,cloud_dns
CommunicationParameters=NoAddrCache
SuspendProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
ResumeProgram=/opt/parallelcluster/scripts/slurm/slurm_resume
ResumeFailProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
SuspendTimeout=120
ResumeTimeout=1800
PrivateData=cloud
ResumeRate=0
SuspendRate=0
#
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=180
UnkillableStepTimeout=180
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
MessageTimeout=60
#
# SCHEDULING, JOB, AND NODE SETTINGS
EnforcePartLimits=ALL
SchedulerType=sched/backfill
ProctrackType=proctrack/cgroup
MpiDefault=none
ReturnToService=1
TaskPlugin=task/affinity,task/cgroup
#
# TRES AND GPU CONFIG OPTIONS
GresTypes=gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU
#
# LOGGING
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
JobCompType=jobcomp/none
#
# WARNING!!! The slurm_parallelcluster.conf file included
# get updated by pcluster process, be careful
# when manually editing!
include slurm_parallelcluster.conf
# Enable jwt auth for Slurmrestd
AuthAltTypes=auth/jwt
#
## /opt/slurm/etc/slurm.conf
#
# ACCOUNTING
JobAcctGatherType=jobacct_gather/linux
JobAcctGatherFrequency=30
#
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=<IP ADDRESS> # cluster headnode's DNS
AccountingStorageUser=db_user
AccountingStoragePort=6839

服务排查结果

slurmrestd 服务状态

执行 sudo journalctl -u slurmrestd 输出日志如下：

-- Logs begin at Thu 2022-05-12 10:46:45 UTC, end at Thu 2022-05-26 03:00:38 UTC. --
May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug:  _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug:  slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug:  Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug:  NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.

执行 sudo systemctl status slurmrestd 输出如下：

● slurmrestd.service - Slurm restd daemon
   Loaded: loaded (/etc/systemd/system/slurmrestd.service; disabled; vendor preset: disabled)
   Active: failed (Result: signal) since Wed 2022-05-25 22:30:25 UTC; 4h 31min ago
 Main PID: 12872 (code=killed, signal=SEGV)

May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug:  _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug:  slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug:  Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug:  NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.

核心异常：slurmrestd 启动时触发段错误（SIGSEGV，信号11）直接崩溃，服务未正常运行，8082端口无进程监听，是连接被拒的直接原因。

slurmctld 服务状态

执行 systemctl status slurmctld 输出如下，服务运行正常，可周期性生成root用户的JWT认证token：

● slurmctld.service - Slurm controller daemon
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-05-25 22:30:25 UTC; 4h 33min ago
 Main PID: 12899 (slurmctld)
   CGroup: /system.slice/slurmctld.service
           ├─12899 /opt/slurm/sbin/slurmctld -D
           └─12903 slurmctld: slurmscriptd

May 26 00:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 03:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds

连通性测试情况

使用教程提供的Python代码测试接口连通性：

import requests
import json

slurm_openapi_ep = 'http://'+slurm_host+':8082/openapi/v3'
print(slurm_openapi_ep)
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.35'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.36'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.37'

_, get_headers = pcluster_helper.update_header_token()

print(get_headers)

resp_api = requests.get(slurm_openapi_ep, headers=get_headers)
print(resp_api)

if resp_api.status_code != 200:
    # This means something went wrong.
    print("Error" , resp_api.status_code)

with open('build/slurm_api.json', 'w') as outfile:
    json.dump(resp_api.json(), outfile)

print(json.dumps(resp_api.json(), indent=2))

执行时打印的请求地址与请求头如下：

http://<HEAD NODE IP ADDRESS HERE>:8082/openapi/v3
{'X-SLURM-USER-NAME': 'ec2-user', 'X-SLURM-USER-TOKEN': '<MY TOKEN HERE>', 'Content-type': 'application/x-www-form-urlencoded', 'Accept': 'application/json'}

最终抛出连接错误：

ConnectionError: HTTPConnectionPool(host='<HEAD NODE IP ADDRESS HERE>', port=8082): Max retries exceeded with url: /openapi/v3 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6151b9940>: Failed to establish a new connection: [Errno 111] Connection refused',))

根因与修复步骤

连接被拒的核心原因不是网络策略、token权限问题，而是slurmrestd服务启动即崩溃，未监听8082端口，按以下顺序修复：

  1. 修正slurmrestd配置加载路径错误
    从日志可见slurmrestd错误将自身配置文件/opt/slurm/etc/slurmrestd.conf作为Slurm主配置文件加载，解析节点参数时触发异常。编辑systemd服务文件/etc/systemd/system/slurmrestd.service，在Service段添加环境变量指定正确主配置路径：
    [Service]
Environment=SLURM_CONF=/opt/slurm/etc/slurm.conf
    保存后执行以下命令重载配置重启服务：
    sudo systemctl daemon-reload
sudo systemctl restart slurmrestd
  2. 修复ARM架构下slurmrestd段错误问题
    Slurm 21.08.8官方预编译的slurmrestd二进制在Graviton2（ARM64）平台存在已知编译优化导致的内存访问bug，如果修正路径后仍触发SIGSEGV崩溃：
    • 先停掉系统slurmrestd服务，手动前台启动查看详细报错：
      sudo systemctl stop slurmrestd
sudo SLURM_CONF=/opt/slurm/etc/slurm.conf /opt/slurm/sbin/slurmrestd -f /opt/slurm/etc/slurmrestd.conf -a rest_auth/jwt 0.0.0.0:8082
    • 若仍崩溃，通过后安装脚本重新编译slurmrestd组件，编译时添加-O0编译选项关闭高等级优化，替换原有二进制文件即可解决段错误问题。
  3. 验证端口监听状态
    服务正常启动后执行以下命令确认8082端口处于监听状态：
    sudo ss -tlnp
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.09.03 02:18:14