Slurm 24.11.5中/slurm/{version}/job/{job_id} API无效JobID错误排查求助

环境：Slurm 24.11.5版本，已部署slurmrestd服务。现象：调用/slurm/{version}/job/{job_id}时始终返回错误"Invalid JobID={job_id} rejected"，但调用/slurm/{version}/jobs可正常获取对应作业的正确信息。

详细操作示例 #

1. 提交作业并确认状态

# sbatch job.slurm
Submitted batch job 67135476
# squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          67135476  compute1 one_job_     root  R       0:04      2 shield-c[1-2]

2. 单作业查询API响应

{
  "errors": [
    {
      "description": "Invalid JobID=67135476 rejected",
      "error_number": 2017,
      "error": "Invalid job id specified",
      "source": "op_handler_job"
    }
  ],
  "meta": {
    "plugin": {
      "type": "openapi/slurmctld",
      "name": "Slurm OpenAPI slurmctld",
      "data_parser": "data_parser/v0.0.42",
      "accounting_storage": "accounting_storage/slurmdbd"
    },
    (...)
    "slurm": {
      "version": {
        "major": "24",
        "micro": "5",
        "minor": "11"
      },
      "release": "24.11.5",
      "cluster": "shield-cluster1"
    }
  }
}

3. 批量作业查询API响应（截取关键部分）

{
  "jobs": [
    {
      (...)
      "job_id": 67135476,
      "job_state": [
        "RUNNING"
      ],
      "nodes": "shield-c[1-2]",
      (...)
    }
  ],
  "errors": [],
  "warnings": []
}

4. Slurmrestd服务日志（单作业查询相关）

Sep 24 13:52:10 shield-head slurmrestd[753093]: debug:  _on_url: [[shield-head]:8218(fd:16)] url path: /slurm/v0.0.42/job/67135476 query: (null)
Sep 24 13:52:10 shield-head slurmrestd[753093]: slurmrestd: operations_router: [[shield-head]:8218(fd:16)] GET /slurm/v0.0.42/job/67135476
Sep 24 13:52:10 shield-head slurmrestd[753093]: slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[shield-head]:8218(fd:16)] attempting token authentication pass through
Sep 24 13:52:10 shield-head slurmrestd[753093]: slurmrestd: error: op_handler_job: [[shield-head]:8218(fd:16)] parser=data_parser/v0.0.42 rc[2017]=Invalid job id specified -> Invalid JobID=67135476 rejected
Sep 24 13:52:10 shield-head slurmrestd[753093]: operations_router: [[shield-head]:8218(fd:16)] GET /slurm/v0.0.42/job/67135476
Sep 24 13:52:10 shield-head slurmrestd[753093]: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[shield-head]:8218(fd:16)] attempting token authentication pass through
Sep 24 13:52:10 shield-head slurmrestd[753093]: debug:  wrap_openapi_ctxt_callback: [[shield-head]:8218(fd:16)] GET using data_parser/v0.0.42
Sep 24 13:52:10 shield-head slurmrestd[753093]: error: op_handler_job: [[shield-head]:8218(fd:16)] parser=data_parser/v0.0.42 rc[2017]=Invalid job id specified -> Invalid JobID=67135476 rejected

排查方向 #

• 校验API版本与数据解析器兼容性：当前使用的API版本是v0.0.42，尝试替换为与Slurm版本匹配的路由前缀（如/slurm/v24.11/job/{job_id}），部分旧版本解析器存在job ID解析bug。
• 确认作业ID传递格式：确保URL中的job_id是纯数字，无空格、转义字符或后缀。用curl直接构造请求（curl http://<slurmrestd-host>:8218/slurm/v24.11/job/67135476）排除客户端编码问题。
• 检查权限与认证配置：虽然批量查询正常，但单作业查询可能有更严格的权限校验。确认认证token包含单作业查询权限，或用root用户发起请求排除权限限制。
• 排查Slurmctld与Slurmdbd同步状态：单作业查询可能直接访问slurmctld，批量查询则从slurmdbd拉取数据。执行sacct -j 67135476查看会计数据库中是否存在该作业，若sacct能查到但API报错，可能是组件同步延迟或配置冲突。
• 检查官方补丁：查看Slurm 24.11.5的官方changelog，确认是否存在op_handler_job或job ID解析相关的已知bug及修复补丁。
• 开启高级调试日志：修改slurmrestd启动参数，添加-vvv开启最高级调试日志，追踪job ID在解析、查询环节的具体处理流程，定位问题节点。

内容的提问来源于stack exchange，提问作者Popoo