Slurm集群使用submitit时open()触发Errno7参数列表过长问题求助

在Slurm GPU集群上用submitit Python包运行任务时，不定时出现OSError: [Errno 7] Argument list too long错误，该错误发生在调用open(path, 'wb')保存JAX模型（agent）的环节，多数保存操作可正常执行。

已知该错误指向参数过多或环境变量过大，已打印os.environ确认各次调用中无变化，保存路径长度差异极小（最多相差一个字符，同长度路径能正常保存），示例路径为logs/GCBC_explore_32c_disc-acttraj/run_logs/configuration_26/phase_0/seed_3/params_50000.pkl。目前无法稳定复现该问题，以下是报错堆栈、任务提交代码及模型保存代码，求解决思路。

报错堆栈 #

1047   │ Traceback (most recent call last):
1048   │   File "<frozen runpy>", line 198, in _run_module_as_main
1049   │   File "<frozen runpy>", line 88, in _run_code
1050   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/submitit/core/_submit.py", line 11, in <module>
1051   │     submitit_main()
1052   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/submitit/core/submission.py", line 76, in submitit_main
1053   │     process_job(args.folder)
1054   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/submitit/core/submission.py", line 69, in process_job
1055   │     raise error
1056   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/submitit/core/submission.py", line 55, in process_job
1057   │     result = delayed.result()
1058   │              ^^^^^^^^^^^^^^^^
1059   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/submitit/core/utils.py", line 137, in result
1060   │     self._result = self.function(*self.args, **self.kwargs)
1061   │                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1062   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/gcrl_landscapes/main.py", line 250, in run_config_slurm_tasks_wrapper
1063   │     return run_config(
1064   │            ^^^^^^^^^^^
1065   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/gcrl_landscapes/main.py", line 160, in run_config
1066   │     eval_trajectory = train(
1067   │                       ^^^^^^
1068   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/gcrl_landscapes/training.py", line 270, in train
1069   │     save_agent(agent, str(save_dir), i)
1070   │   File "/bigwork/username/.conda/envs/gcrl/lib/python3.12/site-packages/ogbench/impls/utils/flax_utils.py", line 175, in save_agent
1071   │     with open(save_path, 'wb') as f:
1072   │          ^^^^^^^^^^^^^^^^^^^^^
1073   │ OSError: [Errno 7] Argument list too long

任务提交代码 #

executor = submitit.AutoExecutor(folder=str(args.logdir / "submitit" / "%j"))
executor.update_parameters(
    cpus_per_task=4,
    slurm_time=int(60 * args.tasks_per_node * ((200000 - args.phase) / 200000)),  # this overestimates, keep safety margin
    slurm_gpus_per_node=1,
    tasks_per_node=args.tasks_per_node,
    slurm_mem_per_cpu="1G",
    slurm_array_parallelism=50,
    slurm_partition=args.partition,
    slurm_job_name=args.jobname,
    slurm_mail_user=...,
    slurm_mail_type="BEGIN,FAIL,END",
)
executor.map_array(run_config_slurm_tasks_wrapper, *chunked_arguments)

模型保存代码 #

def save_agent(agent, save_dir, epoch):
    """Save the agent to a file.

    Args:
        agent: Agent.
        save_dir: Directory to save the agent.
        epoch: Epoch number.
    """

    save_dict = dict(
        agent=flax.serialization.to_state_dict(agent),
    )
    save_path = os.path.join(save_dir, f'params_{epoch}.pkl')
    with open(save_path, 'wb') as f:
        pickle.dump(save_dict, f)

    print(f'Saved to {save_path}')

解决思路 #

• 清理任务环境变量：submitit在数组任务中可能动态累积环境变量，即便初始os.environ无变化，运行中也可能溢出。在任务入口函数run_config_slurm_tasks_wrapper开头保留必要环境变量，清空其余：

  import os
  # 仅保留关键环境变量，按需调整
  keep_vars = ["PATH", "LD_LIBRARY_PATH", "CONDA_PREFIX", "CONDA_DEFAULT_ENV"]
  filtered_env = {k: v for k, v in os.environ.items() if k in keep_vars}
  os.environ.clear()
  os.environ.update(filtered_env)
  

• 更换文件打开方式：改用pathlib.Path的open方法替代os.open，可能绕过系统对参数长度的检测，同时确保目录存在：

  from pathlib import Path
  def save_agent(agent, save_dir, epoch):
      save_dict = dict(agent=flax.serialization.to_state_dict(agent))
      save_path = Path(save_dir) / f'params_{epoch}.pkl'
      save_path.parent.mkdir(parents=True, exist_ok=True)
      with save_path.open('wb') as f:
          pickle.dump(save_dict, f)
      print(f'Saved to {save_path}')
  

• 优化任务提交方式：executor.map_array传递大量参数时可能导致命令行/环境变量溢出，尝试减少单批次参数数量，或改用循环调用executor.submit逐个提交任务，避免数组任务的参数累积。
• 临时文件中转保存：先将模型保存到临时文件，再移动到目标路径，绕过路径相关的参数长度限制：

  import tempfile
  import shutil
  def save_agent(agent, save_dir, epoch):
      save_dict = dict(agent=flax.serialization.to_state_dict(agent))
      save_path = os.path.join(save_dir, f'params_{epoch}.pkl')
      with tempfile.NamedTemporaryFile(mode='wb', delete=False) as tmp_file:
          pickle.dump(save_dict, tmp_file)
      shutil.move(tmp_file.name, save_path)
      print(f'Saved to {save_path}')
  

• 添加错误重试逻辑：因问题不定时出现，在保存环节捕获Errno 7错误后重试几次：

  import time
  def save_agent(agent, save_dir, epoch, max_retries=3):
      save_dict = dict(agent=flax.serialization.to_state_dict(agent))
      save_path = os.path.join(save_dir, f'params_{epoch}.pkl')
      for retry in range(max_retries):
          try:
              with open(save_path, 'wb') as f:
                  pickle.dump(save_dict, f)
              print(f'Saved to {save_path}')
              return
          except OSError as e:
              if e.errno == 7:
                  time.sleep(1)
                  continue
              raise
      raise RuntimeError(f"Failed to save agent after {max_retries} retries")
  

内容的提问来源于stack exchange，提问作者CrunchyFlakes