使用AWS Neuron SDK编译模型时遭遇磁盘空间不足错误求助

在Amazon SageMaker Studio中使用AWS Neuron SDK编译mosaicml/mpt-7b-instruct模型以部署到Inferentia实例时，出现磁盘空间不足错误，核心报错信息如下：

ERROR:Neuron:Not enough free space to write 268435456 bytes
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/torch_neuron/convert.py", line 413, in op_converter
    neuron_function = self.subgraph_compiler(
  File "/opt/conda/lib/python3.9/site-packages/torch_neuron/decorators.py", line 135, in trace
    weight_paths = write_params(model_dir, constant_parameter_tensors, constant_param_key_mapping)
  File "/opt/conda/lib/python3.9/site-packages/torch_neuron/decorators.py", line 1517, in write_params
    np.save(f'{directory}/weights/{file_name}.npy', weight.numpy())
  File "<__array_function__ internals>", line 5, in save
  File "/opt/conda/lib/python3.9/site-packages/numpy/lib/npyio.py", line 529, in save
    format.write_array(fid, arr, allow_pickle=allow_pickle,
  File "/opt/conda/lib/python3.9/site-packages/numpy/lib/format.py", line 691, in write_array
    array.tofile(fp)
OSError: Not enough free space to write 268435456 bytes

使用的编译命令为：

# Compile the model using the wrapper
compiler_args = ['--verbose','1', '--num-neuroncores', str(1)]
optimizations = ['FLOAT32_TO_FLOAT16']
model_neuron = torch.neuron.trace(mpt_model, example, compiler_args=compiler_args, optimizations=optimizations, separate_weights=True, strict=False)

运行环境为ml.m5.4xlarge实例，通过df -h确认编译过程使用overlay文件系统，空间不足。

方案一：扩容SageMaker Studio用户存储磁盘 #

SageMaker Studio默认用户存储为20GB，编译大模型时空间不足，可通过控制台扩容：

1. 登录AWS SageMaker控制台，进入对应的Domain详情页
2. 切换到User profiles标签，选择当前使用的用户，点击Edit
3. 在Storage volume size (GB)项中设置更大的容量（如100GB），点击Save changes
4. 重启当前的SageMaker Studio应用，新的磁盘容量会生效

注意：用户存储仅支持扩容，无法缩容。

方案二：指定Neuron编译临时目录到大容量存储 #

Neuron SDK通过环境变量指定编译临时目录，可将其指向SageMaker默认挂载的EFS目录（/home/sagemaker-user/下，空间充足）：

1. 创建自定义临时目录：

   mkdir -p /home/sagemaker-user/neuron_compile_temp
   

2. 设置环境变量（当前会话生效）：

   export NEURON_COMPILE_CACHE_DIR=/home/sagemaker-user/neuron_compile_temp
   export TMPDIR=/home/sagemaker-user/neuron_compile_temp
   

   若需永久生效，将上述命令添加到~/.bashrc或~/.profile文件，执行source ~/.bashrc使设置立即生效
3. 重新运行模型编译命令即可

内容的提问来源于stack exchange，提问作者Ishank Gulati