PyPSA-eur在Ubuntu WSL中运行Snakemake因内存过高被终止求助

数据老陈

2026-7-3

问题描述

开发波罗的海区域PyPSA-Eur能源系统模型时，在Ubuntu WSL（VS Code环境）中运行克隆自GitHub的PyPSA-Eur代码，执行snakemake --cores all触发build_renewable_profiles规则时，因内存占用过高被系统发送SIGKILL信号终止。已尝试调整Dask Worker内存限制、在WSL中通过.wslconfig分配8GB内存（本机总内存16GB），但问题仍未解决。

错误日志如下：

INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (95.49s)
INFO:atlite.convert:Convert and aggregate 'pv'.
2023-12-27 14:44:34,179 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 1.38 GiB -- Worker memory limit: 1.91 GiB
2023-12-27 14:44:36,119 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 1.36 GiB -- Worker memory limit: 1.91 GiB
2023-12-27 14:44:37,196 - distributed.worker.memory - WARNING - Worker is at 81% memory usage. Pausing worker.  Process memory: 1.55 GiB -- Worker memory limit: 1.91 GiB
[Wed Dec 27 14:44:38 2023]
Error in rule build_renewable_profiles:
    jobid: 10
    input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
    output: resources/profile_solar.nc
    log: logs/build_renewable_profile_solar.log (check log file(s) for error details)
    conda-env: /home/cfl/pypsa-balticsea/.snakemake/conda/a193a967e4b3183c6023115ed840b879_

RuleException:
CalledProcessError in file /home/cfl/pypsa-balticsea/rules/build_electricity.smk, line 309:
Command 'set -euo pipefail;  /home/cfl/miniconda3/envs/pypsa-eur/bin/python3.11 /home/cfl/pypsa-balticsea/.snakemake/scripts/tmpwdpwcq1a.build_renewable_profiles.py' died with <Signals.SIGKILL: 9>.
  File "/home/cfl/pypsa-balticsea/rules/build_electricity.smk", line 309, in __rule_build_renewable_profiles
  File "/home/cfl/miniconda3/envs/pypsa-eur/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
/home/cfl/miniconda3/envs/pypsa-eur/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 24 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
Complete log: .snakemake/log/2023-12-27T144217.073372.snakemake.log

解决方案

限制Snakemake并行核心数
避免使用--cores all，手动指定更少的核心数（比如--cores 4）。多核心并行会叠加每个Dask Worker的内存占用，减少核心可直接降低总内存消耗，适配WSL的内存上限。
激进调整Dask内存管理参数
在PyPSA-Eur的config.yaml配置文件中，修改Dask相关设置：
- 降低dask_worker_memory_limit至1GB左右；
- 添加内存阈值配置，让Worker更早释放内存：
```
dask:
  worker:
    memory:
      target: 0.6
      spill: 0.7
      pause: 0.8
      terminate: 0.95
```
缩小计算范围至波罗的海区域
修改config.yaml中的regions字段，仅保留波罗的海周边国家（如丹麦、德国北部、波兰、立陶宛、拉脱维亚、爱沙尼亚、芬兰、瑞典），重新生成仅覆盖目标区域的cutout文件。这会大幅减少待处理的地理数据量，从根源降低内存需求。
裁剪土地利用数据子集
用GDAL工具裁剪Corine土地利用数据至波罗的海区域，替换原全欧洲数据：
```
gdalwarp -te [波罗的海区域xmin ymin xmax ymax] data/bundle/corine/g250_clc06_V18_5.tif data/bundle/corine/baltic_corine.tif
```
之后在config.yaml中更新土地利用数据路径为裁剪后的文件。
提升WSL内存与交换分区
修改Windows用户目录下的.wslconfig文件，分配更多内存并启用交换分区（本机16GB内存可分配10GB给WSL）：
```
[wsl2]
memory=10GB
swap=8GB
```
交换分区可在内存不足时临时用磁盘空间缓解压力，需重启WSL生效。
单独运行规则并分析内存热点
用单核心单独运行该规则，同时用内存分析工具定位内存占用峰值：
1. 安装memory-profiler：pip install memory-profiler
2. 单独触发规则：snakemake --cores 1 resources/profile_solar.nc
3. 找到生成的临时脚本，在主函数前添加@profile装饰器，运行：
```
python -m memory_profiler /path/to/tmp_script.py
```
根据分析结果针对性优化代码中内存占用高的片段。

内容的提问来源于stack exchange，提问作者Stijn