使用subprocess.run运行Jupyter Notebook脚本遇路径错误求助

我正在对多个CSV文件做数据聚合，为了节省时间开发了Python脚本，遍历子文件夹并部署名为format_script_s&s.ipynb的脚本。脚本成功将该文件复制到各子文件夹，但使用subprocess.run运行时出现错误。错误提示路径出现重复的嵌套结构，怀疑是Python对反斜杠的解析问题，同时不确定运行Jupyter Notebook文件是否需要特殊处理，无法自行解决，附上代码及错误信息求助。

代码 #

import os
import shutil
import subprocess

# Path to the folder containing subfolders
# base_folder = "/path/to/your/folder"
base_folder = "data_complete"

# Path to the script you want to place in each subfolder
script_to_place = "format_scripts/format_script_s&s.ipynb"

# Iterate through each item in the base folder
for item in os.listdir(base_folder):
    item_path = os.path.join(base_folder, item)

    # Check if the item is a directory
    if os.path.isdir(item_path):
        try:
            # Copy the script into the subfolder
            destination_script = os.path.join(item_path, "format_script_s&s.ipynb")
            shutil.copy(script_to_place, destination_script)
            print(f"Copied script to: {destination_script}")

            # Run the script inside the subfolder
            result = subprocess.run(
                ["python", destination_script],
                cwd=item_path,
                capture_output=True,
                text=True
            )

            # Print the output of the script
            print(f"Output from {item}:\n{result.stdout}")
            if result.stderr:
                print(f"Errors from {item}:\n{result.stderr}")

        except Exception as e:
            print(f"An error occurred with {item}: {e}")

错误信息 #

Copied script to: data_complete\2022-04\format_script_s&s.ipynb
data_complete\2022-04\format_script_s&s.ipynb
Output from 2022-04:

Errors from 2022-04:
python: can't open file 
'C:\\path_to_main_folder\\data_complete\\2022-04\\data_complete\\2022-04\\format_script_s&s.ipynb': [Errno 2] No such file or directory

核心问题分析 #

1. Jupyter Notebook无法直接用python命令执行：.ipynb是Jupyter的笔记本格式，不是普通Python脚本，python命令无法直接解析运行。
2. 路径重复问题：设置cwd=item_path后，传入完整路径destination_script会被系统和当前工作目录拼接，导致路径嵌套重复。
3. 文件名含特殊字符：文件名里的&在Windows系统中是命令分隔符，会干扰命令执行。

修改后的代码 #

import os
import shutil
import subprocess

base_folder = "data_complete"
script_to_place = "format_scripts/format_script_s&s.ipynb"

for item in os.listdir(base_folder):
    item_path = os.path.join(base_folder, item)
    if os.path.isdir(item_path):
        try:
            script_filename = "format_script_s&s.ipynb"
            destination_script = os.path.join(item_path, script_filename)
            shutil.copy(script_to_place, destination_script)
            print(f"Copied script to: {destination_script}")

            # 使用jupyter nbconvert执行Notebook
            result = subprocess.run(
                [
                    "jupyter", "nbconvert", "--execute", "--to", "notebook", "--inplace",
                    script_filename
                ],
                cwd=item_path,
                capture_output=True,
                text=True,
                shell=True  # 处理文件名中的&符号
            )

            print(f"Output from {item}:\n{result.stdout}")
            if result.stderr:
                print(f"Errors from {item}:\n{result.stderr}")

        except Exception as e:
            print(f"An error occurred with {item}: {e}")

关键修改点 #

• 替换执行命令：用jupyter nbconvert --execute来运行Notebook文件，--inplace表示在原文件上执行并保存结果，--to notebook保持输出为Notebook格式。
• 简化路径参数：因为已经设置cwd=item_path，只需要传入脚本文件名script_filename即可，避免路径拼接重复。
• 加入shell=True：处理文件名中的&特殊字符（注意：如果脚本路径或文件名来自不可信来源，shell=True会有安全风险，此时建议先修改文件名去掉特殊字符）。

内容的提问来源于stack exchange，提问作者MM3