Jupyter Notebook转生产脚本：Jupyter魔法命令处理方案问询

Hey there! I get it—you've converted your Jupyter Notebook to a .py script with nbconvert, but those handy ! shell magic commands are breaking things. Since this is a one-time static analysis workflow, let's go through practical fixes to replace those commands properly:

1. Swap shell commands for Python standard library functions #

For common Linux commands, there's almost always a Python equivalent that's more integrated and avoids shell dependency:

• Instead of !mkdir -p /foo/bar, use os.makedirs (the exist_ok=True flag replicates the -p behavior of not throwing errors if the directory exists):

  import os
  os.makedirs('/foo/bar', exist_ok=True)
  

• For file copying/moving, use the shutil module instead of !cp or !mv:

  import shutil
  shutil.copy('source/file.txt', 'dest/file.txt')
  

2. Use subprocess for commands without direct Python equivalents #

If you need to keep using tools like AWS CLI or niche shell commands, the subprocess module lets you run shell commands directly from Python, just like the Jupyter magic:

• Replace !aws s3 cp local/data.csv s3://my-bucket/data/ with:

  import subprocess
  # Pass command arguments as a list to avoid shell injection risks
  subprocess.run(
      ['aws', 's3', 'cp', 'local/data.csv', 's3://my-bucket/data/'],
      check=True,  # Raises an error if the command fails (matches Jupyter's behavior)
      text=True    # Captures output as string instead of bytes
  )
  

• If you need to capture the command's output (like you would with !command > output.txt), add capture_output=True and access result.stdout:

  result = subprocess.run(['aws', 's3', 'ls'], capture_output=True, text=True)
  print(result.stdout)
  

3. Pro tip: Consider replacing AWS CLI with boto3 #

Since you're working in Python, using the official AWS SDK (boto3) is more robust than calling AWS CLI via shell commands. It integrates better with your Python code, lets you handle errors programmatically, and avoids relying on the shell environment having AWS CLI configured. For example, the S3 copy command becomes:

import boto3
s3 = boto3.client('s3')
s3.upload_file('local/data.csv', 'my-bucket', 'data/data.csv')

Quick cleanup note #

If you have a lot of magic commands to replace, you can use simple regex find-and-replace in your editor (e.g., find ^! lines and convert them to subprocess.run() calls), but always test each replacement to make sure it behaves exactly like the original magic command.

内容的提问来源于stack exchange，提问作者Rutger Hofste