如何加载SageMaker中未提供的Python模块？如何pip安装Spacy？

Hi there, let's break down your two questions about working with custom Python libraries in Amazon SageMaker—these are really common pain points, so I'm glad you asked!

1. 如何加载Amazon SageMaker平台中未预装的Python模块？ #

The approach depends on whether you need the module for a one-off Notebook session, a persistent Notebook environment, or a training/inference job. Here are the most common methods:

• 临时安装在Notebook会话中
  If you just need the module for your current Notebook run (and don't mind it disappearing after a restart), simply run the pip install command directly in a code cell:

  !pip install your-target-module --upgrade
  

  This works for quick testing or ad-hoc analysis.

• 持久化安装到Notebook实例
  To make the module available every time you start your Notebook instance, use a Lifecycle Configuration:

  1. Head to the SageMaker Console, navigate to "Notebook" > "Lifecycle configurations"
  2. Create a new configuration, and add your pip install command to the "Start notebook" script:

     pip install your-target-module --upgrade
     

  3. When creating or updating your Notebook instance, select this lifecycle configuration. The module will install automatically on every startup.

• 安装到Training/Inference Jobs
  For jobs that run outside of Notebooks (like model training or real-time inference), you have two solid options:

  • Use a requirements.txt file: Create a requirements.txt in your training script directory, listing the modules you need (e.g., your-target-module==1.2.3). When defining your SageMaker Estimator, set the source_dir parameter to point to this directory—SageMaker will automatically run pip install -r requirements.txt before starting your job.
  • Custom Docker Image: If you need a highly customized environment, build a Docker image that includes your required modules, push it to Amazon ECR, and specify the image_uri parameter when creating your Estimator. This is great for complex dependencies or specific version combinations.

2. 针对SageMaker平台未提供的Spacy库，应如何通过pip命令完成安装？ #

Spacy is straightforward to install in SageMaker, and the method varies slightly based on your use case:

• In a SageMaker Notebook
  Run these commands in a code cell to install Spacy and any pre-trained models you need:

  # Install Spacy
  !pip install spacy --upgrade
  
  # Download a pre-trained model (example: English small model)
  !python -m spacy download en_core_web_sm
  

  If you run into permission issues, add the --user flag: !pip install spacy --user

• In a Training Job
  Two reliable ways to get Spacy set up for training:

  1. Via requirements.txt: Add Spacy to your requirements.txt file:

     spacy>=3.5.0
     

     Then, in your training script, add code to download the pre-trained model (if needed):

     import spacy
     spacy.cli.download("en_core_web_sm")
     

     Point your Estimator's source_dir to the directory containing requirements.txt and your training script.
  2. Direct installation in training script (quick workaround, less clean):

     import subprocess
     # Install Spacy
     subprocess.check_call(["pip", "install", "spacy"])
     # Download model
     subprocess.check_call(["python", "-m", "spacy", "download", "en_core_web_sm"])
     

• Persistent Notebook installation
  To have Spacy available every time you start your Notebook instance, add these commands to your lifecycle configuration's "Start notebook" script:

  pip install spacy --upgrade
  python -m spacy download en_core_web_sm
  

内容的提问来源于stack exchange，提问作者Sujay DSa