如何优化CPU环境下PyTorch模型（ResNet18）的推理性能以适配网站部署？

Hey there! Let's tackle that ResNet18 CPU inference bottleneck on your AMD Ryzen 3600—totally understand why you're concerned about Heroku handling that 70% CPU load. Here are practical, actionable optimizations you can implement right away:

These are the easiest wins that require minimal code changes:

• Switch to TorchScript for Static Graph Execution
  Python's dynamic nature adds overhead during inference. TorchScript converts your model to a static graph, eliminating that overhead and enabling better compiler optimizations. Try this:

  # First, set model to evaluation mode (critical for all optimizations!)
  model = model.eval()
  
  # Create an example input matching your input shape (e.g., 1x3x224x224)
  example_input = torch.randn(1, 3, 224, 224)
  
  # Trace the model to generate TorchScript
  traced_model = torch.jit.trace(model, example_input)
  
  # Save and load for deployment
  traced_model.save("resnet18_traced.pt")
  loaded_model = torch.jit.load("resnet18_traced.pt")
  

• Tune Thread Count for AMD's 6-Core CPU
  The Ryzen 3600 has 6 physical cores (12 threads), but over-subscribing threads causes unnecessary context switching. Set these environment variables before running your inference:

  export OMP_NUM_THREADS=6
  export MKL_NUM_THREADS=6
  

  This tells PyTorch's underlying math libraries (MKL-DNN/OpenBLAS) to use exactly 6 threads, matching your physical core count for optimal utilization.
• Quantize Your Model to 8-bit Integers
  Quantization reduces the model's size and speeds up inference on CPUs by converting 32-bit floats to 8-bit integers. PyTorch's dynamic quantization works great for models like ResNet18 with lots of linear layers:

  model = model.eval()
  quantized_model = torch.quantization.quantize_dynamic(
      model, {torch.nn.Linear}, dtype=torch.qint8
  )
  

  You'll get ~2-3x speedups with minimal accuracy loss (often unnoticeable for most use cases).

Small tweaks to how you run inference can add up:

• Batch Process Inputs (When Possible)
  CPUs excel at parallelizing work across batches. If you're handling multiple images at once, process them in batches instead of one-by-one. Just make sure your batch size is reasonable (start with 4-8) to avoid memory bloat.
• Speed Up Image Preprocessing
  Avoid slow Python loops for preprocessing. Use vectorized operations with libraries like OpenCV or NumPy:

  # Example: Batch resize and normalize with OpenCV
  import cv2
  import numpy as np
  
  def preprocess_batch(images):
      resized = np.array([cv2.resize(img, (224, 224)) for img in images])
      normalized = resized / 255.0
      # Convert to PyTorch tensor and add batch/channel dimensions
      return torch.tensor(normalized).permute(0, 3, 1, 2).float()
  

• Always Keep Model in Evaluation Mode
  Don't forget to call model.eval() before inference! This disables training-specific layers like dropout and batch normalization, which add unnecessary computation and variance.

Since you're deploying to Heroku, these adjustments will help the platform handle your optimized model:

• Use a Slim Docker Image
  If you're using Docker, base your image on python:3.10-slim instead of the full Python image. This reduces overhead and ensures only necessary dependencies are installed.
• Match Heroku Dyno Threads to Your Optimizations
  Heroku's standard dynos have 2 vCPUs, so adjust your OMP_NUM_THREADS to 2 when deploying (instead of 6 for your local 3600). This prevents overloading the dyno's CPU resources.
• Use Asynchronous Endpoints
  If you're building a web API, use an async framework like FastAPI. Async endpoints let you handle multiple requests concurrently without blocking the CPU, smoothing out peak load.

For even more gains, convert your model to ONNX format and run it with ONNX Runtime—this framework is heavily optimized for CPU inference:

# Export PyTorch model to ONNX
model = model.eval()
example_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, example_input, "resnet18.onnx", opset_version=13)

# Run inference with ONNX Runtime
import onnxruntime as ort

session = ort.InferenceSession("resnet18.onnx")
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name

# Preprocess your input to a numpy array
input_np = preprocess_single_image(img).numpy()
output = session.run([output_name], {input_name: input_np})

ONNX Runtime often delivers an extra 10-30% speedup over optimized PyTorch, thanks to operator fusion and advanced CPU scheduling.

Start with the quick wins (eval mode, thread tuning, dynamic quantization)—those should cut your CPU usage significantly without too much work. If you still need more speed, move to TorchScript or ONNX Runtime. Heroku should handle the optimized model way better once you reduce that overhead!

内容的提问来源于stack exchange，提问作者이준혁