如何解决Python爬虫中‘No module named proxy’报错问题？

Hey there, let's tackle this ModuleNotFoundError issue step by step. I've run into similar problems with old GitHub crawler repos that haven't been updated for Python 3.x, so here's what you can try:

1. Check for the missing proxy.py file #

The error comes from middleware.py trying to import PROXIES from a proxy module. First, dig through your downloaded crawler project files:

• If there's no proxy.py file in the same directory as middleware.py, this is almost certainly the problem. Many old crawler repos use a custom proxy.py file to store proxy configurations—often excluded from the repo via .gitignore to avoid sharing sensitive proxy details.
• To fix this, create a new proxy.py file in the same folder as middleware.py, and add your proxy list (or an empty list if you don't need proxies right now):

  # proxy.py
  PROXIES = [
      "http://your-proxy-address:port",
      # Add more proxies if you have them, or leave as [] if not using proxies
  ]
  

2. Fix Python 3.x compatibility issues #

You mentioned the code has outdated parts for Python 3.x. Here are the most common fixes for old crawler code:

• Renamed modules: Replace Python 2-specific modules like urlparse with urllib.parse, ConfigParser with configparser, and StringIO with io.StringIO.
• Print statements: Swap Python 2's print "crawler log" with Python 3's print("crawler log").
• String encoding: Old code might mix up str and bytes types. Use .encode('utf-8') or .decode('utf-8') as needed, or explicitly cast with str() where required.
• Dictionary methods: Replace .iteritems() with .items(), .iterkeys() with .keys(), since the old iter- prefixed methods were removed in Python 3.

3. Verify module import paths #

If proxy.py does exist but you still get the error, the directory containing it might not be in Python's import path. Add this snippet at the top of middleware.py to force Python to look in the current directory:

import sys
import os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))

This ensures Python can locate nearby custom modules like proxy.py.

4. Check for third-party proxy packages (last resort) #

In rare cases, the proxy module might be a third-party package. Try installing it with:

pip install proxy

But note this is unlikely to be what the original repo intended—most crawlers use a custom proxy.py for their specific proxy setup, so only try this if the above steps don't work.

Once you fix the proxy module issue, you can work through any remaining Python 3 compatibility errors one by one—old crawler repos usually only need small tweaks to run smoothly on modern Python versions.

内容的提问来源于stack exchange，提问作者Z. Black