Scrapy运行报错ModuleNotFoundError: No module named 'http.client'求助

Hey, let's break down what's happening here and fix that annoying import error.

Root Cause #

The problem boils down to Python's module import path priority. Looking at your traceback, notice this critical line:

File "/opt/python/run/venv/lib/python3.6/site-packages/scrapy/http/__init__.py", line 8, in <module>

When Python tried to load the standard library's http.cookiejar (as part of importing urllib.request), it accidentally grabbed the http module from your Scrapy project (since you were working directly in the django_project/spider directory) instead of the system's built-in http package. Scrapy's http module doesn't have a client submodule, hence the error.

Fixes to Try (Start with the First One—it's the Simplest) #

1. Change to the Correct Working Directory #

Don't run Scrapy commands from inside the django_project/spider folder. Move up to your Django project root (django_project) and execute the command there. Also, make sure you target your actual spider name (myspider, not spider—since your spider file is myspider.py):

cd /path/to/django_project
/opt/python/run/venv/bin/scrapy crawl myspider

If Scrapy can't auto-detect your project settings, explicitly specify them:

/opt/python/run/venv/bin/scrapy crawl myspider --set=SPIDER_MODULES=spider.spiders --set=NEWSPIDER_MODULE=spider.spiders

2. Adjust PYTHONPATH (If You Must Stay in the spider Directory) #

If you need to run the command from django_project/spider, force Python to prioritize standard library and virtual environment packages first by updating your PYTHONPATH:

export PYTHONPATH="/opt/python/run/venv/lib/python3.6/site-packages:/usr/lib64/python3.6:$PYTHONPATH"
/opt/python/run/venv/bin/python3.6 /opt/python/run/venv/local/lib/python3.6/site-packages/scrapy/cmdline.py crawl myspider

This ensures Python looks for standard modules like http.client before checking your local directory.

3. Verify Virtual Environment Dependencies #

If the above fixes don't work, your virtual environment might have corrupted or incompatible packages. Reinstall the key dependencies:

source /opt/python/run/venv/bin/activate
pip install --upgrade scrapy twisted six w3lib

Final Note #

The first fix is the most reliable here—import path conflicts from working in the wrong directory are extremely common in combined Django + Scrapy projects. Just moving up one directory should resolve the http.client error immediately.

内容的提问来源于stack exchange，提问作者Milano