PDF文件下载报错：Scrapy爬虫代码执行异常求助

Hey Eric, let's walk through the common issues that might be causing your Scrapy spider to fail, along with straightforward fixes for each scenario:

1. You're Running the Script Directly (Not Using Scrapy's Command Line) #

Scrapy spiders rely on the framework's engine to handle requests, so running your script with python your_script.py won't work—you'll get initialization errors every time.
Fix:

• Save your code as test_spider.py inside your Scrapy project's spiders folder
• Run it with Scrapy's official crawl command:

scrapy crawl test_spider

2. Permission Denied When Saving the File #

If your system blocks write access to the test directory (or its parent folder), you'll get a permission error.
Fix:

• Run your terminal with elevated permissions (e.g., sudo on Linux/macOS, or as Administrator on Windows)
• Or switch to an absolute path you know you have access to, like your user's Downloads folder:

save_path = os.path.expanduser("~/Downloads/test")

3. Network/Request Blocked by the Target Site #

The sample PDF URL might be unreachable, or the site is rejecting Scrapy's default "bot-like" user-agent.
Fix:

• First, check if you can open http://www.pdf995.com/samples/pdf.pdf in your browser. If it's down, use a different test PDF URL.
• Add a custom user-agent to mimic a real browser by updating your spider:

class TestSpider(scrapy.Spider):
    name = 'test_spider'
    start_urls = ['http://www.pdf995.com/samples/pdf.pdf', ]
    # Mimic a real browser's user-agent
    custom_settings = {
        'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
    }

4. Scrapy Isn't Installed #

If you see ModuleNotFoundError: No module named 'scrapy', you just need to install the framework first:

pip install scrapy

5. Rare: Explicit Path Import Issue #

While os.path is part of the os module, some environments might require an explicit import if you get a NameError for os.path. Fix it by adding:

from os import path

Then replace os.path.join with path.join in your save_page method.

内容的提问来源于stack exchange，提问作者Eric G