如何将Scrapy独立爬虫打包为Windows可执行文件及报错解决

Looks like you're hitting a super common snag with PyInstaller and Scrapy—since Scrapy relies heavily on dynamic imports, PyInstaller's auto-dependency scanner misses some critical internal modules like scrapy.spiderloader. Let's walk through the steps to fix this and get your myspidy.exe running smoothly for users without Python/Scrapy installed.

Step 1: Ditch the scrapy crawl Command — Use a Custom Entry Script #

You can't directly package the scrapy crawl CLI command. Instead, create a Python script that launches your spider programmatically. Make a main.py in your Scrapy project root with this code:

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

# Replace this with your actual spider class import
# Example: if your spider is in spiders/quotes.py, use from your_project.spiders.quotes import QuotesSpider
from quotes_spider.spiders.quotes import QuotesSpider

if __name__ == "__main__":
    # Load your project's settings automatically
    process = CrawlerProcess(get_project_settings())
    # Start your target spider
    process.crawl(QuotesSpider)
    process.start()

Double-check the import path matches your spider's location—adjust it if your project structure is different.

Step 2: Force PyInstaller to Include Missing Scrapy Modules #

PyInstaller doesn't pick up all of Scrapy's dynamic modules on its own, so we need to explicitly list them as "hidden imports". You have two straightforward options:

Option A: Pack Directly via Command Line #

Run this command in your project directory to generate a single-file executable:

pyinstaller --onefile --hidden-import scrapy.spiderloader --hidden-import scrapy.core.downloader.handlers.http11 --hidden-import scrapy.core.downloader.middleware --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.ftp main.py

The --hidden-import flags tell PyInstaller to include modules it would otherwise miss. We're adding scrapy.spiderloader (the one throwing your error) plus other core Scrapy modules that often cause similar missing-module issues.

Option B: Use a .spec File for Reusable Configuration #

If you want a more flexible, reusable setup:

1. Generate a spec file first:

   pyi-makespec --onefile main.py
   

2. Open the generated main.spec file, find the hiddenimports list, and add the missing modules:

   a = Analysis(
       ['main.py'],
       # ... other default settings ...
       hiddenimports=['scrapy.spiderloader', 'scrapy.core.downloader.handlers.http11', 'scrapy.core.downloader.middleware', 'scrapy.core.downloader.handlers.file', 'scrapy.core.downloader.handlers.ftp'],
       # ... rest of the file ...
   )
   

3. Pack using the spec file:

   pyinstaller main.spec
   

Step 3: Double-Check Project Structure & Settings #

• Ensure your Scrapy project has all required __init__.py files (they should exist by default, but double-check if you modified the folder structure).
• If you're working with a standalone spider file (not a full Scrapy project), replace get_project_settings() with a manual settings object in main.py:

  from scrapy.settings import Settings
  
  custom_settings = Settings({
      'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
      # Add other necessary settings like ITEM_PIPELINES, DOWNLOAD_DELAY, etc., as needed
  })
  process = CrawlerProcess(custom_settings)
  

Step 4: Test Your Executable #

After packing, head to the dist folder and run your generated myspidy.exe. If you hit another ModuleNotFoundError, just add that missing module to your hiddenimports list and re-pack—this is normal for Scrapy projects, as some edge-case modules might still slip through.

内容的提问来源于stack exchange，提问作者webbeing