FastAPI集成Scrapy后保存脚本致Scrapy关闭的解决方法咨询

我用FastAPI和Scrapy写了个集成脚本，POST和GET请求都能正常运行，但本地开发时，修改脚本保存后，Scrapy会自动关闭，说明本地运行时Scrapy没有正常完成关闭流程。

示例代码 #

路由脚本 #

from fastapi import APIRouter
from scrapy.crawler import CrawlerProcess
from collections import defaultdict
from spiders.xp import postItem
import sys

rout = APIRouter()

# 全局变量
globalDict = defaultdict(list)

@rout.post("/event")
async def add_todo(tealium: dict = None):
    if tealium is not None:
        # 从POST请求中获取数据并全局存储
        print(tealium)
        
        process = CrawlerProcess(settings={
                "FEEDS": {
                    "post.json": {"format": "json"},
                },
            })
        process.crawl(postItem, tealium=tealium)
        
        process.start(stop_after_crawl=True)

        if "twisted.internet.reactor" in sys.modules:
            del sys.modules["twisted.internet.reactor"]
      
        with open('post.json', 'r') as f:
            value = f.read()
        globalDict['value'].append([value])
        print(globalDict)

        return {'message': 'Post request completed'}

    else:
        print(tealium)
        globalDict['result'] = ['No Post Request Sent']
        return globalDict

@rout.get("/event")
async def retrieve_todos() -> dict:
    return { "todos": globalDict}

应用脚本 #

from fastapi import FastAPI
from router.router1 import rout

app = FastAPI()

@app.get("/")
async def welcome() -> dict:
    return { "message": 'Hello World!' }

app.include_router(rout)

解决方法 #

1. 替换CrawlerProcess为CrawlerRunner #

你当前每次请求都删除twisted.internet.reactor模块，会导致热重载时Reactor状态混乱。改用CrawlerRunner可以避免自动启动/停止Reactor，更好兼容FastAPI的异步环境：

修改路由脚本的爬虫启动逻辑：

from scrapy.crawler import CrawlerRunner
from twisted.internet import reactor
from twisted.internet.defer import Deferred
from asyncio import Future

# 工具函数：将Twisted Deferred转换为FastAPI支持的异步对象
def twisted_deferred_to_async(deferred: Deferred) -> Future:
    future = Future()
    def callback(result):
        if not future.done():
            future.set_result(result)
    def errback(failure):
        if not future.done():
            future.set_exception(failure.value)
    deferred.addCallback(callback)
    deferred.addErrback(errback)
    return future

# 全局初始化Runner，避免重复创建
runner = CrawlerRunner(settings={
        "FEEDS": {
            "post.json": {"format": "json"},
        },
    })

@rout.post("/event")
async def add_todo(tealium: dict = None):
    if tealium is not None:
        print(tealium)
        
        deferred = runner.crawl(postItem, tealium=tealium)
        # 异步等待爬虫完成，不阻塞FastAPI事件循环
        await twisted_deferred_to_async(deferred)
      
        with open('post.json', 'r') as f:
            value = f.read()
        globalDict['value'].append([value])
        print(globalDict)

        return {'message': 'Post request completed'}

2. 配置热重载排除规则 #

如果用uvicorn启动服务，热重载会重新加载所有模块，导致Twisted Reactor重复初始化。启动时添加排除参数，避免Scrapy相关模块被热重载：

uvicorn main:app --reload --reload-exclude "spiders/*" --reload-exclude "scrapy/*"

3. 避免重复操作Reactor模块 #

删除你代码中每次请求后删除twisted.internet.reactor的逻辑，这个操作会破坏Twisted的全局状态，导致热重载时爬虫无法正常重启。

内容的提问来源于stack exchange，提问作者Dollar Tune-bill