Scrapy初始请求陷入无限循环，无法触发parse回调问题排查

问题现象 #

爬虫初始请求陷入无限循环，始终无法触发parse回调函数。预期流程为：发送初始请求→修改请求体→执行parse回调，但实际卡在请求处理阶段，日志显示重复执行process_request，从未进入parse逻辑，简化测试案例也复现该问题。

核心代码 #

spider.py #

def start_requests(self):
    yield scrapy.Request(url=self.url, method='POST', callback=self.parse, dont_filter=True, flags=['this is the start request method'])

def parse(self, response):
    logging.info('started parsing')
    l = ItemLoader(FeedItem())
    json_response = json.loads(response.text)
    l.add_value('feed', json_response)
    yield l.load_items()
    yield scrapy.Request(url=self.url, method='POST', callback=self.parse, flags=['this is the parse method'])

middlewares.py #

def process_request(self, request, spider):
    sinceId = self.cur.execute('SELECT sinceId FROM proposal').fetchone()
    jobT = self.cur.execute('SELECT jobT FROM proposal').fetchone()
    if not sinceId:
        body = self.body.encode('utf-8')
        request = request.replace(body=body)
        spider.logger.info(f'{request.flags}')
        spider.logger.info('Returning unchanged request')
        return request
    body = re.sub(r'("sinceId":")(\d+)(")', '"sinceId":' + f'"{sinceId}"', self.body)
    body = re.sub(r'("jobTime":")(\d+)(")', '"jobTime":' + f'"{jobT}"', body)
    body = self.body.encode('utf-8')
    spider.logger.info('Body changed')
    request = request.replace(body=body)
    spider.logger.info(f'{request.flags}')
    spider.logger.info('Returning changed request')
    return request

def spider_opened(self, spider):
    self.body = '{"query":"\n          query($queryParams: UserSavedSearchesParams) {\n            userSavedSearches(params: $queryParams) {\n              results {\n                id\n                uid:id\n                title\n                ciphertext\n                description\n                type\n                recno\n                freelancersToHire\n                duration\n                durationLabel\n                engagement\n                amount {\n                  amount:displayValue\n                }\n                createdOn:createdDateTime\n                publishedOn:publishedDateTime\n                renewedOn:renewedDateTime\n                prefFreelancerLocation\n                prefFreelancerLocationMandatory\n                connectPrice\n                client {\n                  totalHires\n                  totalPostedJobs\n                  totalSpent {\n                    rawValue\n                    currency\n                    displayValue\n                  }\n                  paymentVerificationStatus,\n                  location {\n                    country\n                  }\n                  totalReviews\n                  totalFeedback\n                  companyRid\n                  edcUserId\n                  lastContractRid\n                  companyOrgUid\n                  hasFinancialPrivacy\n                }\n                enterpriseJob\n                premium\n                jobTs:jobTime\n                skills {\n                  id\n                  name\n                  prettyName\n                  highlighted\n                }\n                contractorTier\n                jobStatus\n                relevanceEncoded\n                totalApplicants\n                proposalsTier\n                isLocal:local\n                locations {\n                  city\n                  country\n                }\n                isApplied:applied\n                attrs {\n                  id\n                  uid:id\n                  prettyName:prefLabel\n                  parentSkillId\n                  prefLabel\n                  highlighted\n                  freeText\n                }\n                hourlyBudget {\n                  type\n                  min\n                  max\n                }\n                clientRelation {\n                  companyRid\n                  companyName\n                  edcUserId\n                  lastContractPlatform\n                  lastContractRid\n                  lastContractTitle\n                }\n                totalFreelancersToHire\n                contractToHire\n              }\n              paging {\n                  total\n                  count\n                  resultSetTs:resultSetTime\n              }\n            }\n          }\n        ","variables":{"queryParams":{"sinceId":"1015914410","jobTime":"1728208823055","paging":"0;20"}}}'
    self.body = self.body.replace('\n','\\n')
    self.con = sqlite3.connect('data.db')
    self.cur = self.con.cursor()
    self.cur.execute('CREATE TABLE IF NOT EXISTS proposal(title, description, _type, duration, salary, sinceID, jobT, UNIQUE(title, description, _type, duration, salary, sinceID, jobT))')
    spider.logger.info("Spider opened: %s" % spider.name)

日志片段 #

2024-10-08 11:12:32 [feed] INFO: ['this is the start request method']
2024-10-08 11:12:32 [feed] INFO: Returning unchanged request
2024-10-08 11:12:33 [feed] INFO: ['this is the start request method']
2024-10-08 11:12:33 [feed] INFO: Returning unchanged request

简化测试案例 #

spider.py #

name='test'
url = 'https://httpbin.org/post'   
def start_requests(self):
    yield scrapy.Request(url=self.url, method='POST', body="{'key': 'value'}", callback=self.parse)

def parse(self, response):
    self.log(response.json())

middlewares.py #

import time

def process_request(self, request, spider):
    new_body = "{'key1': 'new_value1', 'key2': 'new_value2'}"
    request = request.replace(
        body=new_body,
        dont_filter = True
    )
    time.sleep(3)
    spider.logger.info('body changed')
    return request

测试日志重复输出body changed，未触发parse回调。

问题原因与解决方案 #

根本原因 #

Scrapy下载中间件的process_request方法返回规则：

• 返回Request对象：Scrapy会重新调度该请求，导致请求再次经过所有中间件，形成无限循环
• 返回None：Scrapy会继续处理当前请求，发送给下载器，完成后进入parse回调

你的代码中每次process_request都返回新的Request对象，导致请求永远在中间件阶段循环，无法到达下载环节。

修复方法 #

修改中间件的process_request，修改请求属性后返回None，而不是返回Request对象：

针对核心代码的修改

def process_request(self, request, spider):
    sinceId = self.cur.execute('SELECT sinceId FROM proposal').fetchone()
    jobT = self.cur.execute('SELECT jobT FROM proposal').fetchone()
    if not sinceId:
        body = self.body.encode('utf-8')
        # 直接修改原请求的body
        request._set_body(body)
        spider.logger.info(f'{request.flags}')
        spider.logger.info('Body set to initial value')
        return None  # 返回None让Scrapy继续处理请求
    body = re.sub(r'("sinceId":")(\d+)(")', '"sinceId":' + f'"{sinceId}"', self.body)
    body = re.sub(r'("jobTime":")(\d+)(")', '"jobTime":' + f'"{jobT}"', body)
    body = body.encode('utf-8')  # 修正：使用处理后的body而非原self.body
    spider.logger.info('Body changed')
    request._set_body(body)
    spider.logger.info(f'{request.flags}')
    spider.logger.info('Returning modified request')
    return None  # 返回None

针对测试案例的修改

import time

def process_request(self, request, spider):
    new_body = "{'key1': 'new_value1', 'key2': 'new_value2'}"
    request._set_body(new_body)
    request.dont_filter = True
    time.sleep(3)
    spider.logger.info('body changed')
    return None  # 返回None

额外注意点 #

1. 核心代码中原本的body = self.body.encode('utf-8')是错误的，处理后的body应该用修改后的变量，而不是原self.body
2. 使用request._set_body()直接修改请求体，比request.replace()更直接，避免创建新请求对象

内容的提问来源于stack exchange，提问作者TEOTU