使用Scrapy加载已过滤RSS Feed失败，求问题排查与解决

我已通过log.rlsbb.cc网站上的齿轮图标选项完成了RSS Feed的过滤设置，点击RSS图标获取到了包含所需过滤内容的Feed链接。随后我使用该链接通过Scrapy下载RSS XML，但输出结果包含未过滤内容及部分过滤内容，未达到预期效果。之后我在请求头的Cookie字段中添加了过滤参数，但返回的却是空文件。请问我哪里操作错误或存在理解偏差？

参考代码 #

import scrapy

headers = \
{'Host': 'log.rlsbb.cc',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Referer': 'https://log.rlsbb.cc/',
'Cookie': 'filters=foreign-movies,movies,tv-shows,old-movies,_foreign-movies_f-webrip,_foreign-movies_f-dvdrip-bdrip,\
_foreign-movies_f-bluray-720p,_foreign-movies_f-bluray-1080p,_movies_bluray-1080p,_movies_bluray-720p,_movies_bdrip,\
_movies_webrip,_movies_dvdrip,_movies_4k-uhd,_tv-shows_top,_tv-shows_tv-packs,_movies_old,_foreign-movies_f-old',
'Upgrade-Insecure-Requests': '1',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'same-origin',
'Sec-Fetch-User': '?1',
'Sec-GPC': '1',
'DNT': '1',
'TE': 'trailers'}

class ScrapeRlsBBRssSpider(scrapy.Spider):
    name = 'scrape_rlsbb_rss'
    allowed_domains = ['log.rlsbb.cc/feed']
    start_urls = ['http://https://log.rlsbb.cc/feed/']

    custom_settings={ 'FEED_URI': f"{name}_%(time)s.json",
                      'FEED_FORMAT': 'json'}

    def start_requests(self):
        urls = [
            'https://log.rlsbb.cc/feed/',
        ]

        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse, headers=headers)

    def parse(self, response):
        for post in response.xpath('//channel/item'):
            yield {
                'title' : post.xpath('title//text()').extract_first(),
                'link': post.xpath('link//text()').extract_first(),
                'pubDate' : post.xpath('pubDate//text()').extract_first(),
                'category': post.xpath('category//text()').extract_first(),
            }

问题分析与解决 #

• allowed_domains 配置错误：该参数仅需填写域名，不能包含路径。原代码中['log.rlsbb.cc/feed']会导致Scrapy判定请求URL不在允许范围内，修正为['log.rlsbb.cc']。
• start_urls 格式错误：原URL'http://https://log.rlsbb.cc/feed/'多了一层http://前缀，虽然start_requests中用了正确URL，但该错误可能干扰初始配置，建议修正为['https://log.rlsbb.cc/feed/']。
• Cookie 字符串换行问题：Cookie值中的反斜杠换行会引入无效的换行符，导致服务器无法正确解析过滤参数。需将Cookie值合并为一行，去掉换行和反斜杠：

  'Cookie': 'filters=foreign-movies,movies,tv-shows,old-movies,_foreign-movies_f-webrip,_foreign-movies_f-dvdrip-bdrip,_foreign-movies_f-bluray-720p,_foreign-movies_f-bluray-1080p,_movies_bluray-1080p,_movies_bluray-720p,_movies_bdrip,_movies_webrip,_movies_dvdrip,_movies_4k-uhd,_tv-shows_top,_tv-shows_tv-packs,_movies_old,_foreign-movies_f-old',
  

• 冗余请求头字段干扰：原请求头包含Upgrade-Insecure-Requests、Sec-Fetch-*等针对HTML页面的字段，RSS为XML格式，这些字段可能导致服务器返回异常。建议精简请求头，保留必要字段。

修正后代码示例 #

import scrapy

headers = {
    'Host': 'log.rlsbb.cc',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0',
    'Accept': 'application/xml,application/xhtml+xml,text/html;q=0.9,*/*;q=0.8',
    'Cookie': 'filters=foreign-movies,movies,tv-shows,old-movies,_foreign-movies_f-webrip,_foreign-movies_f-dvdrip-bdrip,_foreign-movies_f-bluray-720p,_foreign-movies_f-bluray-1080p,_movies_bluray-1080p,_movies_bluray-720p,_movies_bdrip,_movies_webrip,_movies_dvdrip,_movies_4k-uhd,_tv-shows_top,_tv-shows_tv-packs,_movies_old,_foreign-movies_f-old',
}

class ScrapeRlsBBRssSpider(scrapy.Spider):
    name = 'scrape_rlsbb_rss'
    allowed_domains = ['log.rlsbb.cc']
    start_urls = ['https://log.rlsbb.cc/feed/']

    custom_settings = {
        'FEED_URI': f"{name}_%(time)s.json",
        'FEED_FORMAT': 'json'
    }

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url, callback=self.parse, headers=headers)

    def parse(self, response):
        for post in response.xpath('//channel/item'):
            yield {
                'title': post.xpath('title/text()').get(),
                'link': post.xpath('link/text()').get(),
                'pubDate': post.xpath('pubDate/text()').get(),
                'category': post.xpath('category/text()').get(),
            }

内容的提问来源于stack exchange，提问作者Ozooha Ozooha