Scrapy可下载HTML文件却无法下载XML文件的问题排查

问题描述 #

当前爬虫仅提取已爬页面中的XML链接，但未适配URL存在差异的实际可访问XML页面，需调整爬虫规则；同时脚本虽能抓取所有链接并将XML链接写入CSV，但无法成功下载XML文件，仅HTML文件保存正常。

URL差异示例 #

• 页面提取的原始XML链接：http://www.digitalhumanities.org/dhq/vol/12/4/000401/000401.xml
• 实际可访问的XML页面URL：http://www.digitalhumanities.org/dhq/vol/12/4/000401.xml

报错日志 #

2022-12-22 12:33:30 [scrapy.pipelines.files] WARNING: File (code: 302): Error downloading file from <GET http://www.digitalhumanities.org/dhq/vol/12/4/000407.xml> referred in <None>
2022-12-22 12:33:30 [scrapy.core.engine] DEBUG: Crawled (302) <GET http://www.digitalhumanities.org/dhq/vol/12/4/000408.xml> (referer: None)
2022-12-22 12:33:30 [scrapy.pipelines.files] WARNING: File (code: 302): Error downloading file from <GET http://www.digitalhumanities.org/dhq/vol/12/4/000408.xml> referred in <None>

现有爬虫代码 #

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class DhqSpider(CrawlSpider):
    name = 'dhqfiles'
    allowed_domains = ['digitalhumanities.org']
    start_urls = ['http://www.digitalhumanities.org/dhq/vol/16/3/index.html']

    rules = (
            Rule(LinkExtractor(allow = 'index.html')), 
            Rule(LinkExtractor(allow = 'vol'), callback='parse_article'),        
        )

    def parse_article(self, response):
        article = { 
            'xmllink' : response.urljoin(response.xpath('(//div[@class="toolbar"]/a[contains(@href, ".xml")]/@href)[1]').get()),
        }
        yield {'file_urls':[article['xmllink']]}

修复方案 #

1. 修正XML链接格式 #

针对提取的链接存在冗余目录的问题，通过字符串处理去除重复路径部分：

def parse_article(self, response):
    raw_xml_href = response.xpath('(//div[@class="toolbar"]/a[contains(@href, ".xml")]/@href)[1]').get()
    if raw_xml_href:
        # 拆分链接路径，去除冗余的同名子目录
        path_parts = raw_xml_href.split('/')
        filename = path_parts[-1].replace('.xml', '')
        if len(path_parts) >= 2 and path_parts[-2] == filename:
            corrected_href = '/'.join(path_parts[:-2] + [path_parts[-1]])
        else:
            corrected_href = raw_xml_href
        xmllink = response.urljoin(corrected_href)
        yield {'file_urls': [xmllink]}

2. 处理302重定向与Referer缺失问题 #

在settings.py中添加配置，确保爬虫自动跟随重定向并携带Referer：

# 基础配置
DOWNLOAD_DELAY = 1
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'

# 文件存储配置
FILES_STORE = './downloaded_files'
FILES_URLS_FIELD = 'file_urls'
FILES_RESULT_FIELD = 'files'

# 启用Referer中间件
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.referer.RefererMiddleware': 700,
}

3. 优化爬虫规则 #

缩小链接匹配范围，避免无效页面抓取，确保遍历所有文章详情页：

rules = (
    # 跟进卷期索引页面
    Rule(LinkExtractor(allow=r'/vol/\d+/\d+/index\.html$'), follow=True),
    # 匹配文章详情页并调用解析函数
    Rule(LinkExtractor(allow=r'/vol/\d+/\d+/\d+$'), callback='parse_article'),
)

内容的提问来源于stack exchange，提问作者David Beales